Beaker provision plugin, more distros in testcloud, improved custom results...

Highlights

• New beaker provision using mrack allows to test on all arches from your laptop
• The testcloud plugin is now working with fedora, centos, alma, rocky, oracle, debian and ubuntu
• Improved custom results reporting (fixed logs, docs, now allows results.json as well)
• New kickstart section added to the provision step specification
• Variables TMT_TEST_NAME and TMT_TEST_METADATA now available to tests

Full Changelog

• Change help text of the tmt --root option
• Add support for results.json in custom results
• Proper support for the test duration format
• Prepend '/' to custom test result name if missing
• Document necessary packages for pip install on Ubuntu
• Tag cloud resources to tmt in Testing Farm
• Display guest multihost name even in dry run (#1982)
• Pass the arch option to the Beaker provider
• Use job-id instead of guestname in Beaker class
• Adjust the fix for the default branch handling
• Add support to get ref under the git worktree
• Fix dry mode handling when running a remote plan
• Enable the external polarion plugin tests
• Extract "run a command" functionality into a stand-alone helper
• Increase minimal severity of ShellCheck defects
• Display guest full name in display plugin report
• Push using sudo rsync when necessary
• Avoid warning from installing tmt as pre-commit
• Add test checking repeated test execution results
• Freeze the yq version to fix el8 installation
• Update the CODEOWNERS file with more granularity
• Document current workaround for running scripts
• Install beakerlib before the ShellCheck
• Rename Guest.full_name to Guest.multihost_name
• Display guest full name in html plugin report
• Add test for template-based export plugin
• Add kickstart to the artemis provision plugin
• Extract just tar files in dist-git-source
• Add missing fields to custom results test
• Add shell linter Differential ShellCheck
• Always try to run dhclient in cloud-init in virtual provision
• Fix polarion report pruning and add or fix arguments
• Run chcon only if SELinux fs supported
• Require beaker provision in tmt-all
• Adjust the new mrack plugin spec, test and plan
• Add beaker provision plugin using mrack
• Adjust pip install to always upgrade to the latest
• Move testcloud url guessing logic out of tmt
• Hotfix Ubuntu with virtual provision
• Detect correct category when export to nitrate
• Add an entrypoint for interactive tmt sessions
• Fix internal handling of the where key
• Move logging labels to the beginning of lines
• Refactor CLI error reporting to improve readability
• Remove no longer needed cast around our custom Click context
• Display guest full name when showing its details
• Add kickstart section as a new specification key
• Add more controls for output colorization
• Rephrase results.yaml documentation and examples
• Fix get_bootstrap_logger name and docstring typo
• Expose guest info in results
• Enable root login and disable default core for rhcos
• Sanitize plan/test/story names before filtering
• Set default user core for rhcos in testcloud
• Remove no longer used "err" parameter of logging methods
• testcloud: Raise default limits
• Update log key content of results.yaml examples (#1834)
• Include guest name in execute phase data paths
• Adds "bootstrap logger" for logging before CLI options are recognized (#1839)
• Export TMT_TEST_NAME and TMT_TEST_METADATA (#1888)
• List supported operators in hardware requirement docs (#1867)
• Build tmt usable in inner guests for tests/full
• Target test-complement for tests/full
• Tag tests which are affected by how=full
• Use PROVISION_METHODS in tests
• Report individual test results in tests/full
• Use Require* classes for collection & installation of plugin requirements (#1766)
• Disable tracebacks if default branch is not found
• Assign a data path and serial number to each test in discover (#1876)
• Convert log path for results:custom
• Allow report result for itself in results:custom
• Support to import Makefile having '\\n'
• Require pylero for the polarion subpackage
• Fix forgotten guest when Artemis provisioning times out
• Turn tests.yaml into a list of tests
• Simplify the Result class implementation
• Use Path instead of os.path in export code
• Use Path when working with logfile path
• Fix use of old os.path.symlink() in discover/shell
• Add /root/.local/bin to PATH on Centos Stream 8 in CI
• Install jq/yq for more readable tests in tmt test suite
• Fix Common class ignoring other branches of multiple inheritance tree
• Use Path instead of os.path in prepare/install plugin
• Convert path-like strings to pathlib.Path objects
• Change Plugin.go() to accept logger and extra environment
• Artemis API version may contain multiple integers
• Add logging labels used for prefixing messages
• Adds "full name" guest property for multihost logging

Option to sync git repo for discover.shell; ReportPortal report plugin; web-url

Highlights

• Do not clone the whole remote plan in dry mode
• Include a simple Python code among the examples
• Enable to sync git repo to SUT in shell discover
• Enable url and ref as shell discover options
• Respect TMT_WORKDIR_ROOT variable in testcloud
• Implement the reportportal report plugin
• Add a clickable web link to test to polarion export
• Enhance Links to allow checking for any links at all

Whole change log

• Fix tmt-reboot without custom command
• Fix test /discover/libraries
• Add serialization callbacks to data class fields
• Use own private key for provision.virtual
• Adds a template-backed export plugin
• Polarion export fix component upload bug and upload id first
• Convert story ReST export to use a Jinja2 template
• Convert export-related code to plugins per format
• Do not clone the whole remote plan in dry mode
• Hardcode tmt git URL so test won't fail for PRs
• Add py.typed marker for 3rd party type annotations
• Fixes isort 5.10.1 installation issue
• Improve logging by tmt.utils.wait()
• Check packages are installed via debuginfo-install
• Always ignore failures for recommended packages
• Merge report plugins options into step data fields
• Dynamically find the current Fedora release
• Suggest using a pull request checklist template
• Include a simple Python code among the examples
• Apply normalization callback when updating data with CLI input
• Bump pre-commit linters - Flake8, Mypy, JSON schema, YAML lint & pygrep
• Use base implementation of provision plugin requirements
• Relay 'interactive' value for podman call
• Update Fedora versions in upgrade tests
• Apply ShellScript for the custom reboot command
• Update the shell discover specification
• Enable to sync git repo to SUT in shell discover
• Increase the default utils.format() indent a bit
• Define pull request Copr build job in Packit config
• Decouple logging from objects and base classes
• Enable url and ref as shell discover options
• Export TMT_TREE in other steps as well
• Add a new key system to the hardware spec
• Remove default for the dynamic ref evaluation
• Schema update and test for order in discover step
• Merge report plugins options into step data fields
• Add a test for hardware schema coverage
• Better type annotations of prepare/install scripts
• Move jinja2 require to the main tmt package
• Define the new context dimension initiator
• Respect TMT_WORKDIR_ROOT variable in testcloud
• Annotate commands, command line elements and shell scripts
• Adjust the reportportal plugin implementation
• Implement the reportportal report plugin
• Require the latest testcloud package
• Define srpm_build_deps in the packit config
• Include the new web link in verbose show mode
• Add a clickable web link to test to polarion export
• Enhance Links to allow checking for any links at all
• Drop various guest wake() methods in favor of parent class
• Catch SystemExit during module discovery

Improved export; verbose output for provision and prepare; TMT_WORKDIR_ROOT; provision error handling; prune in finish step

Highlights

• Do not leak "private" fields into export
• Set guest hostname in testcloud provision
• Capture provision error when login is used
• Support TMT_WORKDIR_ROOT environment variable
• Support step data definitions carrying CLI options
• Prune irrelevant files during the finish step
• Support to import empty key from Makefile
• Support absolute paths in HTML reports
• Capture exceptions when getting image_url
• Enable verbose output for provision & prepare
• Initial support for passing ssh options from cli
• Update specification of the where multihost key

Full changelog

• Do not prune html and junit reports
• Skip extending fmf context if cli context missing
• Connect needs is_ready property as well
• Cover setup.py with pre-commit Python checks
• Do not leak "private" fields into export
• Set guest hostname in testcloud provision
• Capture provision error when login is used
• Support TMT_WORKDIR_ROOT environment variable
• Support step data definitions carrying CLI options
• Adds flake8 coverage for bin/ directory
• Prune irrelevant files during the finish step
• Add junit plugin schema
• Support to import empty key from Makefile
• Deleting unsed and duplicite part of finish step
• Support absolute paths in HTML reports
• Capture exceptions when getting image_url
• Enable verbose output for provision & prepare
• Add support for Artemis v0.0.47 upcoming release
• Remove unused variables
• Initial support for passing ssh options from cli
• Update specification of the where multihost key
• Add a simple test demonstrating the upgrade testing
• Use custom subclass of click.Context for better annotations
• Extend the duration for tests using containers
• Change common class constructors to use keyword arguments only
• Make packit build with the next release.dev version
• Add basic test coverage for tmt story export
• Fix export of the story priority field
• Read source from correct directory if ref is used

Fix duration, import remote plan, dynamic ref parametrization, remove obsoleted execution methods

Highlights

• Fix test duration enforcement
• Support fetching remote repo for shell discover
• Fix default framework, remove old execution methods
• Login after each test using the --test parameter
• Enable context based plan parametrization
• Add support for dynamic ref evaluation
• Add import plan feature

Full Changelog

• Protect args in rsync call
• Set tree root for the default plan tree as well
• Properly set the tmt script shebang on rhel-8
• Use image exists to check for container image
• Updates docs with example on adjust & prepare+
• Fix test duration enforcement
• Skip missing debuginfo packages in recommend
• Explicitly document extending the plan environment
• Fix ownership of a tmp directory propagated to container
• Support fetching remote repo for shell discover
• Fix default framework, remove old execution methods
• Add support for Artemis v0.0.46 upcoming release
• Handle an fmf fetch error in remote plan clone
• Do not truncate RunError output in verbose mode
• Warn user about data erasing after prepare step
• Formalize data package passed to the Result class
• Change order of plugin and guest classes in files
• Add compatible as a new hardware specification key
• Add tpm as a new hardware specification key
• Move the hardware specification into a separate page
• Improve fmf-id processing
• Add test for TTY state in test environment
• Login after each test using the --test parameter
• Mention version where important features were added
• Handle dist-git-sources for gitlab
• Fix getting CentOS via --how virtual
• Capture uncaught exceptions when using testcloud
• Update the overview of essential classes
• When following command line --how, do not iterate over step data
• Convert utils' Run unit tests to class-less tests
• Enable variable expansion for dynamic references
• Support beaker libraries as recommended packages
• Add SpecBasedContainer.to_minimal_spec() method (#1637)
• Enable context based plan parametrization
• Coverage for tests defined under discover.shell
• Fix /tests/run/shell access permission problem
• Add step data classes for provisioning and report
• Adjust support for the dynamic ref evaluation
• Add support for dynamic ref evaluation
• Add a test for fmf id parsing and normalization
• Fix NO_COLOR not being honored by executed command output
• Moves common command options into one place
• Log full chain of exceptions, not just the first cause
• Drop deprecated PluginIndex
• Add message to failure tags in junit report
• Update the pip installation plan
• Remove pointless reimport of tmt.base in discover steps
• Do not use f-string as a docstring
• Fix use of variable before assignment in Plan.go
• Fix variable redefinition in discover/shell.py
• Fix guest distro detection, do not throw results away
• Fix a typo in cpu.sockets hardware requirement
• Make links relative for report html
• Review all uses of type: ignore and link relevant issues
• Enhance ClickOptionDecorator type to announce identity
• Update /tests/execute/upgrade/override duration
• Review all uses of Any and link relevant issues
• Annotate all cast() calls with respective issues
• Move the Result class into a separate file
• Unblock mypy's follow-import setting
• Use set comprehension instead of list-in-set sequence
• Use enumerate() instead of range(len())
• Use dict comprehension instead of tuple-in-list-in-dict
• Replace two more list comprehensions with generators
• Replace GeneralError's "origin" with Python's "raise from"
• Fix normalization of the Plan.context key
• Adds a missing import to polarion plugin
• Update all linters to their most recent versions
• Define CPU HW components with more granularity
• Adjust the support for importing remote plans
• Add import plan feature and tests
• Add type annotations to base.py
• Fix test depending on ordering of elements in junit XML
• Adds type annotations to tmt itself
• Remove custom yet same implementation of step's show()
• Make sure repo_copy is gone before make srpm

Custom results, polarion import, typing, refactoring...

Highlights

• Custom results implementation
• Add Polarion as a source for test case import
• Support to install package from URL
• Refactor internal link handling and storage
• Substantial progress on typing annotations

Full Changelog

• Fix recommended packages handling for rpm-ostree
• Add EFI configuration to the tmt-reboot script
• Fix adjust for precommit test
• Fix provision for coreos image
• Emit only non-default keys when constructing a test from execute
• Add flake8 config file for easier integration with IDEs
• Multiple scripts for CLI prepare -h shell
• Allow mypy to cover the whole tmt.plugins and tmt.steps
• Add typing for steps/execute/upgrade.py
• Fix name & default value of polarion's upload field
• Fixes enhancing of environment by local's guest implementation
• Ignore plan -n when searching for upgrade path
• Document & correct use of class conversion methods
• Print fmf tree location when schema unit test fails
• Custom results implementation
• Refactors internal link handling and storage
• Allow mypy to cover whole tmt.steps.prepare
• Add typing for steps/execute/internal.py
• Use workdir with safe names without special chars
• Adjust support for installing remote packages
• Support to install package from URL
• Make sure short option '-x' is covered
• Add Polarion as a source for test case import
• Print path to the used ssh identity
• Add typing for steps/prepare/__init__.py
• Use generator instead of list comprehension with any/all
• Fixes handling of default of --key in connect plugin
• Update test data for the debuginfo install test
• Add a helper for importing a member from a module
• Fix plan schema to allow custom context dimensions
• Allow mypy to cover whole tmt.steps.discover
• Remove support for the obsoleted detach executor
• Add typing for steps/discover/fmf.py
• Fix importing for pylero
• Allow mypy to cover whole tmt.steps.provision
• Replace blank "type: ignore" with more specific waivers
• Use the SerializableContainer for plugins' data
• Enhance SerializableContainer with default key value inspection
• Moves validation and normalization mixins to utils

Remote playbooks in prepare; Common attributes extended; Nested root warning

Main improvements and fixes:

• Error out if reboot timeout is exceeded
• Initialize workdir thread-safe
• Add support for remote playbooks in prepare
• Print escaped command suitable for manual debugging
• Less eager to disable nitrate case during export
• Move tag and tier to common core attributes
• Hint user if 'tmt init' creates nested root

Whole change log:

• Unify Polarion case searching
• Error out if reboot timeout is exceeded
• Initialize workdir thread-safe
• Add support for remote playbooks in prepare
• Add plan schema for errata and minute plugins
• Correct rhts command names in stories file
• Print escaped command suitable for manual debugging
• Fix report plugin not getting arguments from fmf file
• Less eager to disable nitrate case during export
• Move tag and tier to common core attributes
• Use /bin/bash instead of /bin/sh for execute
• Reorder step and their base plugin classes
• Fix prepare/multihost docs to match implementation
• Teach schema validation tests to peek into other trees
• Clarify motivation for creating id during export
• Add link-polarion option and fix link searching bug
• Ignore race in last-run symlink creation
• Fix polarion tcmscaseid search
• Force order of clean operations
• Convert status/clean argument to option
• Report enabled plans/tests/stories via option
• Hint user if 'tmt init' creates nested root
• Require libvirt-daemon-config-network as well
• Add type annotation for /steps/prepare/install.py
• Encapsulate created file within script's dataclass
• Adds normalization layer to base classes based on fmf
• Fixes data class used for local guest creation
• Fixes Artemis guest data class link
• Making tests/full more usable
• Add typing for tmt/steps/provision/podman.py
• Add typing for tmt/steps/provision/testcloud.py
• Add typing for tmt/steps/provision/local.py
• Remove unused keys parameter from wake() methods
• Adds types describing tmt constructs when as stored in raw fmf data
• Typing /steps/provision/connect.py
• Allow raising an exception on validation errors
• Inject logger object to base node classes inheritance
• Fixes use of SSH keys in testcloud and connect plugins
• Annotate tmt.steps.provision
• Ask mypy to show error codes in its messages
• Testcloud expects disk and memory to be int
• Do not inherit classes from object
• Use keyword-only init in base fmf-backed classes
• Use decorator to register plugin methods
• Demonstrate inheritance on a virtual test example
• Add a simple hint how to write user messages
• Add typing for steps/finish/ansible.py
• Remove unneeded parameters for step load/save (#1428)
• Normalize step data to be always stored a list internally
• Display test/plan/story name in parametrized schema tests
• Allow numbers and booleans to be values of environment
• Give a reasonable error for old data format
• Add typing for tmt/steps/finish/init.py
• add typing for steps/discover/shell.py
• Adds a fmf node validation layer to core classes (Test/Plan/Story)
• Add missing keys role and where to schemas
• Extend plan schema with all known step plugin schemas
• Correcting rhts aliases & adding rhts opt.
• Adds a generic "wait for condition" primitive
• Disallow push/pull/execute if guest is unavailable
• Rename "default how" step attribute to enhance its visibility
• Use textwrap.dedent() to unindent docstrings

Reboot timeout and reconnect improvements; polarion report; library recursion

Main improvements and fixes:

• Backwards compatibility for rstrnt-abort, rstrnt-report-log and rstrnt-report-result.
• Package installation using rpm-ostree.
• Polarion report plugin.
• Implement reboot reconnect timeout configuration and make sure machine was rebooted before reconnect.
• Cut circular dependency of libraries.
• Print result for execute -v.
• Fixing the directory name escaping in 'cd' command.
• Precommit hooks to call tmt * lint

Whole change log:

• Reboot has to check for boot time
• Fix path inside pre-commit test
• Cut circular dependency of libraries
• Update 'Develop' section of contribution docs
• Precommit hooks to call tmt * lint
• Schema loading helpers
• Package schemas in subdirectories too
• Implement reboot reconnect timeout configuration
• Add missing report specifications/docs
• Print result for execute -v
• Correct import assumption about script
• Ask mypy to check whole tmt.steps.report package
• Fixing the directory name escaping in 'cd' command
• Add polarion report plugin
• Add schemas for plans
• Write extra-nitrate as soon as possible
• Retry git clone without --depth=1 if it failed
• Support to lint attribute 'id'
• Do not apply test command filter in upgrade
• Fix export.py typing issues
• Refactor location and signature of Phase's go() method
• Simplify abort handling
• Backwards compatibility for rstrnt-abort
• Add type annotations for tmt/cli.py
• Add typing for tmt/export.py
• add typing for steps/discover/init.py
• Convert guest implementations to use serializable container for load/save
• Detect plugins by entry_point as well
• Add typing for steps/execute/__init__.py
• Add typing for tmt/convert.py
• Remove duplicated dist-git-source/dist-git-type
• Add typing for steps/report/junit.py
• Add typing for steps/report/html.py
• add typing for steps/report/display.py
• Add typing for steps/report/init.py
• tmt-file-submit is a bash script
• Add type annotations for tmt/options.py
• Backwards compatibility for rstrnt-report-log
• Support conditional requires with pip as well
• Remove duplicated short option in tmt lint
• Adjust the rstrnt-report-result implementation
• Backwards compatibility for rstrnt-report-result
• Set the 1777 permision for /var/tmp/tmt
• Adjust the fix for the rpm-ostree intallation
• Fix package installation using rpm-ostree
• Handle empty fmf file as an empty dictionary
• Fix distgit testsuite after tmt packaging change

CoreOS support, typing, result attribute, polarion export

Highlights

• Initial polarion support for test export
• Add CoreOS support to the testcloud provision
• Add support for the rpm-ostree package manager
• Implement the test result attribute

Full changelog

• Require fresh testcloud with coreos support
• Bad substitution in tmt-reboot
• Ignore "certificate verify failed" error when handling retries
• Cache content of each loaded environment file
• Initial polarion support for test export
• Fixes names of Artemis API versions
• Convert FmfIdType from TypedDict to a dataclass
• Add CoreOS support to the testcloud provision
• Run containers with root user
• Retry getting environment file
• Test import --general is default now
• Add typing for steps/finish/shell.py
• Enhance tmt.utils.retry_session with timeout support
• Adjust the rpm-ostree install implementation
• Add support for the rpm-ostree package manager
• Add environment-file to possible Plan keys
• Avoid Library url conflict if repo doesn't exist
• Check changes are pushed before export nitrate
• Add typing for beakerlib.py
• Unbundle template from the report.html plugin
• Rename uuid to identifier to prevent conflicts
• Use must for all mandatory spec requirements
• Fail import for packages starting with minus sign
• Adds support for newer Artemis API versions
• Disable the extra verbose progress in testcloud
• Refactor internal executor scripts
• Adds "missing" imports to help IDEs follow objects
• Add typing for steps/init.py
• Implement the test result attribute
• Add typing for plugins/init.py
• Detect legacy relevancy during import as well
• Implement the new user story key priority
• Implement new class SerializableContainer
• Add schema for stories
• Add typing for steps/prepare/shell.py
• Add typing for steps/prepare/ansible.py
• Require fmf >= 1.1.0 (we need validation support)
• Package fmf root into the source tarball as well
• Add JSON Schema for tests
• Exclude namespaced backup in beakerlib
• Use --depth=1 when cloning git repos by default
• Handle missing nitrate user during export
• Removes unused GuestContainer.container_id attribute
• Every subpackage must require the main tmt package
• Introduce dataclasses as a requirement
• Avoid re-using image/instance for different values by testcloud plugin
• Add typing for multihost.py
• Except nitrate xmlrpc issues during import
• Exclude beakerlib's backup dir from guest.pull()
• Increase duration for the reboot-related tests
• Several release-related tests and docs adjustments

Upgrade plugin, improved discover, identifiers...

Highlights

• Implement upgrade execute plugin
• Dist-git-source for Discover (fmf, shell)
• Add a new core key id for unique identifiers

Full Changelog

• Command 'tmt clean' should not run rsync at all
• Dist-git-source for Discover (fmf, shell)
• Adjust the new id key implementation
• Add a new core key id for unique identifiers
• Recommend qemu emulators for other architectures
• Copy the whole git repo only if necessary
• Reveal hidden errors during testcloud booting
• Use time for timeout handling in Guest.reconnect()
• Split Guest class to separate SSH-capable guests
• Explicitly set the docs language in the config
• Kill the running test when rebooting
• Extend the reboot timeout to allow system upgrades
• Allow selecting tasks directly from upgrade config
• Adjust the new upgrade execute plugin
• Allow specifying command for reboot
• Implement upgrade execute plugin
• Buildrequire python3-docutils, append plan adjust
• Implement tmt tests export --nitrate --link-runs
• Detect component from general plan during import
• Adjust the support for steps in standalone mode
• Add results method to ExecutePlugin
• Implement a common ancestor for Action and Plugin
• Allow abstractly excluding steps from runs
• Correctly handle tests --name '.' shortcut
• Rename WorkdirType to WorkdirArgumentType
• Fix workdir parameter type for tmt.utils.Common
• Allows importing jira issues as link-relates
• Enables mypy coverage for empty-ish Python files
• Adds type annotations to tmt.templates
• Prevent infinite recursion if --id is set
• Enable mypy check for Artemis provision plugin
• Adjust provision dry mode propagation, add a test
• Introduce new _options attribute to Common class
• Add specification for remote plans referencing
• Bootstrap type annotations
• Execute script should not be used with discover
• Add the arch key to the hardware specification
• Fix pip install instructions
• Disable network access when building in copr
• Ignore list for dist-git-source
• Remove the obsoleted detach execute method
• Fix login during execute and prepare step
• Import from Makefile with missing build target

Multiarch testcloud, artemis provision, test export improvements...

• Add multiarch support to testcloud provision
• Consistent summary for test export --nitrate
• Allow dry mode for tests export --nitrate
• Add a nice provisioning progress to Artemis plugin
• Add support for the where keyword to execute
• Adjust support for export of multiple tests
• Add support for exporting multiple tests
• Basic multihost test for the httpd web server
• Update multihost specification with guest groups
• Add a provision plugin for Artemis
• Fix exclude option in fmf discover
• Reduce the number of execute calls for reboot
• Add support for reboot in interactive mode