Execute Plugins

Common Keys

The following keys are accepted by all plugins of the execute step.

name

The name of the step phase.

Environment variable: TMT_PLUGIN_EXECUTE_TMT_NAME

In plan metadata:

name:

On command-line:

--name ...
export TMT_PLUGIN_EXECUTE_TMT_NAME=...

order

Order in which the phase should be handled.

Default: 50

Environment variable: TMT_PLUGIN_EXECUTE_TMT_ORDER

In plan metadata:

order:

On command-line:

--order ...
export TMT_PLUGIN_EXECUTE_TMT_ORDER=...

summary

Concise summary describing purpose of the phase.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_TMT_SUMMARY

In plan metadata:

summary:

On command-line:

--summary ...
export TMT_PLUGIN_EXECUTE_TMT_SUMMARY=...

when

If specified, phase is run only if any rule matches plan context.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_TMT_WHEN

In plan metadata:

when: RULE

On command-line:

--when RULE
export TMT_PLUGIN_EXECUTE_TMT_WHEN=RULE

where

Run this phase on given guest or guests with the given role only.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_TMT_WHERE

In plan metadata:

where: GUEST|ROLE

On command-line:

--where 'GUEST|ROLE'
export TMT_PLUGIN_EXECUTE_TMT_WHERE='GUEST|ROLE'

tmt

Warning

Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal.

Use the internal tmt executor to execute tests.

The internal tmt executor runs tests on the guest one by one directly from the tmt code which shows testing progress and supports interactive debugging as well. This is the default execute step implementation. Test result is based on the script exit code (for shell tests) or the results file (for beakerlib tests).

The executor provides the following shell scripts which can be used by the tests for certain operations.

tmt-file-submit - archive the given file in the tmt test data directory. See the Save a log file section for more details.

tmt-reboot - soft reboot the machine from inside the test. After reboot the execution starts from the test which rebooted the machine. Use tmt-reboot -s for systemd soft-reboot which preserves the kernel and hardware state while restarting userspace only. An environment variable TMT_REBOOT_COUNT is provided which the test can use to handle the reboot. The variable holds the number of reboots performed by the test. For more information see the Reboot during test feature documentation.

tmt-report-result - generate a result report file from inside the test. Can be called multiple times by the test. The generated report file will be overwritten if a higher hierarchical result is reported by the test. The hierarchy is as follows: SKIP, PASS, WARN, FAIL. For more information see the Report test result feature documentation.

tmt-abort - generate an abort file from inside the test. This will set the current test result to failed and terminate the execution of subsequent tests. For more information see the Abort test execution feature documentation.

The scripts are hosted by default in the /usr/local/bin directory, except for guests using rpm-ostree, where /var/lib/tmt/scripts is used. The directory can be forced using the TMT_SCRIPTS_DIR environment variable. Note that for guests using rpm-ostree, the directory is added to executable paths using the system-wide /etc/profile.d/tmt.sh profile script.

Warning

Please be aware that for guests using rpm-ostree the provided scripts will only be available in a shell that loads the profile scripts. This is the default for bash-like shells, but might not work for others.

Configuration

See also Common Keys accepted by the plugin.

duration

The maximum time test specified by script is allowed to run. This option does not affect the duration defined in test metadata.

Default: 1h

Environment variable: TMT_PLUGIN_EXECUTE_TMT_DURATION

In plan metadata:

duration:

On command-line:

--duration ...
export TMT_PLUGIN_EXECUTE_TMT_DURATION=...

exit-first

Stop execution after the first test failure or error.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_EXIT_FIRST

In plan metadata:

exit-first: true|false

On command-line:

--exit-first
export TMT_PLUGIN_EXECUTE_TMT_EXIT_FIRST=1|0

ignore-duration

Ignore test duration value and allow test to run forever. Can be set by environment variable even when step is not specified on the commandline. This environment variable will be replaced by fmf config file or CLI arguments.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION

In plan metadata:

ignore-duration: true|false

On command-line:

--ignore-duration
export TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION=1|0

interactive

Run tests in interactive mode, i.e. with input and output streams shared with tmt itself. This allows input to be passed to tests via stdin, e.g. responding to password prompts. Test output in this mode is not captured, and duration has no effect.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE

In plan metadata:

interactive: true|false

On command-line:

--interactive
export TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE=1|0

no-progress-bar

Disable interactive progress bar showing the current test.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_NO_PROGRESS_BAR

In plan metadata:

no-progress-bar: true|false

On command-line:

--no-progress-bar
export TMT_PLUGIN_EXECUTE_TMT_NO_PROGRESS_BAR=1|0

restraint-compatible

Run tests in the restraint-compatible mode. Enable this if your tests depend on the restraint scripts such as rstrnt-report-result or rstrnt-report-log. Currently this option only affects whether the $OUTPUTFILE variable is respected, but in the future it will be used to enable/disable all restraint compatibility features.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_RESTRAINT_COMPATIBLE

In plan metadata:

restraint-compatible: true|false

On command-line:

--restraint-compatible
export TMT_PLUGIN_EXECUTE_TMT_RESTRAINT_COMPATIBLE=1|0

script

Execute arbitrary shell commands and check their exit code which is used as a test result. The script field is provided to cover simple test use cases only and must not be combined with the discover step which is more suitable for more complex test scenarios.

Default shell options are applied to the script, see test for more details. The default duration for tests defined directly under the execute step is 1h. Use the duration attribute to modify the default limit.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_TMT_SCRIPT

In plan metadata:

script: SCRIPT

On command-line:

--script SCRIPT
export TMT_PLUGIN_EXECUTE_TMT_SCRIPT=SCRIPT

---

upgrade

Warning

Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal.

Perform system upgrade during testing.

In order to enable developing tests for upgrade testing, we need to provide a way how to execute these tests easily. This does not cover unit tests for individual actors but rather system tests which verify the whole upgrade story.

The upgrade executor runs the discovered tests (using the internal executor), then performs a set of upgrade tasks from a remote repository, and finally, re-runs the tests on the upgraded guest.

The IN_PLACE_UPGRADE environment variable is set during the test execution to differentiate between the stages of the test. It is set to old during the first execution and new during the second execution. Test names are prefixed with this value to make the names unique. Based on this variable, the test can perform appropriate actions.

• old: setup, test

• new: test, cleanup

• without: setup, test, cleanup

The upgrade tasks performing the actual system upgrade are taken from a remote repository (specified by the url key) based on an upgrade path (e.g. fedora35to36) or other filters (e.g. specified by the filter key). If both upgrade-path and extra filters are specified, the discover keys in the remote upgrade path plan are overridden by the filters specified in the local plan.

The upgrade path must correspond to a plan name in the remote repository whose discover step selects tests (upgrade tasks) performing the upgrade. Currently, selection of upgrade tasks in the remote repository can be done using both fmf and shell discover method. If the url is not provided, upgrade path and upgrade tasks are taken from the current repository. The supported keys in discover are:

• ref

• filter

• exclude

• tests

• test

The environment variables defined in the remote upgrade path plan are passed to the upgrade tasks when they are executed. An example of an upgrade path plan (in the remote repository):

discover: # Selects appropriate upgrade tasks (L1 tests)
    how: fmf
    filter: "tag:fedora"
environment: # This is passed to upgrade tasks
    SOURCE: 35
    TARGET: 36
execute:
    how: tmt

If no upgrade path is specified in the plan, the tests (upgrade tasks) are selected based on the configuration of the upgrade plugin (e.g. based on the filter in its configuration).

If these two possible ways of specifying upgrade tasks are combined, the remote discover plan is used but its options are overridden with the values specified locally.

The same options and config keys and values can be used as in the internal executor.

Minimal execute config example with an upgrade path:

execute:
    how: upgrade
    url: https://github.com/teemtee/upgrade
    upgrade-path: /paths/fedora35to36

Execute config example without an upgrade path:

execute:
    how: upgrade
    url: https://github.com/teemtee/upgrade
    filter: "tag:fedora"

# A simple beakerlib test using the $IN_PLACE_UPGRADE variable
. /usr/share/beakerlib/beakerlib.sh || exit 1

VENV_PATH=/var/tmp/venv_test

rlJournalStart
    # Perform the setup only for the old distro
    if [[ "$IN_PLACE_UPGRADE" !=  "new" ]]; then
        rlPhaseStartSetup
            rlRun "python3.9 -m venv $VENV_PATH"
            rlRun "$VENV_PATH/bin/pip install pyjokes"
        rlPhaseEnd
    fi

    # Execute the test for both old & new distro
    rlPhaseStartTest
        rlAsssertExists "$VENV_PATH/bin/pyjoke"
        rlRun "$VENV_PATH/bin/pyjoke"
    rlPhaseEnd

    # Skip the cleanup phase when on the old distro
    if [[ "$IN_PLACE_UPGRADE" !=  "old" ]]; then
        rlPhaseStartCleanup
            rlRun "rm -rf $VENV_PATH"
        rlPhaseEnd
    fi
rlJournalEnd

Configuration

See also Common Keys accepted by the plugin.

duration

The maximum time test specified by script is allowed to run. This option does not affect the duration defined in test metadata.

Default: 1h

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_DURATION

In plan metadata:

duration:

On command-line:

--duration ...
export TMT_PLUGIN_EXECUTE_UPGRADE_DURATION=...

exclude

Exclude a regular expression from search result.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_EXCLUDE

In plan metadata:

exclude: REGEXP

On command-line:

--exclude REGEXP
export TMT_PLUGIN_EXECUTE_UPGRADE_EXCLUDE=REGEXP

exit-first

Stop execution after the first test failure or error.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_EXIT_FIRST

In plan metadata:

exit-first: true|false

On command-line:

--exit-first
export TMT_PLUGIN_EXECUTE_UPGRADE_EXIT_FIRST=1|0

filter

Include only tests matching the filter.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_FILTER

In plan metadata:

filter: FILTERS

On command-line:

--filter FILTERS
export TMT_PLUGIN_EXECUTE_UPGRADE_FILTER=FILTERS

ignore-duration

Ignore test duration value and allow test to run forever. Can be set by environment variable even when step is not specified on the commandline. This environment variable will be replaced by fmf config file or CLI arguments.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION

In plan metadata:

ignore-duration: true|false

On command-line:

--ignore-duration
export TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION=1|0

interactive

Run tests in interactive mode, i.e. with input and output streams shared with tmt itself. This allows input to be passed to tests via stdin, e.g. responding to password prompts. Test output in this mode is not captured, and duration has no effect.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_INTERACTIVE

In plan metadata:

interactive: true|false

On command-line:

--interactive
export TMT_PLUGIN_EXECUTE_UPGRADE_INTERACTIVE=1|0

no-progress-bar

Disable interactive progress bar showing the current test.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_NO_PROGRESS_BAR

In plan metadata:

no-progress-bar: true|false

On command-line:

--no-progress-bar
export TMT_PLUGIN_EXECUTE_UPGRADE_NO_PROGRESS_BAR=1|0

ref

Branch, tag or commit specifying the git revision.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_REF

In plan metadata:

ref: REVISION

On command-line:

--ref REVISION
export TMT_PLUGIN_EXECUTE_UPGRADE_REF=REVISION

restraint-compatible

Run tests in the restraint-compatible mode. Enable this if your tests depend on the restraint scripts such as rstrnt-report-result or rstrnt-report-log. Currently this option only affects whether the $OUTPUTFILE variable is respected, but in the future it will be used to enable/disable all restraint compatibility features.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_RESTRAINT_COMPATIBLE

In plan metadata:

restraint-compatible: true|false

On command-line:

--restraint-compatible
export TMT_PLUGIN_EXECUTE_UPGRADE_RESTRAINT_COMPATIBLE=1|0

script

Execute arbitrary shell commands and check their exit code which is used as a test result. The script field is provided to cover simple test use cases only and must not be combined with the discover step which is more suitable for more complex test scenarios.

Default shell options are applied to the script, see test for more details. The default duration for tests defined directly under the execute step is 1h. Use the duration attribute to modify the default limit.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_SCRIPT

In plan metadata:

script: SCRIPT

On command-line:

--script SCRIPT
export TMT_PLUGIN_EXECUTE_UPGRADE_SCRIPT=SCRIPT

skip-tests-after

If set, discovered tests would not run after the upgrade.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_AFTER

In plan metadata:

skip-tests-after: true|false

On command-line:

--skip-tests-after
export TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_AFTER=1|0

skip-tests-before

If set, discovered tests would not run before the upgrade.

Default: false

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_BEFORE

In plan metadata:

skip-tests-before: true|false

On command-line:

--skip-tests-before
export TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_BEFORE=1|0

test

Select tests by name.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_TEST

In plan metadata:

test: NAMES

On command-line:

--test NAMES
export TMT_PLUGIN_EXECUTE_UPGRADE_TEST=NAMES

upgrade-path

Upgrade path corresponding to a plan name in the repository with upgrade tasks.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_UPGRADE_PATH

In plan metadata:

upgrade-path: PLAN_NAME

On command-line:

--upgrade-path PLAN_NAME
export TMT_PLUGIN_EXECUTE_UPGRADE_UPGRADE_PATH=PLAN_NAME

url

URL of the git repository with upgrade tasks.

Default: not set

Environment variable: TMT_PLUGIN_EXECUTE_UPGRADE_URL

In plan metadata:

url: REPOSITORY

On command-line:

--url REPOSITORY
export TMT_PLUGIN_EXECUTE_UPGRADE_URL=REPOSITORY