:tocdepth: 0 .. _/plugins/execute: Execute Plugins ~~~~~~~~~~~~~~~ .. include:: execute-header.inc.rst .. _/plugins/execute/common-keys: Common Keys ----------- The following keys are accepted by all plugins of the ``execute`` step. name The name of the step phase. Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_NAME`` In plan metadata: .. code-block:: yaml name: On command-line: .. code-block:: shell --name ... export TMT_PLUGIN_EXECUTE_TMT_NAME=... order Order in which the phase should be handled. Default: ``50`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_ORDER`` In plan metadata: .. code-block:: yaml order: On command-line: .. code-block:: shell --order ... export TMT_PLUGIN_EXECUTE_TMT_ORDER=... summary Concise summary describing purpose of the phase. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_SUMMARY`` In plan metadata: .. code-block:: yaml summary: On command-line: .. code-block:: shell --summary ... export TMT_PLUGIN_EXECUTE_TMT_SUMMARY=... when If specified, phase is run only if any rule matches plan context. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_WHEN`` In plan metadata: .. code-block:: yaml when: RULE On command-line: .. code-block:: shell --when RULE export TMT_PLUGIN_EXECUTE_TMT_WHEN=RULE where Run this phase on given guest or guests with the given role only. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_WHERE`` In plan metadata: .. code-block:: yaml where: GUEST|ROLE On command-line: .. code-block:: shell --where 'GUEST|ROLE' export TMT_PLUGIN_EXECUTE_TMT_WHERE='GUEST|ROLE' .. _/plugins/execute/tmt: tmt --- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Use the internal tmt executor to execute tests. The internal tmt executor runs tests on the guest one by one directly from the tmt code which shows testing :ref:`/stories/cli/steps/execute/progress` and supports :ref:`/stories/cli/steps/execute/interactive` debugging as well. This is the default execute step implementation. Test result is based on the script exit code (for shell tests) or the results file (for beakerlib tests). The executor provides the following shell scripts which can be used by the tests for certain operations. ``tmt-file-submit`` - archive the given file in the tmt test data directory. See the :ref:`/stories/features/report-log` section for more details. ``tmt-reboot`` - soft reboot the machine from inside the test. After reboot the execution starts from the test which rebooted the machine. Use ``tmt-reboot -s`` for systemd soft-reboot which preserves the kernel and hardware state while restarting userspace only. An environment variable ``TMT_REBOOT_COUNT`` is provided which the test can use to handle the reboot. The variable holds the number of reboots performed by the test. For more information see the :ref:`/stories/features/reboot` feature documentation. ``tmt-report-result`` - generate a result report file from inside the test. Can be called multiple times by the test. The generated report file will be overwritten if a higher hierarchical result is reported by the test. The hierarchy is as follows: SKIP, PASS, WARN, FAIL. For more information see the :ref:`/stories/features/report-result` feature documentation. ``tmt-abort`` - generate an abort file from inside the test. This will set the current test result to failed and terminate the execution of subsequent tests. For more information see the :ref:`/stories/features/abort` feature documentation. The scripts are hosted by default in the ``/usr/local/bin`` directory, except for guests using ``rpm-ostree``, where ``/var/lib/tmt/scripts`` is used. The directory can be forced using the ``TMT_SCRIPTS_DIR`` environment variable. Note that for guests using ``rpm-ostree``, the directory is added to executable paths using the system-wide ``/etc/profile.d/tmt.sh`` profile script. .. warning:: Please be aware that for guests using ``rpm-ostree`` the provided scripts will only be available in a shell that loads the profile scripts. This is the default for ``bash``-like shells, but might not work for others. Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. duration The maximum time test specified by ``script`` is allowed to run. This option does not affect the duration defined in test metadata. Default: ``1h`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_DURATION`` In plan metadata: .. code-block:: yaml duration: On command-line: .. code-block:: shell --duration ... export TMT_PLUGIN_EXECUTE_TMT_DURATION=... exit-first Stop execution after the first test failure or error. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_EXIT_FIRST`` In plan metadata: .. code-block:: yaml exit-first: true|false On command-line: .. code-block:: shell --exit-first export TMT_PLUGIN_EXECUTE_TMT_EXIT_FIRST=1|0 ignore-duration Ignore test duration value and allow test to run forever. Can be set by environment variable even when step is not specified on the commandline. This environment variable will be replaced by fmf config file or CLI arguments. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION`` In plan metadata: .. code-block:: yaml ignore-duration: true|false On command-line: .. code-block:: shell --ignore-duration export TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION=1|0 interactive Run tests in interactive mode, i.e. with input and output streams shared with tmt itself. This allows input to be passed to tests via stdin, e.g. responding to password prompts. Test output in this mode is not captured, and ``duration`` has no effect. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE`` In plan metadata: .. code-block:: yaml interactive: true|false On command-line: .. code-block:: shell --interactive export TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE=1|0 no-progress-bar Disable interactive progress bar showing the current test. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_NO_PROGRESS_BAR`` In plan metadata: .. code-block:: yaml no-progress-bar: true|false On command-line: .. code-block:: shell --no-progress-bar export TMT_PLUGIN_EXECUTE_TMT_NO_PROGRESS_BAR=1|0 restraint-compatible Run tests in the restraint-compatible mode. Enable this if your tests depend on the restraint scripts such as ``rstrnt-report-result`` or ``rstrnt-report-log``. Currently this option only affects whether the ``$OUTPUTFILE`` variable is respected, but in the future it will be used to enable/disable all restraint compatibility features. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_RESTRAINT_COMPATIBLE`` In plan metadata: .. code-block:: yaml restraint-compatible: true|false On command-line: .. code-block:: shell --restraint-compatible export TMT_PLUGIN_EXECUTE_TMT_RESTRAINT_COMPATIBLE=1|0 script Execute arbitrary shell commands and check their exit code which is used as a test result. The ``script`` field is provided to cover simple test use cases only and must not be combined with the :ref:`/spec/plans/discover` step which is more suitable for more complex test scenarios. Default shell options are applied to the script, see :ref:`/spec/tests/test` for more details. The default :ref:`/spec/tests/duration` for tests defined directly under the execute step is ``1h``. Use the ``duration`` attribute to modify the default limit. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_SCRIPT`` In plan metadata: .. code-block:: yaml script: SCRIPT On command-line: .. code-block:: shell --script SCRIPT export TMT_PLUGIN_EXECUTE_TMT_SCRIPT=SCRIPT ---- .. _/plugins/execute/upgrade: upgrade ------- .. warning:: Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal. Perform system upgrade during testing. In order to enable developing tests for upgrade testing, we need to provide a way how to execute these tests easily. This does not cover unit tests for individual actors but rather system tests which verify the whole upgrade story. The upgrade executor runs the discovered tests (using the internal executor), then performs a set of upgrade tasks from a remote repository, and finally, re-runs the tests on the upgraded guest. The ``IN_PLACE_UPGRADE`` environment variable is set during the test execution to differentiate between the stages of the test. It is set to ``old`` during the first execution and ``new`` during the second execution. Test names are prefixed with this value to make the names unique. Based on this variable, the test can perform appropriate actions. * ``old``: setup, test * ``new``: test, cleanup * ``without``: setup, test, cleanup The upgrade tasks performing the actual system upgrade are taken from a remote repository (specified by the ``url`` key) based on an upgrade path (e.g. ``fedora35to36``) or other filters (e.g. specified by the ``filter`` key). If both ``upgrade-path`` and extra filters are specified, the discover keys in the remote upgrade path plan are overridden by the filters specified in the local plan. The upgrade path must correspond to a plan name in the remote repository whose discover step selects tests (upgrade tasks) performing the upgrade. Currently, selection of upgrade tasks in the remote repository can be done using both fmf and shell discover method. If the ``url`` is not provided, upgrade path and upgrade tasks are taken from the current repository. The supported keys in discover are: * ``ref`` * ``filter`` * ``exclude`` * ``tests`` * ``test`` The environment variables defined in the remote upgrade path plan are passed to the upgrade tasks when they are executed. An example of an upgrade path plan (in the remote repository): .. code-block:: yaml discover: # Selects appropriate upgrade tasks (L1 tests) how: fmf filter: "tag:fedora" environment: # This is passed to upgrade tasks SOURCE: 35 TARGET: 36 execute: how: tmt If no upgrade path is specified in the plan, the tests (upgrade tasks) are selected based on the configuration of the upgrade plugin (e.g. based on the filter in its configuration). If these two possible ways of specifying upgrade tasks are combined, the remote discover plan is used but its options are overridden with the values specified locally. The same options and config keys and values can be used as in the internal executor. Minimal execute config example with an upgrade path: .. code-block:: yaml execute: how: upgrade url: https://github.com/teemtee/upgrade upgrade-path: /paths/fedora35to36 Execute config example without an upgrade path: .. code-block:: yaml execute: how: upgrade url: https://github.com/teemtee/upgrade filter: "tag:fedora" .. code-block:: yaml # A simple beakerlib test using the $IN_PLACE_UPGRADE variable . /usr/share/beakerlib/beakerlib.sh || exit 1 VENV_PATH=/var/tmp/venv_test rlJournalStart # Perform the setup only for the old distro if [[ "$IN_PLACE_UPGRADE" != "new" ]]; then rlPhaseStartSetup rlRun "python3.9 -m venv $VENV_PATH" rlRun "$VENV_PATH/bin/pip install pyjokes" rlPhaseEnd fi # Execute the test for both old & new distro rlPhaseStartTest rlAsssertExists "$VENV_PATH/bin/pyjoke" rlRun "$VENV_PATH/bin/pyjoke" rlPhaseEnd # Skip the cleanup phase when on the old distro if [[ "$IN_PLACE_UPGRADE" != "old" ]]; then rlPhaseStartCleanup rlRun "rm -rf $VENV_PATH" rlPhaseEnd fi rlJournalEnd Configuration ^^^^^^^^^^^^^ See also :ref:`Common Keys` accepted by the plugin. duration The maximum time test specified by ``script`` is allowed to run. This option does not affect the duration defined in test metadata. Default: ``1h`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_DURATION`` In plan metadata: .. code-block:: yaml duration: On command-line: .. code-block:: shell --duration ... export TMT_PLUGIN_EXECUTE_UPGRADE_DURATION=... exclude Exclude a regular expression from search result. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_EXCLUDE`` In plan metadata: .. code-block:: yaml exclude: REGEXP On command-line: .. code-block:: shell --exclude REGEXP export TMT_PLUGIN_EXECUTE_UPGRADE_EXCLUDE=REGEXP exit-first Stop execution after the first test failure or error. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_EXIT_FIRST`` In plan metadata: .. code-block:: yaml exit-first: true|false On command-line: .. code-block:: shell --exit-first export TMT_PLUGIN_EXECUTE_UPGRADE_EXIT_FIRST=1|0 filter Include only tests matching the filter. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_FILTER`` In plan metadata: .. code-block:: yaml filter: FILTERS On command-line: .. code-block:: shell --filter FILTERS export TMT_PLUGIN_EXECUTE_UPGRADE_FILTER=FILTERS ignore-duration Ignore test duration value and allow test to run forever. Can be set by environment variable even when step is not specified on the commandline. This environment variable will be replaced by fmf config file or CLI arguments. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION`` In plan metadata: .. code-block:: yaml ignore-duration: true|false On command-line: .. code-block:: shell --ignore-duration export TMT_PLUGIN_EXECUTE_TMT_IGNORE_DURATION=1|0 interactive Run tests in interactive mode, i.e. with input and output streams shared with tmt itself. This allows input to be passed to tests via stdin, e.g. responding to password prompts. Test output in this mode is not captured, and ``duration`` has no effect. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_INTERACTIVE`` In plan metadata: .. code-block:: yaml interactive: true|false On command-line: .. code-block:: shell --interactive export TMT_PLUGIN_EXECUTE_UPGRADE_INTERACTIVE=1|0 no-progress-bar Disable interactive progress bar showing the current test. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_NO_PROGRESS_BAR`` In plan metadata: .. code-block:: yaml no-progress-bar: true|false On command-line: .. code-block:: shell --no-progress-bar export TMT_PLUGIN_EXECUTE_UPGRADE_NO_PROGRESS_BAR=1|0 ref Branch, tag or commit specifying the git revision. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_REF`` In plan metadata: .. code-block:: yaml ref: REVISION On command-line: .. code-block:: shell --ref REVISION export TMT_PLUGIN_EXECUTE_UPGRADE_REF=REVISION restraint-compatible Run tests in the restraint-compatible mode. Enable this if your tests depend on the restraint scripts such as ``rstrnt-report-result`` or ``rstrnt-report-log``. Currently this option only affects whether the ``$OUTPUTFILE`` variable is respected, but in the future it will be used to enable/disable all restraint compatibility features. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_RESTRAINT_COMPATIBLE`` In plan metadata: .. code-block:: yaml restraint-compatible: true|false On command-line: .. code-block:: shell --restraint-compatible export TMT_PLUGIN_EXECUTE_UPGRADE_RESTRAINT_COMPATIBLE=1|0 script Execute arbitrary shell commands and check their exit code which is used as a test result. The ``script`` field is provided to cover simple test use cases only and must not be combined with the :ref:`/spec/plans/discover` step which is more suitable for more complex test scenarios. Default shell options are applied to the script, see :ref:`/spec/tests/test` for more details. The default :ref:`/spec/tests/duration` for tests defined directly under the execute step is ``1h``. Use the ``duration`` attribute to modify the default limit. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_SCRIPT`` In plan metadata: .. code-block:: yaml script: SCRIPT On command-line: .. code-block:: shell --script SCRIPT export TMT_PLUGIN_EXECUTE_UPGRADE_SCRIPT=SCRIPT skip-tests-after If set, discovered tests would not run *after* the upgrade. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_AFTER`` In plan metadata: .. code-block:: yaml skip-tests-after: true|false On command-line: .. code-block:: shell --skip-tests-after export TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_AFTER=1|0 skip-tests-before If set, discovered tests would not run *before* the upgrade. Default: ``false`` Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_BEFORE`` In plan metadata: .. code-block:: yaml skip-tests-before: true|false On command-line: .. code-block:: shell --skip-tests-before export TMT_PLUGIN_EXECUTE_UPGRADE_SKIP_TESTS_BEFORE=1|0 test Select tests by name. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_TEST`` In plan metadata: .. code-block:: yaml test: NAMES On command-line: .. code-block:: shell --test NAMES export TMT_PLUGIN_EXECUTE_UPGRADE_TEST=NAMES upgrade-path Upgrade path corresponding to a plan name in the repository with upgrade tasks. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_UPGRADE_PATH`` In plan metadata: .. code-block:: yaml upgrade-path: PLAN_NAME On command-line: .. code-block:: shell --upgrade-path PLAN_NAME export TMT_PLUGIN_EXECUTE_UPGRADE_UPGRADE_PATH=PLAN_NAME url URL of the git repository with upgrade tasks. Default: *not set* Environment variable: ``TMT_PLUGIN_EXECUTE_UPGRADE_URL`` In plan metadata: .. code-block:: yaml url: REPOSITORY On command-line: .. code-block:: shell --url REPOSITORY export TMT_PLUGIN_EXECUTE_UPGRADE_URL=REPOSITORY