Discover Plugins

Common Keys

The following keys are accepted by all plugins of the discover step.

name

The name of the step phase.

Environment variable: TMT_PLUGIN_DISCOVER_FMF_NAME

In plan metadata:

name:

On command-line:

--name ...
export TMT_PLUGIN_DISCOVER_FMF_NAME=...

order

Order in which the phase should be handled.

Default: 50

Environment variable: TMT_PLUGIN_DISCOVER_FMF_ORDER

In plan metadata:

order:

On command-line:

--order ...
export TMT_PLUGIN_DISCOVER_FMF_ORDER=...

summary

Concise summary describing purpose of the phase.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_SUMMARY

In plan metadata:

summary:

On command-line:

--summary ...
export TMT_PLUGIN_DISCOVER_FMF_SUMMARY=...

when

If specified, phase is run only if any rule matches plan context.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_WHEN

In plan metadata:

when: RULE

On command-line:

--when RULE
export TMT_PLUGIN_DISCOVER_FMF_WHEN=RULE

where

Run this phase on given guest or guests with the given role only.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_WHERE

In plan metadata:

where: GUEST|ROLE

On command-line:

--where 'GUEST|ROLE'
export TMT_PLUGIN_DISCOVER_FMF_WHERE='GUEST|ROLE'

fmf

Warning

Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal.

Discover available tests from fmf metadata.

By default all available tests from the current repository are used so the minimal configuration looks like this:

discover:
    how: fmf

Full config example:

discover:
    how: fmf
    url: https://github.com/teemtee/tmt
    ref: main
    path: /fmf/root
    test: /tests/basic
    filter: 'tier: 1'

If no ref is provided, the default branch from the origin is used.

Archive

By default url is treated as a git url to be cloned, but you can set url-content-type to archive to instead treat it as an archive url and download and extract it. For example:

discover:
    how: fmf
    url: https://github.com/teemtee/tmt/archive/refs/heads/main.tar.gz
    url-content-type: archive
    path: /tmt-main/fmf/root

Dist Git

For DistGit repo one can download sources and use code from them in the tests. Sources are extracted into $TMT_SOURCE_DIR path, patches are applied by default. See options to install build dependencies or to just download sources without applying patches. To apply patches, special prepare phase with order 60 is added, and prepare step has to be enabled for it to run.

It can be used together with ref, path and url, however ref is not possible without using url.

discover:
    how: fmf
    dist-git-source: true

Name Filter

Use the test key to limit which tests should be executed by providing regular expression matching the test name:

discover:
    how: fmf
    test: ^/tests/area/.*

tmt run discover --how fmf --verbose --test "^/tests/core.*"

When several regular expressions are provided, tests matching each regular expression are concatenated. In this way it is possible to execute a single test multiple times:

discover:
    how: fmf
    test:
      - ^/test/one$
      - ^/test/two$
      - ^/special/setup$
      - ^/test/one$
      - ^/test/two$

tmt run discover -h fmf -v -t '^/test/one$' -t '^/special/setup$' -t '^/test/two$'

The include key also allows to select tests by name, with two important distinctions from the test key:

The original test order is preserved so it does not matter in which order tests are listed under the include key.
Test duplication is not allowed, so even if a test name is repeated several times, test will be executed only once.

Finally, the exclude key can be used to specify regular expressions matching tests which should be skipped during the discovery.

The test, include and exclude keys use search mode for matching patterns. See the Regular Expressions section for detailed information about how exactly the regular expressions are handled.

Link Filter

Selecting tests containing specified link is possible using link key accepting RELATION:TARGET format of values. Regular expressions are supported for both relation and target part of the value. Relation can be omitted to target match any relation.

discover:
    how: fmf
    link: verifies:.*issue/850$

Multiple links can be provided as well:

discover:
    how: fmf
    link:
      - verifies:.*issue/850$
      - verifies:.*issue/1374$

Advanced Filter

The filter key can be used to apply an advanced filter based on test metadata attributes. These can be especially useful when tests are grouped by the tag or tier keys:

discover:
    how: fmf
    filter: tier:3 & tag:provision

tmt run discover --how fmf --filter "tier:3 & tag:provision"

See the pydoc fmf.filter documentation for more details about the supported syntax and available operators.

Modified Tests

It is also possible to limit tests only to those that have changed in git since a given revision. This can be particularly useful when testing changes to tests themselves (e.g. in a pull request CI).

Related keys: modified-only, modified-url, modified-ref

Example to compare local repo against upstream main branch:

discover:
    how: fmf
    modified-only: True
    modified-url: https://github.com/teemtee/tmt
    modified-ref: reference/main

Note that internally the modified tests are appended to the list specified via test, so those tests will also be selected even if not modified.

Adjust Tests

Use the adjust-tests key to modify the discovered tests’ metadata directly from the plan. For example, extend the test duration for slow hardware or modify the list of required packages when you do not have write access to the remote test repository. The value should follow the adjust rules syntax.

The following example adds an avc check for each discovered test, doubles its duration and replaces each occurrence of the word python3.11 in the list of required packages.

discover:
    how: fmf
    adjust-tests:
      - check+:
          - how: avc
      - duration+: '*2'
        because: Slow system under test
        when: arch == i286
      - require~:
          - '/python3.11/python3.12/'

Configuration

See also Common Keys accepted by the plugin.

adjust-tests

Modify metadata of discovered tests from the plan itself. Use the same format as for adjust rules.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_ADJUST_TESTS

In plan metadata:

adjust-tests:

On command-line:

--adjust-tests ...
export TMT_PLUGIN_DISCOVER_FMF_ADJUST_TESTS=...

dist-git-download-only

Just download the sources. No rpmbuild -bp, nor installation of require or buildddeps happens.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_DOWNLOAD_ONLY

In plan metadata:

dist-git-download-only: true|false

On command-line:

--dist-git-download-only
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_DOWNLOAD_ONLY=1|0

dist-git-extract

What to copy from extracted sources, globbing is supported. Defaults to the top fmf root if it is present, otherwise top directory (shortcut “/”).

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_EXTRACT

In plan metadata:

dist-git-extract:

On command-line:

--dist-git-extract ...
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_EXTRACT=...

dist-git-init

Set to true to initialize fmf root inside extracted sources at dist-git-extract location or top directory. To be used when the sources contain fmf files (for example tests) but do not have an associated fmf root.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_INIT

In plan metadata:

dist-git-init: true|false

On command-line:

--dist-git-init
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_INIT=1|0

dist-git-install-builddeps

Install package build dependencies according to the specfile.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_INSTALL_BUILDDEPS

In plan metadata:

dist-git-install-builddeps: true|false

On command-line:

--dist-git-install-builddeps
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_INSTALL_BUILDDEPS=1|0

dist-git-merge

Set to true to combine fmf root from the sources and fmf root from the plan. It allows to have plans and tests defined in the DistGit repo which use tests and other resources from the downloaded sources. Any plans in extracted sources will not be processed.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_MERGE

In plan metadata:

dist-git-merge: true|false

On command-line:

--dist-git-merge
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_MERGE=1|0

dist-git-remove-fmf-root

Remove fmf root from extracted source (top one or selected by copy-path, happens before dist-git-extract.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_REMOVE_FMF_ROOT

In plan metadata:

dist-git-remove-fmf-root: true|false

On command-line:

--dist-git-remove-fmf-root
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_REMOVE_FMF_ROOT=1|0

dist-git-require

Additional required package to be present before sources are prepared. The rpm-build package itself is installed automatically.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_REQUIRE

In plan metadata:

dist-git-require: PACKAGE

On command-line:

--dist-git-require PACKAGE
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_REQUIRE=PACKAGE

dist-git-source

Download DistGit sources and rpmbuild -bp them (can be skipped).

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_SOURCE

In plan metadata:

dist-git-source: true|false

On command-line:

--dist-git-source
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_SOURCE=1|0

dist-git-type

Use the provided DistGit handler instead of the auto detection. Useful when running from forked repositories.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_TYPE

In plan metadata:

dist-git-type: fedora|centos|redhat|local

On command-line:

--dist-git-type 'fedora|centos|redhat|local'
export TMT_PLUGIN_DISCOVER_FMF_DIST_GIT_TYPE='fedora|centos|redhat|local'

exclude

Exclude tests matching given regular expression. The search mode is used for pattern matching. See the Regular Expressions section for details.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_EXCLUDE

In plan metadata:

exclude: REGEXP

On command-line:

--exclude REGEXP
export TMT_PLUGIN_DISCOVER_FMF_EXCLUDE=REGEXP

filter

Apply advanced filter based on test metadata attributes. See pydoc fmf.filter for more info.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_FILTER

In plan metadata:

filter: FILTERS

On command-line:

--filter FILTERS
export TMT_PLUGIN_DISCOVER_FMF_FILTER=FILTERS

fmf-id

Only print fmf identifiers of discovered tests to the standard output and exit.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_FMF_ID

In plan metadata:

fmf-id: true|false

On command-line:

--fmf-id
export TMT_PLUGIN_DISCOVER_FMF_FMF_ID=1|0

include

Include only tests matching given regular expression. Respect the order defined in test. The search mode is used for pattern matching. See the Regular Expressions section for details.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_INCLUDE

In plan metadata:

include: REGEXP

On command-line:

--include REGEXP
export TMT_PLUGIN_DISCOVER_FMF_INCLUDE=REGEXP

link

Select tests using the link keys. Values must be in the form of RELATION:TARGET, tests containing at least one of them are selected. Regular expressions are supported for both relation and target. Relation part can be omitted to match all relations.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_LINK

In plan metadata:

link: RELATION:TARGET

On command-line:

--link RELATION:TARGET
export TMT_PLUGIN_DISCOVER_FMF_LINK=RELATION:TARGET

modified-only

Set to true if you want to filter modified tests only. The test is modified if its name starts with the name of any directory modified since modified-ref.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_MODIFIED_ONLY

In plan metadata:

modified-only: true|false

On command-line:

--modified-only
export TMT_PLUGIN_DISCOVER_FMF_MODIFIED_ONLY=1|0

modified-ref

The branch, tag or commit specifying the reference git revision (if not provided, the default branch is used). Note that you need to specify reference/<branch> to compare to a branch from the repository specified in modified-url.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_MODIFIED_REF

In plan metadata:

modified-ref: REVISION

On command-line:

--modified-ref REVISION
export TMT_PLUGIN_DISCOVER_FMF_MODIFIED_REF=REVISION

modified-url

An additional remote repository to be used as the reference for comparison. Will be fetched as a reference remote in the test dir.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_MODIFIED_URL

In plan metadata:

modified-url: REPOSITORY

On command-line:

--modified-url REPOSITORY
export TMT_PLUGIN_DISCOVER_FMF_MODIFIED_URL=REPOSITORY

path

Path to the metadata tree root. Must be relative to the git repository root if url was provided, absolute local filesystem path otherwise. By default . is used.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_PATH

In plan metadata:

path: ROOT

On command-line:

--path ROOT
export TMT_PLUGIN_DISCOVER_FMF_PATH=ROOT

prune

Copy only immediate directories of executed tests and their required files.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_PRUNE

In plan metadata:

prune: true|false

On command-line:

--prune
export TMT_PLUGIN_DISCOVER_FMF_PRUNE=1|0

ref

Branch, tag or commit specifying the desired git revision. Defaults to the remote repository’s default branch if url was set or to the current HEAD of the current repository.

Additionally, one can set ref dynamically. This is possible using a special file in tmt format stored in the default branch of a tests repository. This special file should contain rules assigning attribute ref in an adjust block, for example depending on a test run context.

Dynamic ref assignment is enabled whenever a test plan reference has the format ref: @FILEPATH.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_REF

In plan metadata:

ref: REVISION

On command-line:

--ref REVISION
export TMT_PLUGIN_DISCOVER_FMF_REF=REVISION

repository

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_REPOSITORY

In plan metadata:
repository:
On command-line:
--repository ...
export TMT_PLUGIN_DISCOVER_FMF_REPOSITORY=...

require-test

A list of test names that must be discovered during the run. If an execute step is present, these tests must also be executed. If any of the specified tests are not discovered or executed, an exception is raised.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_REQUIRE_TEST

In plan metadata:

require-test: NAMES

On command-line:

--require-test NAMES
export TMT_PLUGIN_DISCOVER_FMF_REQUIRE_TEST=NAMES

revision

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_REVISION

In plan metadata:
revision:
On command-line:
--revision ...
export TMT_PLUGIN_DISCOVER_FMF_REVISION=...

sync-repo

Force the sync of the whole git repo. By default, the repo is copied only if the used options require it.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_FMF_SYNC_REPO

In plan metadata:

sync-repo: true|false

On command-line:

--sync-repo
export TMT_PLUGIN_DISCOVER_FMF_SYNC_REPO=1|0

test

List of test names or regular expressions used to select tests by name. Duplicate test names are allowed to enable repetitive test execution, preserving the listed test order. The search mode is used for pattern matching. See the Regular Expressions section for details.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_TEST

In plan metadata:

test: NAMES

On command-line:

--test NAMES
export TMT_PLUGIN_DISCOVER_FMF_TEST=NAMES

upgrade-path

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_UPGRADE_PATH

In plan metadata:
upgrade-path:
On command-line:
--upgrade-path ...
export TMT_PLUGIN_DISCOVER_FMF_UPGRADE_PATH=...

url

External URL containing the metadata tree. Current git repository used by default. See url-content-type key for details on what content is accepted.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_FMF_URL

In plan metadata:

url: URL

On command-line:

--url URL
export TMT_PLUGIN_DISCOVER_FMF_URL=URL

url-content-type

How to handle the url key.

Default: git

Environment variable: TMT_PLUGIN_DISCOVER_FMF_URL_CONTENT_TYPE

In plan metadata:

url-content-type: git|archive

On command-line:

--url-content-type 'git|archive'
export TMT_PLUGIN_DISCOVER_FMF_URL_CONTENT_TYPE='git|archive'

shell

Warning

Please, be aware that the documentation below is a work in progress. We are working on fixing it, adding missing bits and generally making it better. Also, it was originally used for command line help only, therefore the formatting is often suboptimal.

Use provided list of shell script tests.

List of test cases to be executed can be defined manually directly in the plan as a list of dictionaries containing test name and actual test script. It is also possible to define here any other test metadata such as the duration or a path to the test. The default duration for tests defined directly in the discover step is 1h.

Example config:

discover:
    how: shell
    tests:
      - name: /help/main
        test: tmt --help
      - name: /help/test
        test: tmt test --help
      - name: /help/smoke
        test: ./smoke.sh
        path: /tests/shell

For DistGit repo one can download sources and use code from them in the tests. Sources are extracted into $TMT_SOURCE_DIR path, patches are applied by default. See options to install build dependencies or to just download sources without applying patches. To apply patches, special prepare phase with order 60 is added, and prepare step has to be enabled for it to run.

discover:
    how: shell
    dist-git-source: true
    tests:
      - name: /upstream
        test: cd $TMT_SOURCE_DIR/*/tests && make test

To clone a remote repository and use it as a source specify url. It accepts also ref to checkout provided reference. Dynamic reference feature is supported as well.

discover:
    how: shell
    url: https://github.com/teemtee/tmt.git
    ref: "1.18.0"
    tests:
      - name: first test
        test: ./script-from-the-repo.sh

Configuration

See also Common Keys accepted by the plugin.

dist-git-download-only

Just download the sources. No rpmbuild -bp, nor installation of require or buildddeps happens.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_DOWNLOAD_ONLY

In plan metadata:

dist-git-download-only: true|false

On command-line:

--dist-git-download-only
export TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_DOWNLOAD_ONLY=1|0

dist-git-install-builddeps

Install package build dependencies according to the specfile.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_INSTALL_BUILDDEPS

In plan metadata:

dist-git-install-builddeps: true|false

On command-line:

--dist-git-install-builddeps
export TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_INSTALL_BUILDDEPS=1|0

dist-git-require

Additional required package to be present before sources are prepared. The rpm-build package itself is installed automatically.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_REQUIRE

In plan metadata:

dist-git-require: PACKAGE

On command-line:

--dist-git-require PACKAGE
export TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_REQUIRE=PACKAGE

dist-git-source

Download DistGit sources and rpmbuild -bp them (can be skipped).

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_SOURCE

In plan metadata:

dist-git-source: true|false

On command-line:

--dist-git-source
export TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_SOURCE=1|0

dist-git-type

Use the provided DistGit handler instead of the auto detection. Useful when running from forked repositories.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_TYPE

In plan metadata:

dist-git-type: fedora|centos|redhat|local

On command-line:

--dist-git-type 'fedora|centos|redhat|local'
export TMT_PLUGIN_DISCOVER_SHELL_DIST_GIT_TYPE='fedora|centos|redhat|local'

keep-git-metadata

By default the .git directory is removed to save disk space. Set to true to sync the git metadata to guest as well. Implicit if dist-git-source is used.

Default: false

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_KEEP_GIT_METADATA

In plan metadata:

keep-git-metadata: true|false

On command-line:

--keep-git-metadata
export TMT_PLUGIN_DISCOVER_SHELL_KEEP_GIT_METADATA=1|0

ref

Branch, tag or commit specifying the desired git revision. Defaults to the remote repository’s default branch.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_REF

In plan metadata:

ref: REVISION

On command-line:

--ref REVISION
export TMT_PLUGIN_DISCOVER_SHELL_REF=REVISION

require-test

A list of test names that must be discovered during the run. If an execute step is present, these tests must also be executed. If any of the specified tests are not discovered or executed, an exception is raised.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_REQUIRE_TEST

In plan metadata:

require-test: NAMES

On command-line:

--require-test NAMES
export TMT_PLUGIN_DISCOVER_SHELL_REQUIRE_TEST=NAMES

tests

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_TESTS

In plan metadata:
tests:
On command-line:
--tests ...
export TMT_PLUGIN_DISCOVER_SHELL_TESTS=...

url

URL of the git repository with tests to be fetched.

Default: not set

Environment variable: TMT_PLUGIN_DISCOVER_SHELL_URL

In plan metadata:

url: REPOSITORY

On command-line:

--url REPOSITORY
export TMT_PLUGIN_DISCOVER_SHELL_URL=REPOSITORY