Plans
As a tester I want to easily execute selected tests or selected test steps in given environment.
Plans, also called L2 metadata, are used to group relevant tests and enable the CI. They describe how to discover tests for execution, how to provision the environment, how to prepare it for testing, how to execute tests, report results and finally how to finish the test job.
Each of the six steps mentioned above supports multiple implementations. The default methods are listed below.
Thanks to clearly separated test steps it is possible to run
only selected steps, for example tmt run discover to see
which tests would be executed.
In addition to the attributes defined here, plans also support common Core attributes which are shared across all metadata levels.
Examples:
# Enabled a minimal smoke test
execute:
script: foo --version
# Run tier one tests in a container
discover:
how: fmf
filter: tier:1
provision:
how: container
execute:
how: tmt
# Verify that apache can serve pages
summary: Basic httpd smoke test
provision:
how: virtual
memory: 4096
prepare:
- name: packages
how: install
package: [httpd, curl]
- name: service
how: shell
script: systemctl start httpd
execute:
how: tmt
script:
- echo foo > /var/www/html/index.html
- curl http://localhost/ | grep foo
ansible
Define general Ansible settings for a plan
As a user, I want to configure Ansible-related settings for the plan, for example, the ability to define the structure of the Ansible inventory using a customizable layout template.
When running tests, a static inventory file is generated from
provided metadata. The ansible.inventory.layout key allows
users to specify a YAML template file defining the desired inventory group
hierarchy and layout.
This layout will be used to organize provisioned guests under Ansible groups accordingly. If no layout is provided, a simple default basic layout is used.
Note
Ansible execution with ansible inventory is only supported for SSH-based provisioning plugins (e.g. virtual.testcloud). It is not implemented (yet) for container and local provisioning plugins.
Added in version 1.60.
Examples:
# sample-plan.fmf (input fmf file)
ansible:
inventory:
layout: path/to/the/layout.yaml
provision:
- name: test-master
role: master # Ansible group can be specified with role key.
- name: test-runner
role: runner
- name: test-client
role: client
- name: sidecar-vm # if no group is specified, the guest is added to 'ungrouped'
execute:
how: tmt
# path/to/the/layout.yaml (input inventory layout, relative to fmf root)
all:
children:
linux:
children:
server:
children:
master: {}
replica: {}
client: {}
runner: {}
# Output inventory file
all:
children:
linux:
children:
server:
children:
master:
hosts:
test-master: {}
# An empty group is created if no matching guests are found
replica: {}
runner:
hosts:
test-runner: {}
client:
hosts:
test-client: {}
ungrouped:
hosts:
sidecar-vm: {}
hosts:
test-master:
ansible_host: 127.0.0.1
ansible_user: root
ansible_port: 10028
test-client:
ansible_host: 127.0.0.1
ansible_user: root
ansible_port: 10031
test-runner:
ansible_host: 127.0.0.1
ansible_user: root
ansible_port: 10033
sidecar-vm:
ansible_host: 127.0.0.1
ansible_user: root
ansible_port: 10034
Status: implemented
Implemented by /tmt/ansible.py
cleanup
Cleanup tasks
Final cleanup actions to be performed after the test execution has been completed. Includes stopping and removing guests and workdir pruning. For more information about available options see the Cleanup Plugins section.
Added in version 1.54.
Examples:
cleanup:
how: tmt
Status: implemented and verified
Implemented by /tmt/steps/cleanup
Verified by /tests/cleanup/guest
Verified by /tests/cleanup/prune
context
Definition of the test execution context
As a user I want to define the default context in which tests included in given plan should be executed.
Define the default context values for all tests executed under
the given plan. Can be overridden by context provided directly
on the command line. See the Context definition
for the full list of supported context dimensions. Must be a
dictionary.
Note
The context dimensions which are defined in the plan are
not evaluated when the adjust rules are
applied to the plan itself. Use the command line option
--context to provide the desired context instead.
Examples:
summary:
Test basic functionality of the httpd24 collection
discover:
how: fmf
execute:
how: tmt
context:
collection: httpd24
summary:
Verify dash against the shared shell tests repository
discover:
how: fmf
url: https://src.fedoraproject.org/tests/shell
execute:
how: tmt
context:
component: dash
Status: implemented
Implemented by /tmt/base.py
discover
Discover tests relevant for execution
Gather information about tests which are supposed to be run.
Provide method tests() returning a list of discovered
tests and requires() returning a list of all required
packages aggregated from the require attribute of the
individual test metadata.
Store the list of aggregated tests with their corresponding
metadata in the tests.yaml file. The format must be a
list of dictionaries structured in the following way:
- name: /test/one
summary: Short test summary.
description: Long test description.
contact: Petr Šplíchal <psplicha@redhat.com>
component: [tmt]
test: tmt --help
path: /test/path/
require: [package1, package2]
environment:
key1: value1
key2: value2
key3: value3
duration: 5m
enabled: true
result: respect
tag: [tag]
tier: 1
serial-number: 1
- name: /test/two
summary: Short test summary.
description: Long test description.
...
Examples:
# Enabled a minimal smoke test
execute:
script: foo --version
# Run tier one tests in a container
discover:
how: fmf
filter: tier:1
provision:
how: container
execute:
how: tmt
# Verify that apache can serve pages
summary: Basic httpd smoke test
provision:
how: virtual
memory: 4096
prepare:
- name: packages
how: install
package: [httpd, curl]
- name: service
how: shell
script: systemctl start httpd
execute:
how: tmt
script:
- echo foo > /var/www/html/index.html
- curl http://localhost/ | grep foo
dist-git-source
Download rpm sources for dist-git repositories
Downloads the source files specified in the sources
file of a DistGit (Fedora, CentOS) repository. Plan using
the option has to be defined in a DistGit repository or
the url option needs to point to the root of such
repository.
All source files are available for further use in the
directory provided in the TMT_SOURCE_DIR variable.
Patches are applied by rpm-build -bp command which
runs in prepare step on the provisioned guest, with
order 60. All created files and directories by this command
are directly in TMT_SOURCE_DIR.
To run the plan without uploading source files to the lookaside
the type local can be used. Sources have to be listed in the source
file and present in the same directory as the spec file.
The fmf plugin supports additional dist-git options, see its documentation for details.
Note
In order to discover which tests would be executed, without actually running them, it is necessary to enable the provision and prepare steps as well:
tmt run -v discover provision prepare finish
Examples:
# Download & extract sources from another repo, print
# single file as a test
discover:
how: shell
url: https://src.fedoraproject.org/rpms/tmt
dist-git-source: true
tests:
- name: /print/pyproject
test: cat $TMT_SOURCE_DIR/tmt-*/pyproject.toml
# Just download sources, test is responsible for rpmbuild
# and running tests
discover:
how: shell
dist-git-source: true
dist-git-download-only: true
tests:
- name: /unit
test: >
rpmbuild -bp
--define "_sourcedir $TMT_SOURCE_DIR"
--define "_builddir $TMT_SOURCE_DIR/BUILD"
$TMT_SOURCE_DIR/*.spec &&
cd $TMT_SOURCE_DIR/BUILD/* &&
make test
require:
- rpm-build
# Source file wasn't uploaded to look aside yet, but exists next
# to the spec file.
discover:
how: shell
dist-git-source: true
dist-git-type: local
Status: implemented
Implemented by discover/fmf
Implemented by discover/shell
when
Conditional step configuration
Using the when key makes it easier to restrict a step configuration
to run only if any of the specified rules matches.
The syntax is the same as in adjust and Context.
Values can be single string with the rule or list of rules.
Examples:
discover:
- name: Private tests, applicable only for RHEL
when: distro == rhel
how: fmf
url: https://secret
- name: Public tests
how: fmf
url: https://public
- name: Just a demo of more rules in the 'when' key
how: shell
script: ./something.sh
when:
- initiator == konflux && distro == fedora
- initiator == human && distro == fedora
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/steps/when
Documented by Guide
where
Execute tests on selected guests
In the multihost scenarios it
is often necessary to execute test code on selected guests
only or execute different test code on individual guests.
The where key allows to select guests where the tests
should be executed by providing their name or the
role they play in the scenario. Use a list to specify
multiple names or roles. By default, when the where
key is not defined, tests are executed on all provisioned
guests.
Examples:
# Run a different script for each guest or role
discover:
- how: shell
where: client
tests:
- name: run-the-client-code
test: client.py
- how: shell
where: server
tests:
- name: run-the-server-code
test: server.py
# Filter different sets of tests for each guest or role
discover:
- how: fmf
filter: tag:client-tests
where: client
- how: fmf
filter: tag:server-tests
where: server
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/multihost/complete
Verified by /tests/multihost/web
Documented by Guide
environment
Environment variables
Specifies environment variables available in all steps. Plugins need to include these environment variables while running commands or other programs. These environment variables override test environment if present. Command line option --environment can be used to override environment variables defined in both tests and plan. Use the environment-file key to load variables from files. The environment+ notation can be used to extend environment defined in the parent plan, see also the Inherit Plans section for more examples.
Examples:
# Environment variables defined in a plan
environment:
KOJI_TASK_ID: 42890031
RELEASE: f33
execute:
script: echo "Testing $KOJI_TASK_ID on release $RELEASE"
# Share common variables across plans using inheritance
/plans:
environment:
COMMON: This is a common variable content
/mini:
environment+:
VARIANT: mini
/full:
environment+:
VARIANT: full
# Variables from the command line
tmt run --environment X=1 --environment Y=2
tmt run --environment "X=1 Y=2"
# Make sure to quote properly values which include spaces
tmt run --environment "BUGS='123 456 789'"
Status: implemented and verified
Implemented by /tmt/base.py
Implemented by /tmt/steps/discover
Verified by /tests/core/environment
environment-file
Environment variables from files
In addition to the environment key it is also possible to provide environment variables in a file. Supported formats are dotenv/shell with KEY=VALUE pairs and yaml. Full url can be used to fetch variables from a remote source. The environment key has a higher priority. File path must be relative to the metadata tree root.
Examples:
# Load from a dotenv/shell format
/plan:
environment-file:
- env
# Load from a yaml format
/plan:
environment-file:
- environment.yml
- environment.yaml
# Fetch from remote source
/plan:
environment-file:
- https://example.org/project/environment.yaml
Status: implemented and verified
Implemented by /tmt/base.py
Verified by /tests/core/environment-file
execute
Define how tests should be executed
Execute discovered tests in the provisioned environment using
selected test executor. By default tests are executed using
the internal tmt executor which allows to show detailed
progress of the testing and supports interactive debugging.
This is a required attribute. Each plan has to define this step.
For each test, a separate directory is created for storing
artifacts related to the test execution. Its path is
constructed from the test name and it’s stored under the
execute/data directory. It contains a metadata.yaml
file with the aggregated L1 metadata which can be used by the
test framework. In addition to supported
Tests attributes it also contains fmf name of
the test.
In each plan, the execute step must produce a results.yaml file
with results for executed tests. The format of the file is described
at Results.
Examples:
# Enabled a minimal smoke test
execute:
script: foo --version
# Run tier one tests in a container
discover:
how: fmf
filter: tier:1
provision:
how: container
execute:
how: tmt
# Verify that apache can serve pages
summary: Basic httpd smoke test
provision:
how: virtual
memory: 4096
prepare:
- name: packages
how: install
package: [httpd, curl]
- name: service
how: shell
script: systemctl start httpd
execute:
how: tmt
script:
- echo foo > /var/www/html/index.html
- curl http://localhost/ | grep foo
exit-first
Stop execution after the first test failure or error
As a user I want to avoid waiting for all discovered tests to finish if one of them fails.
Optional boolean attribute exit-first can be used to make the executor stop executing tests once a test failure or error is encountered.
Examples:
execute:
how: tmt
exit-first: true
Status: implemented and verified
Implemented by execute/tmt
Verified by /tests/execute/exit-first
isolate
Run tests in an isolated environment
Note
This is a draft, the story is not implemented yet.
Optional boolean attribute isolate can be used to request a clean test environment for each test.
Examples:
execute:
how: tmt
isolate: true
Status: idea
script
Execute shell scripts
As a user I want to easily run shell script as a test.
Examples:
# Run a simple smoke test
execute:
how: tmt
script: tmt --help
# Modify the default maximum duration
execute:
how: tmt
script: a-long-test-suite
duration: 3h
# Run a script file, note that files are expected to be executable
execute:
script: ./run_tests.sh
Implemented by execute/tmt
Multi-line script
Multi-line shell script
Providing a multi-line shell script is also supported. Note that the first command with non-zero exit code will finish the execution. See the test key for details about default shell options.
Examples:
execute:
script: |
dnf -y install httpd curl
systemctl start httpd
echo foo > /var/www/html/index.html
curl http://localhost/ | grep foo
Status: implemented
Implemented by execute/tmt
Multiple commands
Multiple shell commands
You can also include several commands as a list. Executor will run commands one-by-one and check exit code of each.
Examples:
execute:
script:
- dnf -y install httpd curl
- systemctl start httpd
- echo foo > /var/www/html/index.html
- curl http://localhost/ | grep foo
Status: implemented
Implemented by execute/tmt
The simplest usage
Simple use case should be super simple to write
As the how keyword can be omitted when using the default executor you can just define the shell script to be run. This is how a minimal smoke test configuration for the tmt command can look like:
Examples:
execute:
script: tmt --help
Status: implemented
Implemented by execute/tmt
finish
Finishing tasks
Additional actions to be performed after the test execution has been completed. Counterpart of the prepare step, useful for various cleanup actions. For more information about available options see the Finish Plugins section.
Examples:
finish:
- how: shell
script: upload-logs.sh
- how: ansible
playbook: playbooks/cleanup.yaml
Implemented by /tmt/steps/finish
when
Conditional step configuration
Using the when key makes it easier to restrict a step configuration
to run only if any of the specified rules matches.
The syntax is the same as in adjust and Context.
Values can be single string with the rule or list of rules.
Examples:
finish:
- name: finish config to run only on Fedora
when: distro == fedora
how: shell
script: ./fedora_specific.
- name: Runs always
how: shell
script: ./setup.sh
- name: Just a demo of more rules in the 'when' key
how: shell
script: ./something.sh
when:
- initiator == konflux && distro == fedora
- initiator == human && distro == fedora
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/steps/when
Documented by Guide
where
Perform finishing tasks on selected guests
In the multihost scenarios it
is often necessary to perform different completion tasks
on individual guests. The where key allows to select
guests where the finishing tasks should be applied by
providing their name or the role they play in the
scenario. Use a list to specify multiple names or roles.
By default, when the where key is not defined,
finishing tasks are performed on all provisioned guests.
Examples:
# Stop Apache on the server
prepare:
- how: shell
script: systemctl stop httpd
where: server
Status: implemented and documented
Implemented by /tmt/steps
Documented by Guide
prepare
Prepare the environment for testing
The prepare step is used to define how the guest
environment should be prepared so that the tests can be
successfully executed.
The install plugin provides an easy way to install required or recommended packages from disk and from the official distribution or copr repositories. Use the ansible plugin for applying custom playbooks or execute shell scripts to perform arbitrary preparation tasks.
See the Prepare Plugins section for the full list of available prepare plugins and their supported options.
Use the order attribute to select in which order the
preparation should happen if there are multiple configs.
The following are predefined order values of various
preparations by tmt:
- 30
Installation of essential plugin and check requirements.
- 50
The default order of any object.
- 70
Installation of packages required by tests.
- 75
Installation of packages recommended by tests.
Note
Individual plugins may define their own special
ordervalues, and you shall find the relevant information in Plugins documentation.
Note
If you want to use the prepare step to generate data
files needed for testing during the execute step,
move or copy them into ${TMT_PLAN_DATA} directory. Only
files in this directory are guaranteed to be preserved.
Examples:
# Install fresh packages from a custom copr repository
prepare:
- how: install
copr: psss/tmt
package: tmt+all
# Install required packages and start the service
prepare:
- name: packages
how: install
package: [httpd, curl]
- name: service
how: shell
script: systemctl start httpd
Implemented by /tmt/steps/prepare
Verified by /tests/prepare/adjust
Verified by /tests/prepare/ansible
Verified by /tests/prepare/artifact
Verified by /tests/prepare/basic
Verified by /tests/prepare/feature/basic
Verified by /tests/prepare/feature/crb
Verified by /tests/prepare/feature/epel
Verified by /tests/prepare/feature/fips
Verified by /tests/prepare/freeze
Verified by /tests/prepare/install
Verified by /tests/prepare/multihost
Verified by /tests/prepare/recommend
Verified by /tests/prepare/require
Verified by /tests/prepare/shell
when
Conditional step configuration
Using the when key makes it easier to restrict a step configuration
to run only if any of the specified rules matches.
The syntax is the same as in adjust and Context.
Values can be single string with the rule or list of rules.
Examples:
prepare:
- name: Prepare config to run only on Fedora
when: distro == fedora
how: shell
script: ./fedora_specific.
- name: Runs always
how: shell
script: ./setup.sh
- name: Just a demo of more rules in the 'when' key
how: shell
script: ./something.sh
when:
- initiator == konflux && distro == fedora
- initiator == human && distro == fedora
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/steps/when
Documented by Guide
where
Apply preparation on selected guests
In the multihost scenarios it
is often necessary to perform different preparation tasks on
individual guests. The where key allows to select guests
where the preparation should be applied by providing their
name or the role they play in the scenario. Use a
list to specify multiple names or roles. By default, when
the where key is not defined, preparation is done on all
provisioned guests.
Examples:
# Start Apache on the server
prepare:
- how: shell
script: systemctl start httpd
where: server
# Apply common setup on the primary server and all replicas
prepare:
- how: ansible
playbook: common.yaml
where: [primary, replica]
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/multihost/complete
Verified by /tests/multihost/web
Verified by /tests/multihost/corner-cases
Documented by Guide
provision
Provision a system for testing
See the Provision Plugins documentation for details about supported options.
As part of the provision step it is also possible to specify detailed hardware requirements for the testing environment. See the Hardware specification section for details.
As part of the provision step it is also possible to specify kickstart file used during the installation. See the kickstart specification section for details.
Examples:
# Provision a local virtual machine with the latest Fedora
provision:
how: virtual
image: fedora
Implemented by /tmt/steps/provision
ansible
Per-guest Ansible customization
As a user I want to assign an Ansible group and host variables for a provisioned guest to be reflected in the generated inventory.
Each provisioned guest may define an optional ansible key
to define its corresponding Ansible inventory section.
- group
Assigns the guest to an Ansible specific group. If omitted, the role key would serve as the Ansible group name. If neither
ansible.groupnorroleis set, the host will be placed in the ungrouped group.- vars
Defines host-specific Ansible variables to include under that host in the inventory.
Added in version 1.60.
Examples:
# Plan file
provision:
# If both role and ansible.group are set,
# ansible.group takes precedence
- name: master-guest
role: master
ansible:
group: server
vars:
var1: value1
var2: value2
- name: client-guest
role: client
ansible:
vars:
ansible_user: fedora
ansible_port: 2222
# If a specified group doesn't exist in the layout,
# it will be created automatically in the output inventory.
- name: database-guest
role: database
# Orphan guest without role or ansible.group,
# will be placed in `ungrouped`
- name: orphan-guest
# Generated inventory file
all:
children:
server:
hosts:
master-guest: {}
client:
hosts:
client-guest: {}
database:
hosts:
database-guest: {}
ungrouped:
hosts:
orphan-guest: {}
hosts:
master-guest:
var1: value1
var2: value2
client-guest:
ansible_user: fedora
ansible_port: 2222
database-guest: {}
orphan-guest: {}
Status: implemented
Implemented by /tmt/steps/provision
kickstart
As a tester I want to specify detailed installation of a guest using the kickstart script.
As part of the provision step it is possible to
use the kickstart key to specify additional requirements for the
installation of a guest. It is possible to specify a kickstart
script that will for example specify specific partitioning.
The structure of a kickstart file is separated into several sections.
- pre-install
Corresponds to the
%presection of a file. It can containbashcommands, this part is run before the installation of a guest.- post-install
Corresponds to the
%postsection of a file. It can containbashcommands, this part is run after the installation of a guest.- script
Contains the kickstart specific commands that are run during the installation of a guest.
It is also possible to specify metadata. This part may be
interpreted differently for each of the pools that the guest is
created from. For example, in Beaker this section can be used to
modify the default kickstart template used by Beaker. Similarly
works the kernel-options and kernel-options-post. Kernel
options are passed on the kernel command line when the installer is
booted. Post-install kernel options are set in the boot loader
configuration, to be passed on the kernel command line after
installation.
Note
The implementation for the kickstart key is in progress.
Support of a kickstart file is currently limited to Beaker
provisioning, as implemented by tmt’s beaker and artemis
plugins, and may not be fully supported by other provisioning
plugins in the future. Check individual plugin documentation
for additional information on the kickstart support.
Examples:
# Use the artemis plugin to provision a guest from Beaker.
# The following `kickstart` specification will be run
# during the guest installation.
provision:
how: artemis
pool: beaker
image: rhel-7
kickstart:
pre-install: |
%pre --log=/dev/console
disk=$(lsblk | grep disk | awk '{print $1}')
echo $disk
%end
script: |
lang en_US.UTF-8
zerombr
clearpart --all --initlabel
part /boot --fstype="xfs" --size=200
part swap --fstype="swap" --size=4096
part / --fstype="xfs" --size=10000 --grow
post-install: |
%post
systemctl disable firewalld
%end
metadata: "no_autopart harness=restraint"
kernel-options: "ksdevice=eth1"
kernel-options-post: "quiet"
Status: implemented
Implemented by provision/beaker (since 1.37)
Implemented by provision/artemis (since 1.22)
multihost
Multihost testing specification
Changed in version 1.24.
As a part of the provision step it is possible to request
multiple guests to be provisioned for testing. Each guest
has to be assigned a unique name which is used to
identify it.
The optional parameter role can be used to mark
related guests so that common actions can be applied to
all such guests at once. An example role name can be
client or server but arbitrary identifier can be used.
Both name and role can be used together with the
where key to select guests on which the
preparation
tasks should be applied or where the test
execution should
take place.
See Guest Topology Format for details on how
this information is exposed to tests and prepare and
finish tasks.
Examples:
# Request two guests
provision:
- name: server
how: virtual
- name: client
how: virtual
# Assign role to guests
provision:
- name: main-server
role: primary
- name: backup-one
role: replica
- name: backup-two
role: replica
- name: tester-one
role: client
- name: tester-two
role: client
Status: implemented and verified
Implemented by /tmt/steps/__init__.py
Implemented by /tmt/queue.py
Verified by /tests/multihost/complete
Verified by /tests/multihost/container-network
Verified by /tests/multihost/corner-cases
Verified by /tests/multihost/provision
Verified by /tests/multihost/web
report
Report test results
Report test results according to user preferences. For more information about the supported report methods, check the Report Plugins documentation.
Implemented by /tmt/steps/report
when
Conditional step configuration
Using the when key makes it easier to restrict a step configuration
to run only if any of the specified rules matches.
The syntax is the same as in adjust and Context.
Values can be single string with the rule or list of rules.
Examples:
report:
- name: Open html report
when:
- trigger is not defined
- initiator == human
how: html
open: true
- how: display
Status: implemented, verified and documented
Implemented by /tmt/steps
Verified by /tests/steps/when
Documented by Guide
source-script
As a tester I want to source a file at the beginning of every test.
tmt will bash source the file exposed at
TMT_PLAN_SOURCE_SCRIPT at the beginning of every
prepare shell,
finish shell
phases and every test.
Initially, this file is empty, but you can write to
this file some content which will be executed in
the subsequent steps that use this file.
Unlike TMT_PLAN_ENVIRONMENT_FILE, this feature
is not available for arbitrary phases. Only
prepare shell,
finish shell
and tests use this file, for now.
Examples:
prepare:
name: Populate the contents of TMT_PLAN_SOURCE_SCRIPT
how: shell
script: |
cat >> $TMT_PLAN_SOURCE_SCRIPT <<EOF
source /etc/profile.d/modules.sh
module load mpi/openmpi-$(arch)
EOF
execute:
how: tmt
script: |
module list
mpirun -n 2 /usr/bin/hostname
Status: implemented and verified
Implemented by /tmt/base.py
Verified by /tests/core/source-script
summary
Concise summary describing the plan
Should shortly describe purpose of the test plan. Must be a one-line string, should be up to 50 characters long. It is challenging to be both concise and descriptive, but that is what a well-written summary should do.
Examples:
/pull-request:
/pep:
summary: All code must comply with the PEP8 style guide
/lint:
summary: Run pylint to catch common problems (no gating)
/build:
/smoke:
summary: Basic smoke test (Tier1)
/features:
summary: Verify important features
Status: implemented
Implemented by /tmt/base.py
Import Plans
Importing plans from a remote repository
As a user I want to reference a plan from a remote repository in order to prevent duplication and minimize maintenance.
In some cases the configuration stored in a plan can be quite large, for example the prepare step can define complex scripts to set up the guest for testing. Using a reference to a remote plan makes it possible to reuse the same config on multiple places without the need to duplicate the information. This can be useful for example when enabling integration testing between related components.
Remote plans are identified by the plan key which must
contain an import dictionary with an fmf identifier of
the remote plan:
/some-plan:
plan:
import:
url: ...
name: ...
ref: ...
path: ...
The url and name keys are required, ref and path
are optional. If ref is set, the given git ref will be
checked out before looking for plans; path defines where in
the remote repository the metadata tree lives.
By default, only a single remote plan is allowed to be imported,
and it always replaces the importing plan. To import multiple plans,
set scope key to all-plans:
/some-plan:
plan:
import:
...
# This is the default, only the first discovered plan is imported,
# the rest is ignored.
scope: first-plan-only
# To limit the import to one and one plan only, and fail if
# more plans would match the criteria:
scope: single-plan-only
# To allow import of multiple plans:
scope: all-plans
If, instead of replacing the current plan, you want to make the imported
plans “children” of the current plan with the importing
key:
/some-plan:
plan:
import:
...
# This is the default, replace the current plan.
importing: replace
# All imported plans will get their names to begin with `/some-plan/...`.
importing: become-parent
The name key is treated as a
regular expression, and only plans with
matching names will be imported.
Note
Regular expression accepted by the name key is applied in the
“match” mode, i.e. it must match from the start of the string.
Additionally, one can utilize dynamic ref assignment when
importing a plan in order to avoid hardcoding ref value in
the importing plan. See the Dynamic ref Evaluation section for
usage details and examples.
Plan steps must not be defined in the remote plan reference.
The imported plan can be modified in only two ways. First way
is via environment and context variables. Imported plan inherits
all environment and context variables from the importing plan.
This behavior can be changed by setting the inherit-context
or inherit-environment options:
/some-plan:
context:
...
environment:
...
plan:
import:
...
inherit-context: false
inherit-environment: false
Note
By default, the imported plan inherits the context and environment
variables from the importing plan. This behavior is controlled by
the inherit-context and inherit-environment options, which
are set to true by default. In case of variable conflicts,
values from the importing plan take precedence over those in the
imported plan. Command line variables are always inherited,
regardless of the options, and have the highest priority.
Added in version 1.51.
The imported plan can also be altered using the enabled key.
If the local plan is enabled, it will follow the status of the
remote plan – whether it’s enabled or disabled. If the local
plan is disabled, the remote plan will also be disabled.
Adjust rules are respected during this process.
Added in version 1.19.
Examples:
# Minimal reference is using 'url' and 'name'
plan:
import:
url: https://github.com/teemtee/tmt
name: /plans/features/basic
# A 'ref' can be used to select specific branch or commit
plan:
import:
url: https://github.com/teemtee/tmt
name: /plans/features/basic
ref: fedora
# Use 'path' when fmf tree is deeper in the git repository
plan:
import:
url: https://github.com/teemtee/tmt
path: /examples/httpd
name: /smoke
# Use 'name' as a regular expression to import multiple plans
plan:
import:
url: https://github.com/teemtee/tmt
name: /plans/provision/(connect|local)
Status: implemented and verified
Relates to https://github.com/teemtee/tmt/issues/975
Verified by /tests/plan/import/basic
Verified by /tests/plan/import/errors
Verified by /tests/plan/import/inheritance
Verified by /tests/plan/import/local-and-remote
Verified by /tests/plan/import/modify
Verified by /tests/plan/import/plan-of-plans
Implemented by /tmt/base.py
Guest Topology Format
Define format of on-disk description of provisioned guest topology
The following text defines structure of files tmt uses for exposing guest names, roles and other properties to tests and steps that run on a guest (prepare, execute, finish). tmt saves these files on every guest used, and exposes their paths to processes started by tmt on these guests through environment variables:
TMT_TOPOLOGY_YAMLfor a YAML fileTMT_TOPOLOGY_BASHfor a shell-friendlyNAME=VALUEfile
Both files are always available, and both carry the same information.
Warning
The shell-friendly file contains arrays, therefore it’s compatible with Bash 4.x and newer.
Note
The shell-friendly file is easy to ingest for a shell-based tests,
it can be simply source-ed. For parsing the YAML file in shell,
pure shell parsers like https://github.com/sopos/yash can be used.
# Guest on which the test or script is running.
guest:
name: ...
role: ...
hostname: ...
# List of names of all provisioned guests.
guest-names:
- guest1
- guest2
...
# Same as `guest`, but one for each provisioned guest, with guest names
# as keys.
guests:
guest1:
name: guest1
role: ...
hostname: ...
guest2:
name: guest2
role: ...
hostname: ...
...
# List of all known roles.
role-names:
- role1
- role2
...
# Roles and their guests, with role names as keys.
roles:
role1:
- guest1
- guest2
- ...
role2:
- guestN
...
# Guest on which the test is running.
declare -A TMT_GUEST
TMT_GUEST[name]="..."
TMT_GUEST[role]="..."
TMT_GUEST[hostname]="..."
# Space-separated list of names of all provisioned guests.
TMT_GUEST_NAMES="guest1 guest2 ..."
# Same as `guest`, but one for each provisioned guest. Keys are constructed
# from guest name and the property name.
declare -A TMT_GUESTS
TMT_GUESTS[guest1.name]="guest1"
TMT_GUESTS[guest1.role]="..."
TMT_GUESTS[guest1.hostname]="..."
TMT_GUESTS[guest2.name]="guest2"
TMT_GUESTS[guest2.role]="..."
TMT_GUESTS[guest2.hostname]="..."
...
# Space-separated list of all known roles.
TMT_ROLE_NAMES="client server"
# Roles and their guests, with role names as keys.
declare -A TMT_ROLES
TMT_ROLES[role1]="guest1 guest2 ..."
TMT_ROLES[role2]="guestN ..."
...
Examples:
# A trivial pseudo-test script
. "$TMT_TOPOLOGY_BASH"
echo "I'm running on ${TMT_GUEST[name]}"
Status: implemented and verified
Verified by /tests/multihost/complete
Implemented by /tmt/steps/__init__.py