Deferred
As tmt contributors we want to keep separately some older, postponed but still valuable ideas for possible future use.
This section contains user stories which are not planned to be implemented in the near future. They still might be useful for inspiration or even might be later resurrected from the sleep.
file
As a tester I want to store test results of all plans into a single file including the overall aggregated result.
Note
This is a draft, the story is not implemented yet.
Save the report into a report.yaml file.
The OVERALL_RESULT is the overall result of all plan
results. It is counted the same way as PLAN_RESULT.
The TEST_RESULT is the same as defined in the
Results definition.
info - test finished and produced only information message
passed - test finished and passed
failed - test finished and failed
error - a problem encountered during test execution
Note the priority of test results is as written above, with
info having the lowest priority and error has the
highest. This is important for PLAN_RESULT.
The PLAN_RESULT is the overall result or all test results
for the plan run. It has the same values as TEST_RESULT.
Plan result is counted according to the priority of the test
outcome values. For example:
if the test results are info, passed, passed - the plan result will be passed
if the test results are info, passed, failed - the plan result will be failed
if the test results are failed, error, passed - the plan result will be error
The LOG_PATH is the test log output path, relative to the
execute step plan run directory. The log key will be a
list of such paths, even if there is just a single log.
Examples:
result: OVERALL_RESULT
plans:
/plan/one:
result: PLAN_RESULT
tests:
/test/one:
result: TEST_RESULT
log:
- LOG_PATH
/test/two:
result: TEST_RESULT
log:
- LOG_PATH
- LOG_PATH
- LOG_PATH
/plan/two:
result: PLAN_RESULT
/test/one:
result: TEST_RESULT
log:
- LOG_PATH
Status: idea
gate
Gates relevant for testing
As a user I want to easily enable plan for selected gates.
Note
This is a draft, the story is not implemented yet.
Multiple gates can be defined in the process of releasing a change. Currently we define the following gates:
- merge-pull-request
block merging a pull request into a git branch
- add-build-to-update
attaching a build to an erratum / bodhi update
- add-build-to-compose
block adding a build to a compose
- release-update
block releasing an erratum / bodhi update
Each plan can define one or more gates it should be blocking. Attached is an example of configuring multiple gates.
Examples:
/test:
/pull-request:
/pep:
summary: All code must comply with the PEP8 style guide
# Do not allow ugly code to be merged into the main branch
gate:
- merge-pull-request
/lint:
summary: Run pylint to catch common problems (no gating)
/build:
/smoke:
summary: Basic smoke test (Tier1)
# Basic smoke test is used by three gates
gate:
- merge-pull-request
- add-build-to-update
- add-build-to-compose
/features:
summary: Verify important features
/update:
# This enables the 'release-update' gate for all three plans
gate:
- release-update
/basic:
summary: Run all Tier1, Tier2 and Tier3 tests
/security:
summary: Security tests (extra job to get quick results)
/integration:
summary: Integration tests with related components
Status: idea
interactive
Provide way similar to git rebase –interactive
Note
This is a draft, the story is not implemented yet.
Provide users with list of steps and let them edit them. Allow adding commands between, or at least interactive “pause”. When user would be finished with single step, just continue recipe. Do not force users to remember all steps when working with them. Make it possible to repeat single step or abort current run. Allow repeating some steps just by repeating lines with task.
Examples:
tmt run --all --interactive
tmt run --continue
tmt run --abort
Status: idea
restraint
As a tester I want the option to execute tests using the ``restraint`` harness.
Note
This is a draft, the story is not implemented yet.
Restraint is the default harness in beaker for RHEL8 and
beyond. In order to provide compatibility with beaker
style tests, I would like a way to invoke tmt using
the Restaint harness. This would enable Restraint tests to
be invoked by tmt without modification. Some common
commands include rstrnt-reboot, rstrnt-abort, and
rstrnt-report-result.
Although implementation of the full execute step plugin has been deferred, backward compatibility scripts have been provided for several most commonly used restraint commands. See the Restraint Compatibility section for details.
Examples:
tmt run --all execute --how restraint
Status: idea
where
An alternative syntax for the where key
As a user I want a bit more concise way to define which tests should run on which guest.
Note
This is a draft, the story is not implemented yet.
This is an alternative syntax proposed for the where key which uses a dictionary encapsulating the discover config under keys corresponding to guest names or roles. This can result in much more concise config especially when defining several shell scripts for each guest or role.
Examples:
# Filter discovered fmf tests by tag
discover:
where:
client:
- how: fmf
filter: tag:client-tests
server:
- how: fmf
filter: tag:server-tests
# Directly define tests as shell scripts
discover:
where:
server:
how: shell
tests:
- test: server-script-one.sh
- test: server-script-two.sh
- test: server-script-three.sh
client:
how: shell
tests:
- test: client-script-one.sh
- test: client-script-two.sh
- test: client-script-three.sh
Status: idea