.. _/stories/deferred: Deferred ======== *As tmt contributors we want to keep separately some older, postponed but still valuable ideas for possible future use.* This section contains user stories which are not planned to be implemented in the near future. They still might be useful for inspiration or even might be later resurrected from the sleep. .. _/stories/deferred/file: file ^^^^ *As a tester I want to store test results of all plans into a single file including the overall aggregated result.* .. note:: This is a draft, the story is not implemented yet. Save the report into a ``report.yaml`` file. The ``OVERALL_RESULT`` is the overall result of all plan results. It is counted the same way as ``PLAN_RESULT``. The ``TEST_RESULT`` is the same as defined in the :ref:`/spec/results` definition. * info - test finished and produced only information message * passed - test finished and passed * failed - test finished and failed * error - a problem encountered during test execution Note the priority of test results is as written above, with ``info`` having the lowest priority and ``error`` has the highest. This is important for ``PLAN_RESULT``. The ``PLAN_RESULT`` is the overall result or all test results for the plan run. It has the same values as ``TEST_RESULT``. Plan result is counted according to the priority of the test outcome values. For example: * if the test results are info, passed, passed - the plan result will be passed * if the test results are info, passed, failed - the plan result will be failed * if the test results are failed, error, passed - the plan result will be error The ``LOG_PATH`` is the test log output path, relative to the execute step plan run directory. The ``log`` key will be a list of such paths, even if there is just a single log. **Examples:** .. code-block:: yaml result: OVERALL_RESULT plans: /plan/one: result: PLAN_RESULT tests: /test/one: result: TEST_RESULT log: - LOG_PATH /test/two: result: TEST_RESULT log: - LOG_PATH - LOG_PATH - LOG_PATH /plan/two: result: PLAN_RESULT /test/one: result: TEST_RESULT log: - LOG_PATH **Status:** idea .. _/stories/deferred/gate: gate ^^^^ Gates relevant for testing *As a user I want to easily enable plan for selected gates.* .. note:: This is a draft, the story is not implemented yet. Multiple gates can be defined in the process of releasing a change. Currently we define the following gates: merge-pull-request block merging a pull request into a git branch add-build-to-update attaching a build to an erratum / bodhi update add-build-to-compose block adding a build to a compose release-update block releasing an erratum / bodhi update Each plan can define one or more gates it should be blocking. Attached is an example of configuring multiple gates. **Examples:** .. code-block:: yaml /test: /pull-request: /pep: summary: All code must comply with the PEP8 style guide # Do not allow ugly code to be merged into the main branch gate: - merge-pull-request /lint: summary: Run pylint to catch common problems (no gating) /build: /smoke: summary: Basic smoke test (Tier1) # Basic smoke test is used by three gates gate: - merge-pull-request - add-build-to-update - add-build-to-compose /features: summary: Verify important features /update: # This enables the 'release-update' gate for all three plans gate: - release-update /basic: summary: Run all Tier1, Tier2 and Tier3 tests /security: summary: Security tests (extra job to get quick results) /integration: summary: Integration tests with related components **Status:** idea .. _/stories/deferred/interactive: interactive ^^^^^^^^^^^ *Provide way similar to git rebase --interactive* .. note:: This is a draft, the story is not implemented yet. Provide users with list of steps and let them edit them. Allow adding commands between, or at least interactive "pause". When user would be finished with single step, just continue recipe. Do not force users to remember all steps when working with them. Make it possible to repeat single step or abort current run. Allow repeating some steps just by repeating lines with task. **Examples:** .. code-block:: yaml tmt run --all --interactive .. code-block:: yaml tmt run --continue .. code-block:: yaml tmt run --abort **Status:** idea .. _/stories/deferred/restraint: restraint ^^^^^^^^^ *As a tester I want the option to execute tests using the ``restraint`` harness.* .. note:: This is a draft, the story is not implemented yet. Restraint is the default harness in beaker for RHEL8 and beyond. In order to provide compatibility with beaker style tests, I would like a way to invoke ``tmt`` using the Restaint harness. This would enable Restraint tests to be invoked by ``tmt`` without modification. Some common commands include ``rstrnt-reboot``, ``rstrnt-abort``, and ``rstrnt-report-result``. Although implementation of the full execute step plugin has been deferred, backward compatibility scripts have been provided for several most commonly used restraint commands. See the :ref:`restraint-compatibility` section for details. **Examples:** .. code-block:: yaml tmt run --all execute --how restraint **Status:** idea .. _/stories/deferred/where: where ^^^^^ An alternative syntax for the ``where`` key *As a user I want a bit more concise way to define which tests should run on which guest.* .. note:: This is a draft, the story is not implemented yet. This is an alternative syntax proposed for the ``where`` key which uses a dictionary encapsulating the ``discover`` config under keys corresponding to guest names or roles. This can result in much more concise config especially when defining several shell scripts for each guest or role. **Examples:** .. code-block:: yaml # Filter discovered fmf tests by tag discover: where: client: - how: fmf filter: tag:client-tests server: - how: fmf filter: tag:server-tests .. code-block:: yaml # Directly define tests as shell scripts discover: where: server: how: shell tests: - test: server-script-one.sh - test: server-script-two.sh - test: server-script-three.sh client: how: shell tests: - test: client-script-one.sh - test: client-script-two.sh - test: client-script-three.sh **Status:** idea