.. _/spec/results:

Results
=======
*To integrate tmt with other tools, tmt should standardize and document the format in which test results are saved on storage.*


The following text defines a YAML file structure tmt uses for storing
results. tmt itself will use it when saving results of ``execute`` step,
and custom test results are required to follow it when creating their
``results.yaml`` file.

Tests may choose JSON instead of YAML for their custom results file and
create ``results.json`` file, but tmt itself will always stick to YAML,
the final results would be provided in ``results.yaml`` file in any case.

Results are saved as a single list of dictionaries, each describing
a single test result.

.. code-block::

   # String, name of the test.
   name: /test/one

   # fmf ID of the test.
   fmf-id:
     url: http://some.git.host.com/project/tests.git
     ref: main
     name: /test/one
     path: /

   # String, the effective outcome of the test execution.
   result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

   # String, the original outcome of the test execution.
   original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

   # List of strings, optional comments to report with the result.
   note:
     - Things were great.

   # List of strings, paths to log files.
   log:
     - path/to/log1.txt
     - path/to/log2.log
       ...

   # Mapping, collection of various test IDs, if there are any to track.
   ids:
     some-id: foo
     another-id: bar

   # String, when the test started, in an ISO 8601 format.
   start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

   # String, when the test finished, in an ISO 8601 format.
   end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

   # String, how long did the test run.
   duration: hh:mm:ss

   # Integer, serial number of the test in the sequence of all tests of a plan.
   serial-number: 1

   # Mapping, describes the guest on which the test was executed.
   guest:
     name: client-1
     role: clients

   # String, path to /data directory storing possible test artifacts
   data-path: path/to/test/data

   # Mapping, stores the actual fmf context defined for this test.
   # It's a combination of the context provided via command line
   # and plan's `context` key.
   context:
     some-dimension:
       - its-value
     another-dimension:
       - first-value
       - second-value
         ...

  # Represents results of all test checks executed as driven by test's `check`
  # key. Fields have the same meaning as fields of the "parent" test result, but
  # relate to each check alone.
  check:
      # String, the effective outcome of the check execution.
    - result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

      # String, the original outcome of the check execution.
      original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

      # List of strings, optional comments to report with the result.
      note:
        - Things were great.

      # List of strings, paths to logs files.
      log:
        - path/to/check/log1.txt
        - path/to/check/log2.log
          ...

      # String, when the check started, in an ISO 8601 format.
      start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

      # String, when the check finished, in an ISO 8601 format.
      end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

      # String, how long did the check run.
      duration: hh:mm:ss

      # String, name of the check. Corresponds to the name of the check
      # specified in test metadata.
      name: dummy

      # String, the place in test workflow when the check was executed.
      event: "before-test"|"after-test"

  # Represents results of all subtests, if reported by the test. Fields
  # have the same meaning as fields of the "parent" test result, but relate
  # to each test alone.
  subresult:
      # String, name of the test phase.
    - name: First test case

      # String, the effective outcome of the phase execution.
      result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

      # String, the original outcome of the phase execution.
      original-result: "pass"|"fail"|"info"|"warn"|"error"|"skip"|"pending"

      # List of strings, optional comments to report with the result.
      note:
        - Things were great.

      # List of strings, paths to log files.
      log:
        - path/to/subresult/log1.txt
        - path/to/subresult/log2.log
          ...

      # String, when the subtest started, in an ISO 8601 format.
      start-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

      # String, when the subtest finished, in an ISO 8601 format.
      end-time: "yyyy-mm-ddThh:mm:ss.mmmmm+ZZ:ZZ"

      # String, how long did the subtest run.
      duration: hh:mm:ss

      # List, results of checks performed for this subtest.
      # It follows the same structure as the `check` key in the
      # "parent" test result.
      check:
        ...

.. _/spec/results/outcomes:

The ``result`` key can have the following values:

pass
    Test execution successfully finished and passed.

info
    Test finished but only produced an informational
    message. Represents a soft pass, used for tests
    with the :ref:`/spec/tests/result` attribute set
    to ``info``. Automation must treat this as a
    passed test.

warn
    A problem appeared during test execution which does
    not affect test results but might be worth checking
    and fixing. For example test cleanup phase failed.
    Automation must treat this as a failed test.

error
    Undefined problem encountered during test execution.
    Human inspection is needed to investigate whether it
    was a test bug, infrastructure error or a real test
    failure. Automation must treat it as a failed test.

fail
    Test execution successfully finished and failed.

skip
    Test was discovered but not executed. Can be used when
    a single process produces multiple results but not all
    tests were run.

pending
    Test was discovered and is waiting for execution.


The ``name`` and ``result`` keys are required. Also, ``name``, ``result``,
and ``event`` keys are required for entries under ``check`` key, and
``name`` and ``result`` keys are required for entries under ``subresult``
key. Custom result files may omit all other keys, although tmt plugins
will strive to provide as many keys as possible.

When importing the :ref:`custom results file </spec/tests/result>`, each
test name referenced in the file by the ``name`` key would be prefixed by
the original test name. A special case, ``name: /``, sets the result for
the original test itself.

The ``log`` key must list **relative** paths. Paths in the custom
results file are treated as relative to ``${TMT_TEST_DATA}`` path.
Paths in the final results file, saved by the execute step, will be
relative to the location of the results file itself.

The first ``log`` item is considered to be the "main" log, presented
to the user by default.

The ``serial-number``, ``guest`` and ``fmf-id`` keys, if present in the
custom results file, will be overwritten by tmt during their import after
test completes. This happens on purpose, to assure this vital
information is correct.

Similarly, the ``duration``, ``start-time`` and ``end-time`` keys, if
present in the special custom result, representing the original test
itself - ``name: /`` -, will be overwritten by tmt with actual
observed values. This also happens on purpose: while tmt cannot
tell how long it took to produce various custom results, it is still
able to report the duration of the whole test.

The same applies to ``context``, tmt will set this key for the
original test result in the custom result set to the value known
to tmt.

The ``original-result`` key holds the outcome of a test, check or
subresult as reported by the test, check or subresult itself, without
any additional influence.
A reported outcome may be a subject to interpretation, and tmt may
consider additional inputs and eventually report different effective
outcome in the ``result`` key.

The following rules apply when it comes to test, test check and subresult
outcomes and their interpretation and effects:

* Test outcome is interpreted according to
  :py:ref:`/spec/tests/result`. It is not yet planned for test check
  outcomes be interpreted. It is not yet planned for subresult
  outcomes to be interpreted.
* Test checks and subresults do not influence the effective test
  outcome. It is however planned for test check outcomes to affect
  the test outcome, see https://github.com/teemtee/tmt/issues/3185
  for more details. It is not planned for subresults to affect
  the test outcome, and we do not expect them to gain this effect.

See also the complete `JSON schema`__.

For custom results files in JSON format, the same rules and schema
apply.

.. versionchanged:: 1.30
   fmf context is now saved within results, in the ``context`` key.

.. versionchanged:: 1.34
   phase results are now defined, under the ``phase`` key.

.. versionchanged:: 1.36
   phase results are now renamed to the ``subresult`` key.

.. versionchanged:: 1.37
   original result of test, subtest and check is stored in ``original-result`` key.

.. versionchanged:: 1.41
   ``note`` field changed from a string to a list of strings.

__ https://github.com/teemtee/tmt/blob/main/tmt/schemas/results.yaml


**Examples:**


.. code-block:: yaml

   # Example content of results.yaml
   - name: /test/passing
     result: pass
     serial-number: 1
     log:
       - pass_log
     start-time: "2023-03-10T09:44:14.439120+00:00"
     end-time: "2023-03-10T09:44:24.242227+00:00"
     duration: 00:00:09
     note:
       - good result
     ids:
       extra-nitrate: some-nitrate-id
     guest:
       name: default-0

   - name: /test/failing
     result: fail
     serial-number: 2
     log:
       - fail_log
       - another_log
     start-time: "2023-03-10T09:44:14.439120+00:00"
     end-time: "2023-03-10T09:44:24.242227+00:00"
     duration: 00:00:09
     note:
       - fail result
     guest:
       name: default-0



.. code-block:: yaml

   # Example content of custom results file
   - name: /test/passing
     result: pass
     log:
       - pass_log
     duration: 00:11:22
     note:
       - good result
     ids:
       extra-nitrate: some-nitrate-id

   - name: /test/failing
     result: fail
     log:
       - fail_log
       - another_log
     duration: 00:22:33
     note:
       - fail result



.. code-block:: yaml

   # Example of a perfectly valid, yet stingy custom results file
   - name: /test/passing
     result: pass

   - name: /test/failing
     result: fail



.. code-block:: yaml

   # Example of test check results
   - name: /test/passing
     result: pass
     serial-number: 1
     log:
       - pass_log
     start-time: "2023-03-10T09:44:14.439120+00:00"
     end-time: "2023-03-10T09:44:24.242227+00:00"
     duration: 00:00:09
     note:
       - good result
     ids:
       extra-nitrate: some-nitrate-id
     guest:
       name: default-0
     check:
       - name: abrt
         event: after-test
         result: pass
         log: []
         note: []
       - name: kernel-panic
         event: after-test
         result: pass
         log: []
         note: []



.. code-block:: j

   /* Example content of custom results.json */
   [
     {
       "name": "/test/passing",
       "result": "pass",
       "log": ["pass_log"],
       "duration": "00:11:22",
       "note": ["good result"]
     }
   ]


**Status:** implemented and verified

* Verified by `/tests/execute/result/basic <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/check <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/custom <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/failure-logs <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/repeated <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/special <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_
* Verified by `/tests/execute/result/subresults <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tests/execute/result/main.fmf>`_


* Implemented by `/tmt/result.py <https://github.com/teemtee/tmt/tree/1.63.0-9-gcda563a/tmt/result.py>`_