From e9c53af76652f5235087763b7b1522e7774e8d79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Prchl=C3=ADk?= Date: Thu, 16 Jan 2025 12:30:28 +0100 Subject: [PATCH] Move results format specification out of L2 metadata level (#3421) This should make the result format documentation more visible, more prominent. It is important part of tmt integration, and deserves its own place in specification, similar to what HW requirements have. --- docs/guide.rst | 2 +- docs/releases.rst | 8 ++++---- docs/scripts/generate-stories.py | 1 + docs/spec.rst | 1 + spec/plans/execute.fmf | 2 +- spec/{plans => }/results.fmf | 8 ++++---- spec/tests/result.fmf | 2 +- tmt/checks/avc.py | 4 ++-- 8 files changed, 15 insertions(+), 13 deletions(-) rename spec/{plans => }/results.fmf (98%) diff --git a/docs/guide.rst b/docs/guide.rst index 47c7c1a789..8f656234d6 100644 --- a/docs/guide.rst +++ b/docs/guide.rst @@ -971,7 +971,7 @@ to their services, synchronization, etc. tmt fully supports one test being executed multiple times. This is especially visible in the format of results, see -:ref:`/spec/plans/results`. Every test is assigned a "serial +:ref:`/spec/results`. Every test is assigned a "serial number", if the same test appears in multiple discover phases, each instance would be given a different serial number. The serial number and the guest from which a result comes from are then saved for each diff --git a/docs/releases.rst b/docs/releases.rst index 09ec1dc287..71588b4a03 100644 --- a/docs/releases.rst +++ b/docs/releases.rst @@ -29,7 +29,7 @@ support for submitting jobs on behalf of a group through the ``beaker-job-group`` key. The submitting user must be a member of the given job group. -The ``note`` field of tmt :ref:`/spec/plans/results` changes from +The ``note`` field of tmt :ref:`/spec/results` changes from a string to a list of strings, to better accommodate multiple notes. The ``Node`` alias for the ``Core`` class has been dropped as it @@ -256,7 +256,7 @@ committish reference, either branch, tag, git-describe, or if all fails the commit hash. You may encounter this in the verbose log of ``tmt tests show`` or plan/test imports. -:ref:`Result specification` now defines +:ref:`Result specification` now defines ``original-result`` key holding the original outcome of a test, subtest or test checks. The effective outcome, stored in ``result`` key, is computed from the original outcome, and it is @@ -286,7 +286,7 @@ tmt-1.36.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tmt will now emit a warning when :ref:`custom test results` -file does not follow the :ref:`result specification`. +file does not follow the :ref:`result specification`. We have started to use ``warnings.deprecated`` to advertise upcoming API deprecations. @@ -621,7 +621,7 @@ The **avc** :ref:`/spec/tests/check` allows to detect avc denials which appear during the test execution. A new ``skip`` custom result outcome has been added to the -:ref:`/spec/plans/results` specification. +:ref:`/spec/results` specification. All context :ref:`/spec/context/dimension` values are now handled in a case insensitive way. diff --git a/docs/scripts/generate-stories.py b/docs/scripts/generate-stories.py index 8faea9247d..509119002a 100755 --- a/docs/scripts/generate-stories.py +++ b/docs/scripts/generate-stories.py @@ -38,6 +38,7 @@ def __getattr__(cls, name: str) -> 'Mock': '/spec/stories': 'Stories', '/spec/context': 'Context', '/spec/hardware': 'Hardware', + '/spec/results': 'Results' } diff --git a/docs/spec.rst b/docs/spec.rst index 7150dd289e..e23d3392b8 100644 --- a/docs/spec.rst +++ b/docs/spec.rst @@ -54,4 +54,5 @@ Level 3: Stories spec/stories spec/context spec/hardware + spec/results spec/lint diff --git a/spec/plans/execute.fmf b/spec/plans/execute.fmf index fc3d2d3b43..7fb883ee36 100644 --- a/spec/plans/execute.fmf +++ b/spec/plans/execute.fmf @@ -20,7 +20,7 @@ description: | In each plan, the execute step must produce a ``results.yaml`` file with results for executed tests. The format of the file is described - at :ref:`/spec/plans/results`. + at :ref:`/spec/results`. /upgrade: summary: Perform system upgrades during testing diff --git a/spec/plans/results.fmf b/spec/results.fmf similarity index 98% rename from spec/plans/results.fmf rename to spec/results.fmf index c1a4b044eb..ba96efbb75 100644 --- a/spec/plans/results.fmf +++ b/spec/results.fmf @@ -1,6 +1,6 @@ -summary: Define format of on-disk storage of results -title: Results Format -order: 90 +story: + To integrate tmt with other tools, tmt should standardize and + document the format in which test results are saved on storage. description: | The following text defines a YAML file structure tmt uses for storing @@ -153,7 +153,7 @@ description: | check: ... - .. _/spec/plans/results/outcomes: + .. _/spec/results/outcomes: The ``result`` key can have the following values: diff --git a/spec/tests/result.fmf b/spec/tests/result.fmf index 4a842e5ec9..d7ec6db52a 100644 --- a/spec/tests/result.fmf +++ b/spec/tests/result.fmf @@ -23,7 +23,7 @@ description: | test needs to create its own ``results.yaml`` or ``results.json`` file in the ``${TMT_TEST_DATA}`` directory. The format of the file, notes and detailed examples - are documented at :ref:`/spec/plans/results`. + are documented at :ref:`/spec/results`. restraint handle ``tmt-report-result``, ``rstrnt-report-result`` and ``rhts-report-result`` commands as a separate test and diff --git a/tmt/checks/avc.py b/tmt/checks/avc.py index 96694440ee..a090455e32 100644 --- a/tmt/checks/avc.py +++ b/tmt/checks/avc.py @@ -277,9 +277,9 @@ class AvcDenials(CheckPlugin[Check]): To work correctly, the check requires SELinux to be enabled on the guest, and ``auditd`` must be running. Without SELinux, the check will turn into no-op, reporting - :ref:`skip` result, and + :ref:`skip` result, and without ``auditd``, the check will discover no AVC denials, - reporting :ref:`pass`. + reporting :ref:`pass`. If the test manipulates ``auditd`` or SELinux in general, the check may report unexpected results.