Merge branch 'main' into result-store-original-result

teemtee · Sep 25, 2024 · 744bbcc · 744bbcc
2 parents 276ba11 + 6e5b8b3
commit 744bbcc
Show file tree

Hide file tree

Showing 47 changed files with 1,318 additions and 791 deletions.
diff --git a/.github/workflows/doc-tests.yml b/.github/workflows/doc-tests.yml
@@ -17,7 +17,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        builder: [ html ]
+        builder: [ html, linkcheck ]
         include:
           # Run default html builder with warnings as error
           - builder: html

diff --git a/docs/contribute.rst b/docs/contribute.rst
@@ -10,6 +10,10 @@ Introduction
 
 Feel free and welcome to contribute to this project. You can start
 with filing issues and ideas for improvement in GitHub tracker__.
+Before creating a new issue you might want to check the existing
+issues to prevent filing a duplicate. Important issues affecting
+many users are marked with the `known issue`__ label.
+
 Our favorite thoughts from The Zen of Python:
 
 * Beautiful is better than ugly.
@@ -33,7 +37,8 @@ first word capitalized and enclose any names in single quotes:
 
     self.warn(f"File '{path}' not found.")
 
-__ https://github.com/teemtee/tmt
+__ https://github.com/teemtee/tmt/issues
+__ https://github.com/teemtee/tmt/issues?q=label%3A%22known+issue%22
 __ https://www.python.org/dev/peps/pep-0008/
 
 
@@ -662,6 +667,14 @@ make clean
 Release
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+The ``tmt`` project is released monthly. If there are urgent
+changes which need to be released quickly, a hotfix release may be
+created to address the important problem sooner.
+
+
+Regular
+------------------------------------------------------------------
+
 Follow the steps below to create a new major or minor release:
 
 * Update ``overview.rst`` with new contributors since the last release
@@ -700,3 +713,20 @@ __ https://tmt.readthedocs.io/en/stable/releases.html
 __ https://src.fedoraproject.org/rpms/tmt/pull-requests
 __ https://copr.fedorainfracloud.org/coprs/g/teemtee/tmt/builds/
 __ https://pypi.org/project/tmt/
+
+
+Hotfix
+------------------------------------------------------------------
+
+The following steps should be followed when an important urgent
+fix needs to be released before the regular schedule:
+
+* Create a new branch from the ``fedora`` branch
+* Use ``git cherry-pick`` to apply the selected change
+* Mention the hotfix release on the release page
+* Add a ``Release x.y.z`` commit, empty if needed: ``git commit --allow-empty -m "Release x.y.z"``
+* Create a new pull request with the target branch set to ``fedora``
+* Make sure that tests pass and merge the pull request
+* Tag the commit and publish the release in the same way as for
+  regular release
+* Create a pull request with the hotfix release notes changes
diff --git a/docs/releases.rst b/docs/releases.rst
@@ -4,9 +4,19 @@
     Releases
 ======================
 
-tmt-1.37
+tmt-1.37.0
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+The :ref:`/plugins/report/polarion` report plugin now uses Jinja template to
+generate the XUnit file. It doesn't do any extra modifications to the XML tree
+using an ``ElementTree`` anymore. Also the schema is now validated against the
+XSD.
+
+The values in the generated ``tmt-report-results.yaml`` file are
+now wrapped in double quotes, and any double quotes within the
+values are escaped to ensure that the resulting file is always
+valid YAML.
+
 The :ref:`/plugins/report/junit` report plugin now validates all the XML
 flavors against their respective XSD schemas and tries to prettify the final
 XML output. These functionalities are always disabled for ``custom`` flavors.
@@ -27,8 +37,20 @@ computed from the original outcome, and it is affected by inputs like
 :ref:`test result interpretation</spec/tests/result>` or
 :ref:`test checks</spec/tests/check>`.
 
+The ``fmf-id.ref`` will now try to report the most human-readable committish
+reference, either branch, tag, git-describe, or if all fails the commit hash.
+You may encounter this in the verbose log of ``tmt tests show`` or plan/test
+imports.
 
-tmt-1.37.0
+In verbose mode, the ``discover`` step now prints information
+about the beakerlib libraries which were fetched for the test
+execution. Use ``tmt run discover -vvv`` to see the details.
+
+``tmt try`` now supports :ref:`/stories/cli/try/option/epel` option
+backed by :ref:`prepare/feature</plugins/prepare/feature>` plugin.
+
+
+tmt-1.36.1
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 tmt will now put SSH master control socket into ``ssh-socket``
@@ -54,7 +76,8 @@ support for submitting jobs on behalf of other users, through
 ``beaker-job-owner`` key. The current user must be a submission delegate
 for the given job owner.
 
-In preparation for subresults: subresults and their checks have been integrated into HTML report and display plugin, result phase renamed to subresult.
+In preparation for subresults: subresults and their checks have been integrated
+into HTML report and display plugin, result phase renamed to subresult.
 
 
 tmt-1.35.0

diff --git a/docs/templates/story.rst.j2 b/docs/templates/story.rst.j2
@@ -54,7 +54,7 @@
     :param tmt.base.Link link: link to render.
 #}
 {% macro emit_tmt_repo_link(link) %}
-{{ _emit_remote_link(link, link.target, "https://github.com/teemtee/tmt/tree/" + (STORY.fmf_id.ref or 'main') + "/" + link.target.lstrip('/')) }}
+{{ _emit_remote_link(link, link.target, link.target | web_git_url(STORY.fmf_id.url, STORY.fmf_id.ref)) }}
 {% endmacro %}
 
 {#
@@ -78,7 +78,7 @@
 {% elif plugin_name == "provision/testcloud" %}
     {% set plugin_name = "provision/virtual" %}
 {% endif %}
-{{ _emit_remote_link(link, plugin_name, "https://github.com/teemtee/tmt/tree/" + (STORY.fmf_id.ref or 'main') + "/" + link.target) }}
+{{ _emit_remote_link(link, plugin_name, link.target | web_git_url(STORY.fmf_id.url, STORY.fmf_id.ref)) }}
 {% endmacro %}
 
 {#

diff --git a/pyproject.toml b/pyproject.toml
@@ -182,6 +182,7 @@ dependencies = ["tmt[docs]"]
 # script) e.g. -E, -d, and -D. These should not affect generated documentation
 # files below, but it may be providing additional metadata for RTD to index.
 html = "sphinx-build -b html {root}/docs {root}/docs/_build {args}"
+linkcheck = "sphinx-build -b linkcheck {root}/docs {root}/docs/_build {args}"
 man = [
     "cp {root}/docs/header.txt {root}/man.rst",
     "tail -n+8 docs/overview.rst >> {root}/man.rst",

diff --git a/stories/cli/try.fmf b/stories/cli/try.fmf
@@ -127,3 +127,22 @@ link:
             package: vim
         execute:
             how: tmt
+
+/option:
+    story:
+        As a user I want an easy way to run common options in ``1minutetip`` and ``tmt run``
+
+    /epel:
+        story:
+            As a user I want an easy way to enable epel repository
+        description: |
+            Enable EPEL repository
+
+            .. versionadded:: 1.37
+
+        example:
+            - tmt try centos-stream-9@virtual --epel
+
+        link:
+          - implemented-by: /tmt/trying.py
+          - implemented-by: /tmt/steps/prepare/feature.py
diff --git a/tests/discover/filtering.sh b/tests/discover/filtering.sh
@@ -86,7 +86,10 @@ rlJournalStart
         rlAssertEquals "Check that number of fmf-ids equals to tests number" \
                        "$ids_amount" "$tests_amount"
         rlAssertEquals "Check url" "$url_discover" "$url_fmf_id"
-        rlAssertEquals "Check ref" "HEAD" "$ref_fmf_id"
+        # Expected ref comes from running git describe on tmt repo:
+        #  git describe --tags eae4d527
+        expected_ref="0.10-8-geae4d527"
+        rlAssertEquals "Check ref" "$expected_ref" "$ref_fmf_id"
     rlPhaseEnd
 
     rlPhaseStartTest 'fmf-id (w/o url): Show fmf ids for discovered tests'

diff --git a/tests/execute/restraint/report-result/data/report.sh b/tests/execute/restraint/report-result/data/report.sh
@@ -13,43 +13,43 @@ rlJournalStart
         rlRun "rstrnt-report-result --server http://test-example.com report SKIP" 0 "Generating Restraint report of skipped test."
         rlRun "ls $RESULT_FILE" 0 "Result report successfully generated."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: skip' $rlRun_LOG
+        rlAssertGrep 'result: "skip"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rstrnt-report-result --server http://test-example.com --port 55 --disable-plugin avc --message 'Example output message.' -o /tmp/example_output.txt report PASS 66" 0 "Generating Restraint report of passed test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep "- report_example_output.txt" $rlRun_LOG
-        rlAssertGrep 'name: /report' $rlRun_LOG
-        rlAssertGrep 'result: pass' $rlRun_LOG
+        rlAssertGrep '- "report_example_output.txt"' $rlRun_LOG
+        rlAssertGrep 'name: "/report"' $rlRun_LOG
+        rlAssertGrep 'result: "pass"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rstrnt-report-result --server http://test-example.com report WARN" 0 "Generating Restraint report of warned test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: warn' $rlRun_LOG
+        rlAssertGrep 'result: "warn"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rstrnt-report-result --server http://test-example.com report FAIL" 0 "Generating Restraint report of failed test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: fail' $rlRun_LOG
+        rlAssertGrep 'result: "fail"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
     rlPhaseEnd
 
     rlPhaseStartTest 'Verify mocked RHTS rhts-report-result file generated correctly.'
         rlRun "rhts-report-result rhts-report SKIP /tmp/example_output.txt" 0 "Generating RHTS report of skipped test without optional metric."
         rlRun "ls $RESULT_FILE" 0 "Result report successfully generated."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: skip' $rlRun_LOG
+        rlAssertGrep 'result: "skip"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rhts-report-result rhts-report PASS /tmp/example_output.txt 66" 0 "Generating RHTS report of passed test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep "- rhts-report_example_output.txt" $rlRun_LOG
-        rlAssertGrep 'name: /rhts-report' $rlRun_LOG
-        rlAssertGrep 'result: pass' $rlRun_LOG
+        rlAssertGrep '- "rhts-report_example_output.txt"' $rlRun_LOG
+        rlAssertGrep 'name: "/rhts-report"' $rlRun_LOG
+        rlAssertGrep 'result: "pass"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rhts-report-result rhts-report WARN /tmp/example_output.txt 66" 0 "Generating RHTS report of warned test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: warn' $rlRun_LOG
+        rlAssertGrep 'result: "warn"' $rlRun_LOG
 	rlRun "rm -f $RESULT_FILE" 0 "Result report successfully deleted."
         rlRun "rhts-report-result rhts-report FAIL /tmp/example_output.txt 66" 0 "Generating RHTS report of failed test."
         rlRun -s "cat $RESULT_FILE"
-        rlAssertGrep 'result: fail' $rlRun_LOG
+        rlAssertGrep 'result: "fail"' $rlRun_LOG
     rlPhaseEnd
 
     rlPhaseStartCleanup

diff --git a/tests/execute/result/main.fmf b/tests/execute/result/main.fmf
@@ -7,3 +7,6 @@
 /repeated:
     summary: Repeated test should provide multiple results
     test: ./repeated.sh
+/special:
+    summary: Test special characters generated to tmt-report-results.yaml
+    test: ./special.sh
diff --git a/tests/execute/result/special.sh b/tests/execute/result/special.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+. /usr/share/beakerlib/beakerlib.sh || exit 1
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlRun "run=\$(mktemp -d)" 0 "Create run directory"
+        rlRun "pushd special"
+    rlPhaseEnd
+
+    rlPhaseStartTest "Check characters are correctly escaped in tmt-report-result output"
+        rlRun -s "tmt run -v -i $run" 0
+
+        RESULT_FILE="$run/special-chars/execute/data/guest/default-0/0-7-special-characters-in-the-name-1/data/tmt-report-results.yaml"
+        rlRun "yq -e '.' $RESULT_FILE" 0 "Check the YAML is valid"
+
+        rlAssertGrep 'name: "/0\.\.7 \\"special\\": \\" characters: \*\$@|&>< in: the: name"' "$RESULT_FILE"
+        rlAssertGrep "result: \"pass\"" "$RESULT_FILE"
+        rlAssertGrep "end-time: \".*\"" "$RESULT_FILE"
+        rlAssertGrep "data-path: \"/.*\"" "$RESULT_FILE"
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun "popd"
+        rlRun "rm -r ${run}" 0 "Remove run directory"
+    rlPhaseEnd
+rlJournalEnd
diff --git a/tests/execute/result/special/.fmf/version b/tests/execute/result/special/.fmf/version
@@ -0,0 +1 @@
+1
diff --git a/tests/execute/result/special/special-chars.fmf b/tests/execute/result/special/special-chars.fmf
@@ -0,0 +1,10 @@
+execute:
+    how: tmt
+discover:
+    how: shell
+    tests:
+      - name: '0..7 special: characters: *$@|&>< in: the: name'
+        result: restraint
+        test: "tmt-report-result '0..7 \"special\": \" characters: *$@|&>< in: the: name' PASS"
+provision:
+    how: container
diff --git a/tests/libraries/info/data/.fmf/version b/tests/libraries/info/data/.fmf/version
@@ -0,0 +1 @@
+1
diff --git a/tests/libraries/info/data/main.fmf b/tests/libraries/info/data/main.fmf
@@ -0,0 +1,9 @@
+/plan:
+    discover:
+        how: fmf
+    execute:
+        how: tmt
+
+/test:
+    test: /bin/true
+    require: library(httpd/http)
diff --git a/tests/libraries/info/main.fmf b/tests/libraries/info/main.fmf
@@ -0,0 +1,4 @@
+summary: Check info about fetched libraries
+description:
+    In verbose level the discover steps should print detailed
+    information about the beakerlib libraries being fetched.
diff --git a/tests/libraries/info/test.sh b/tests/libraries/info/test.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+. /usr/share/beakerlib/beakerlib.sh || exit 1
+
+rlJournalStart
+    rlPhaseStartSetup
+        rlRun "run=\$(mktemp -d)" 0 "Create a run directory"
+        rlRun "pushd data"
+    rlPhaseEnd
+
+    rlPhaseStartTest
+        rlRun -s "tmt run --id $run --remove discover -vvv"
+
+        rlAssertGrep "url: https://github.com/beakerlib/httpd" $rlRun_LOG
+        rlAssertGrep "url: https://github.com/beakerlib/openssl" $rlRun_LOG
+        rlAssertGrep "url: https://github.com/redhat-qe-security/certgen" $rlRun_LOG
+
+        rlRun "grep -A2 'https://github.com/redhat-qe-security/certgen' $rlRun_LOG"
+
+        rlAssertGrep "name: /certgen" $rlRun_LOG
+        rlAssertGrep "type: library" $rlRun_LOG
+    rlPhaseEnd
+
+    rlPhaseStartCleanup
+        rlRun "popd"
+        rlGetTestState || rlFileSubmit "$run/log.txt"
+        rlRun "rm -r $run" 0 "Remove the run directory"
+    rlPhaseEnd
+rlJournalEnd
diff --git a/tests/lint/plan/data/invalid-plugin-key.fmf b/tests/lint/plan/data/invalid-plugin-key.fmf
@@ -0,0 +1,5 @@
+prepare:
+    how: feature
+    wrong: key
+execute:
+    how: tmt
diff --git a/tests/lint/plan/test.sh b/tests/lint/plan/test.sh
@@ -67,6 +67,10 @@ rlJournalStart
         rlAssertGrep "fail P001 unknown key \"discove\" is used" $rlRun_LOG
         rlAssertGrep "fail P001 unknown key \"environmen\" is used" $rlRun_LOG
         rlAssertGrep "fail P001 unknown key \"summaryABCDEF\" is used" $rlRun_LOG
+
+        rlRun -s "$tmt plan lint invalid-plugin-key" 0
+        rlAssertGrep 'warn C000 key "wrong" not recognized by schema$' $rlRun_LOG
+        rlAssertGrep 'warn C000 key "wrong" not recognized by schema /schemas/prepare/feature' $rlRun_LOG
     rlPhaseEnd
 
     rlPhaseStartTest "P007: step phases require existing guests and roles"