Improve tests/README

tests/README.rst

@@ -6,8 +6,14 @@ such as unit tests, rpm tests and translation tests.  All the tests will be run
 together if you follow the steps below.  For integration tests there is a
 separate repository kickstart-tests_ containing also tooling for running the tests.
 
+Testing in containers
+---------------------
+
+Most of our current testing is set inside the containers. This section will describe
+how to correctly run and build these containers.
+
 Run unit tests inside of container
-----------------------------------
+__________________________________
 This is the primary and recommended way to run the tests.
 
 Right now only unit tests are supported by the container, not rpm-tests.
@@ -45,7 +51,7 @@ modified/touched by the container (it works on an internal copy of the host's
 anaconda directory).
 
 Interactively work inside of container
---------------------------------------
+______________________________________
 
 For interactively working in the container you can run::
 
@@ -93,27 +99,17 @@ In case the *ci* target fails there is also a *coverage-report* target
 which can be used to combine the multiple `.coverage` files into one and
 produce a human readable report.
 
-Note
-----
-
-Please update your container from time to time to have newest dependencies.
-To do that, run `podman pull quay.io/rhinstaller/anaconda-ci:main` or build
-it locally again.
-
 Run rpm tests inside of container
----------------------------------
+_________________________________
 
-First, build the container image for running the test, as it does not yet get
-published to any registry::
+The rpm tests are taking care that rpm file has all necessary content.
 
-    make -f Makefile.am anaconda-rpm-build
-
-Then run the test in that container::
+To run the test in a container::
 
     make -f Makefile.am container-rpm-test
 
 Run unit tests with patched pykickstart or other libraries
-----------------------------------------------------------
+__________________________________________________________
 
 1. Pull the container::
 
@@ -137,6 +133,18 @@ Run unit tests with patched pykickstart or other libraries
 5. Run other commands for container ci as usual. Don't forget to append ``CI_TAG=<your-tag>`` to
    make calls if you committed the container under a custom tag.
 
+Keep your containers updated
+____________________________
+
+Please update your container from time to time to have newest dependencies.
+To do that, run::
+
+    podman pull quay.io/rhinstaller/anaconda-ci:main
+
+or build it locally again by::
+
+    make -f ./Makefile.am anaconda-ci-build
+
 
 GitHub workflows
 ----------------
@@ -145,32 +153,27 @@ All test and maintenance actions are run by `GitHub workflows`_.  These YAML
 files completely describe what steps are required to run some action, what are
 its triggers and so on.
 
+Because we are using self-hosted runners, ``pull_request_trigger`` and other reasons,
+we have our GitHub repositories configured that they need approval for every execution
+of the tests (including after force push) for every external contributors.
+
 Pull request for main:
 ________________________
 
-Unit and rpm tests are run by the `validate.yml workflow`_.  We use GitHub's
+Unit and rpm tests are using the GitHub `pull_request` trigger.  We use GitHub's
 runners for this so we don't have to care about what is executed there.
 
-The workflow rebuilds the ``anaconda-ci`` container if the container files
+The test workflow rebuilds the ``anaconda-ci`` container if the container files
 have changed, otherwise it is pulling the container from `quay.io`_. For more
 information see below.
 
 Pull request for RHEL:
 ______________________
 
-Unit and rpm tests are run by the `validate-rhel-8.yml workflow`_ on (fully
-automatically deployed) self-hosted runners in our Upshift instance.
-
-These runners are ``anaconda-ci:rhel8`` containers with all the dependencies in
-place so the yml configuration will just execute tests.  You can start runners
-locally by running the container and providing GitHub token. That is pretty
-valuable in case of workflow testing.  See `github-action-run-once`_ for more
-details.
-
-To protect our self-hosted runners, tests only run automatically for
-`rhinstaller organization members <https://github.com/orgs/rhinstaller/people>`_.
-For external contributors, an organization member needs to approve the test run
-by sending a comment starting with ``/tests``.
+Unit and rpm tests are using a similar solution as the upstream ones. Containers
+are build on top of ``quay.io/centos/centos:streamXX`` images where ``XX`` is RHEL major release
+number. Code for RHEL is shared with CentOS Stream so we decided to run tests on
+CentOS Stream containers as these are easier to integrate.
 
 Running kickstart-tests:
 ________________________
@@ -181,14 +184,14 @@ comment that starts with ``/kickstart-tests <options>`` to the pull request to
 trigger it. It is possible to use tests updated via a kickstart-tests
 repository PR. See the `kickstart-tests.yml workflow`_ for supported
 options. For more detailed information on tests selection see the
-`kickstart launch script`_ documentation and-its ``--help``
+`kickstart launch script`_ documentation and-its ``--help``.
 
 Container maintenance
 ---------------------
 
 All active branches run tests in containers. Containers have all the
 dependencies installed and the environment prepared to run tests or connect our
-GitHub runners (used by RHEL only).
+GitHub runners (for places where we need /dev/kvm access).
 
 Automatic container build
 _________________________
@@ -208,53 +211,27 @@ container you can push your branch to the origin repo and run it from there.
 Security precautions for testing RHEL
 -------------------------------------
 
-Getting into our host/internal network
-______________________________________
-
-One of the main precautions is that each container test run has
-a limited time and is destroyed after timeout/end of test. That should narrow
-what attackers could do or how they can create a backdoor. See the image for
-more info:
-
-.. image:: ../docs/images/tests/GH-self-hosted-runners.png
+Beware of the ``pull_request_target``
+_____________________________________
 
+For many reasons, we are using ``pull_request_trigger`` in our workflows, however,
+this trigger is not secure in some scenarios. See `GitHub documentation`_ for more
+information. We need to make sure that this trigger is not executed on an unsafe code.
 
-Another hardening of this is potential issue is that only PRs
-approved by/created by users with permission to write are able to run the tests.
-To achieve this we have two ways how to start the test.
+The main issue starts with running these on checkout code from PR. In this case,
+the attacker has a free hand to change our code, do a release, or use our
+self-hosted runners.
 
-**PR created by rhinstaller member** -- these are started from the RHEL branch
-workflow file by ``pull_request_target`` as usual. This workflow has two
-dependent jobs. First will check user privileges, second will run the tests in
-case the first one succeeded.
-
-**PR created by external contributors** -- these have to be started by workflow
-file `validate-rhel-8.yml workflow`_ from the ``main`` branch
-checking all the comments. If comment starts with ``/test`` phrase it will check
-the owner of the comment. When everything succeed it will set progress on the pull
-request originating the comment and start the tests. This progress is updated
-based on the result of the tests. As explained above, the whole implementation
-of the workflow is in the ``main`` branch which could be pretty confusing.
-
-Changing workflow file by attacker
-__________________________________
-
-Because test description is part of the repository, attackers may change
-workflow files by creating PR to do their malicious attack. Because of that we
-are using ``pull_request_target`` instead of ``pull_request`` trigger. The main
-difference is that ``pull_request_target`` will run your PR tests on the target
-branch not on your PR branch. So workflow configuration has to be merged first
-to apply workflow changes. This has to be set on all workflow files in all
-branches, otherwise attackers could change existing workflow files to use our
-runners even for branches where they are not normally used. Unfortunately,
-self-hosted runners can’t be bound to the branch, they are bound to the repo.
+As the first line of defense, we are not running automatically any workflows on
+a pull request from external contributors and each test run have to be manually
+approved by developer.
 
 How can I change the workflow
 _____________________________
 
-Due to our hardening it’s not possible to just create PR and see the result
-of your change on the PR checks tab. You have to create PR on your fork branch
-which has the updated workflow. I would recommend you to create a test
+It depends on a `GitHub trigger`_ used by the workflow. However, if it is not
+possible to create a PR and see your changes, you can create PR on your fork
+branch which has the updated workflow. I would recommend you to create a test
 organization for this and avoid creating a new account.
 
 Similar situation works even for workflow to automatically update our containers.
@@ -272,8 +249,6 @@ represents a different class of tests. They are
 
 - *cppcheck/* - static C/C++ code analysis using the *cppcheck* tool;
 - *shellcheck/* - shell code analyzer config;
-- *dd_tests/* - Python unit tests for driver disk utilities (dracut/dd);
-- *unit_tests/dracut_tests/* - Python unit tests for the dracut hooks used to configure the
   installation environment and load Anaconda;
 - *gettext/* - sanity tests of files used for translation; Written in Python and
   Bash;
@@ -282,18 +257,25 @@ represents a different class of tests. They are
   a temporary directory without failing dependencies or other RPM issues and checks if
   all files are correctly present in the RPM;
 - *lib/* - helper modules used during testing;
+- *unit_tests/dd_tests/* - Python unit tests for driver disk utilities (dracut/dd);
+- *unit_tests/dracut_tests/* - Python unit tests for the dracut hooks used to configure the
 - *unit_tests/pyanaconda_tests/* - unit tests for the :mod:`pyanaconda` module;
-- *pylint/* - checks the validity of Python source code using the *pocketlint*
+- *unit_tests/regex_tests/* - Python unit tests for regular expressions defined in
+- *unit_tests/shell_tests/* - Python unit tests for the shell code in Dracut
+- *pylint/* - checks the validity of Python source code
   tool;
 - *ruff/* - config for fast but not 100% correct linter for Python;
-- *unit_tests/regex_tests/* - Python unit tests for regular expressions defined in
+- *vulture/* - scripts to execute vulture linter used to find a dead code in the project
   :mod:`pyanaconda.regexes`;
 
 .. NOTE::
 
     All Python unit tests inherit from the standard :class:`unittest.TestCase`
     class unless specified otherwise!
 
+    Also tests are written in the Python `unittests library`_ style but they are executed
+    by `pytest`_.
+
     Some tests require root privileges and will be skipped if running as regular
     user!
 
@@ -306,10 +288,11 @@ The launcher scripts are listed under `TESTS` in `tests/Makefile.am`.
 .. _quay.io: https://quay.io/repository/rhinstaller/anaconda-ci
 .. _pytest -k: https://docs.pytest.org/en/7.1.x/reference/reference.html#command-line-flags
 .. _GitHub workflows: https://docs.github.com/en/free-pro-team@latest/actions
-.. _validate.yml workflow: ../.github/workflows/validate.yml
-.. _validate-rhel-8.yml workflow: ../.github/workflows/validate-rhel-8.yml
 .. _kickstart-tests.yml workflow: ../.github/workflows/kickstart-tests.yml
 .. _kickstart launch script: https://github.com/rhinstaller/kickstart-tests/blob/master/containers/runner/README.md
 .. _container-autoupdate.yml workflow: ../.github/workflows/container-autoupdate.yml
 .. _actions tab: https://github.com/rhinstaller/anaconda/actions?query=workflow%3A%22Refresh+container+images%22
-.. _github-action-run-once: https://github.com/rhinstaller/anaconda/blob/rhel-8/dockerfile/anaconda-ci/github-action-run-once
+.. _unittests library: https://docs.python.org/3/library/unittest.html
+.. _pytest: https://docs.pytest.org/en/stable/
+.. _GitHub documentation: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target
+.. _GitHub trigger: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows