diff --git a/docs/images/tests/GH-self-hosted-runners.png b/docs/images/tests/GH-self-hosted-runners.png deleted file mode 100644 index 8379bd74c5b..00000000000 Binary files a/docs/images/tests/GH-self-hosted-runners.png and /dev/null differ diff --git a/tests/README.rst b/tests/README.rst index 052b3de3e79..8401ffcab9a 100644 --- a/tests/README.rst +++ b/tests/README.rst @@ -6,8 +6,14 @@ such as unit tests, rpm tests and translation tests. All the tests will be run together if you follow the steps below. For integration tests there is a separate repository kickstart-tests_ containing also tooling for running the tests. +Testing in containers +--------------------- + +Most of our current testing is set inside the containers. This section will describe +how to correctly run and build these containers. + Run unit tests inside of container ----------------------------------- +__________________________________ This is the primary and recommended way to run the tests. Right now only unit tests are supported by the container, not rpm-tests. @@ -45,7 +51,7 @@ modified/touched by the container (it works on an internal copy of the host's anaconda directory). Interactively work inside of container --------------------------------------- +______________________________________ For interactively working in the container you can run:: @@ -93,27 +99,17 @@ In case the *ci* target fails there is also a *coverage-report* target which can be used to combine the multiple `.coverage` files into one and produce a human readable report. -Note ----- - -Please update your container from time to time to have newest dependencies. -To do that, run `podman pull quay.io/rhinstaller/anaconda-ci:main` or build -it locally again. - Run rpm tests inside of container ---------------------------------- +_________________________________ -First, build the container image for running the test, as it does not yet get -published to any registry:: +The rpm tests are taking care that rpm file has all necessary content. - make -f Makefile.am anaconda-rpm-build - -Then run the test in that container:: +To run the test in a container:: make -f Makefile.am container-rpm-test Run unit tests with patched pykickstart or other libraries ----------------------------------------------------------- +__________________________________________________________ 1. Pull the container:: @@ -137,6 +133,18 @@ Run unit tests with patched pykickstart or other libraries 5. Run other commands for container ci as usual. Don't forget to append ``CI_TAG=`` to make calls if you committed the container under a custom tag. +Keep your containers updated +____________________________ + +Please update your container from time to time to have newest dependencies. +To do that, run:: + + podman pull quay.io/rhinstaller/anaconda-ci:main + +or build it locally again by:: + + make -f ./Makefile.am anaconda-ci-build + GitHub workflows ---------------- @@ -145,32 +153,27 @@ All test and maintenance actions are run by `GitHub workflows`_. These YAML files completely describe what steps are required to run some action, what are its triggers and so on. +Because we are using self-hosted runners, ``pull_request_trigger`` and other reasons, +we have our GitHub repositories configured that they need approval for every execution +of the tests (including after force push) for every external contributors. + Pull request for main: ________________________ -Unit and rpm tests are run by the `validate.yml workflow`_. We use GitHub's +Unit and rpm tests are using the GitHub `pull_request` trigger. We use GitHub's runners for this so we don't have to care about what is executed there. -The workflow rebuilds the ``anaconda-ci`` container if the container files +The test workflow rebuilds the ``anaconda-ci`` container if the container files have changed, otherwise it is pulling the container from `quay.io`_. For more information see below. Pull request for RHEL: ______________________ -Unit and rpm tests are run by the `validate-rhel-8.yml workflow`_ on (fully -automatically deployed) self-hosted runners in our Upshift instance. - -These runners are ``anaconda-ci:rhel8`` containers with all the dependencies in -place so the yml configuration will just execute tests. You can start runners -locally by running the container and providing GitHub token. That is pretty -valuable in case of workflow testing. See `github-action-run-once`_ for more -details. - -To protect our self-hosted runners, tests only run automatically for -`rhinstaller organization members `_. -For external contributors, an organization member needs to approve the test run -by sending a comment starting with ``/tests``. +Unit and rpm tests are using a similar solution as the upstream ones. Containers +are build on top of ``quay.io/centos/centos:streamXX`` images where ``XX`` is RHEL major release +number. Code for RHEL is shared with CentOS Stream so we decided to run tests on +CentOS Stream containers as these are easier to integrate. Running kickstart-tests: ________________________ @@ -181,14 +184,14 @@ comment that starts with ``/kickstart-tests `` to the pull request to trigger it. It is possible to use tests updated via a kickstart-tests repository PR. See the `kickstart-tests.yml workflow`_ for supported options. For more detailed information on tests selection see the -`kickstart launch script`_ documentation and-its ``--help`` +`kickstart launch script`_ documentation and-its ``--help``. Container maintenance --------------------- All active branches run tests in containers. Containers have all the dependencies installed and the environment prepared to run tests or connect our -GitHub runners (used by RHEL only). +GitHub runners (for places where we need /dev/kvm access). Automatic container build _________________________ @@ -208,53 +211,27 @@ container you can push your branch to the origin repo and run it from there. Security precautions for testing RHEL ------------------------------------- -Getting into our host/internal network -______________________________________ - -One of the main precautions is that each container test run has -a limited time and is destroyed after timeout/end of test. That should narrow -what attackers could do or how they can create a backdoor. See the image for -more info: - -.. image:: ../docs/images/tests/GH-self-hosted-runners.png +Beware of the ``pull_request_target`` +_____________________________________ +For many reasons, we are using ``pull_request_trigger`` in our workflows, however, +this trigger is not secure in some scenarios. See `GitHub documentation`_ for more +information. We need to make sure that this trigger is not executed on an unsafe code. -Another hardening of this is potential issue is that only PRs -approved by/created by users with permission to write are able to run the tests. -To achieve this we have two ways how to start the test. +The main issue starts with running these on checkout code from PR. In this case, +the attacker has a free hand to change our code, do a release, or use our +self-hosted runners. -**PR created by rhinstaller member** -- these are started from the RHEL branch -workflow file by ``pull_request_target`` as usual. This workflow has two -dependent jobs. First will check user privileges, second will run the tests in -case the first one succeeded. - -**PR created by external contributors** -- these have to be started by workflow -file `validate-rhel-8.yml workflow`_ from the ``main`` branch -checking all the comments. If comment starts with ``/test`` phrase it will check -the owner of the comment. When everything succeed it will set progress on the pull -request originating the comment and start the tests. This progress is updated -based on the result of the tests. As explained above, the whole implementation -of the workflow is in the ``main`` branch which could be pretty confusing. - -Changing workflow file by attacker -__________________________________ - -Because test description is part of the repository, attackers may change -workflow files by creating PR to do their malicious attack. Because of that we -are using ``pull_request_target`` instead of ``pull_request`` trigger. The main -difference is that ``pull_request_target`` will run your PR tests on the target -branch not on your PR branch. So workflow configuration has to be merged first -to apply workflow changes. This has to be set on all workflow files in all -branches, otherwise attackers could change existing workflow files to use our -runners even for branches where they are not normally used. Unfortunately, -self-hosted runners can’t be bound to the branch, they are bound to the repo. +As the first line of defense, we are not running automatically any workflows on +a pull request from external contributors and each test run have to be manually +approved by developer. How can I change the workflow _____________________________ -Due to our hardening it’s not possible to just create PR and see the result -of your change on the PR checks tab. You have to create PR on your fork branch -which has the updated workflow. I would recommend you to create a test +It depends on a `GitHub trigger`_ used by the workflow. However, if it is not +possible to create a PR and see your changes, you can create PR on your fork +branch which has the updated workflow. I would recommend you to create a test organization for this and avoid creating a new account. Similar situation works even for workflow to automatically update our containers. @@ -272,8 +249,6 @@ represents a different class of tests. They are - *cppcheck/* - static C/C++ code analysis using the *cppcheck* tool; - *shellcheck/* - shell code analyzer config; -- *dd_tests/* - Python unit tests for driver disk utilities (dracut/dd); -- *unit_tests/dracut_tests/* - Python unit tests for the dracut hooks used to configure the installation environment and load Anaconda; - *gettext/* - sanity tests of files used for translation; Written in Python and Bash; @@ -282,11 +257,15 @@ represents a different class of tests. They are a temporary directory without failing dependencies or other RPM issues and checks if all files are correctly present in the RPM; - *lib/* - helper modules used during testing; +- *unit_tests/dd_tests/* - Python unit tests for driver disk utilities (dracut/dd); +- *unit_tests/dracut_tests/* - Python unit tests for the dracut hooks used to configure the - *unit_tests/pyanaconda_tests/* - unit tests for the :mod:`pyanaconda` module; -- *pylint/* - checks the validity of Python source code using the *pocketlint* +- *unit_tests/regex_tests/* - Python unit tests for regular expressions defined in +- *unit_tests/shell_tests/* - Python unit tests for the shell code in Dracut +- *pylint/* - checks the validity of Python source code tool; - *ruff/* - config for fast but not 100% correct linter for Python; -- *unit_tests/regex_tests/* - Python unit tests for regular expressions defined in +- *vulture/* - scripts to execute vulture linter used to find a dead code in the project :mod:`pyanaconda.regexes`; .. NOTE:: @@ -294,6 +273,9 @@ represents a different class of tests. They are All Python unit tests inherit from the standard :class:`unittest.TestCase` class unless specified otherwise! + Also tests are written in the Python `unittests library`_ style but they are executed + by `pytest`_. + Some tests require root privileges and will be skipped if running as regular user! @@ -306,10 +288,11 @@ The launcher scripts are listed under `TESTS` in `tests/Makefile.am`. .. _quay.io: https://quay.io/repository/rhinstaller/anaconda-ci .. _pytest -k: https://docs.pytest.org/en/7.1.x/reference/reference.html#command-line-flags .. _GitHub workflows: https://docs.github.com/en/free-pro-team@latest/actions -.. _validate.yml workflow: ../.github/workflows/validate.yml -.. _validate-rhel-8.yml workflow: ../.github/workflows/validate-rhel-8.yml .. _kickstart-tests.yml workflow: ../.github/workflows/kickstart-tests.yml .. _kickstart launch script: https://github.com/rhinstaller/kickstart-tests/blob/master/containers/runner/README.md .. _container-autoupdate.yml workflow: ../.github/workflows/container-autoupdate.yml .. _actions tab: https://github.com/rhinstaller/anaconda/actions?query=workflow%3A%22Refresh+container+images%22 -.. _github-action-run-once: https://github.com/rhinstaller/anaconda/blob/rhel-8/dockerfile/anaconda-ci/github-action-run-once +.. _unittests library: https://docs.python.org/3/library/unittest.html +.. _pytest: https://docs.pytest.org/en/stable/ +.. _GitHub documentation: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target +.. _GitHub trigger: https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows