Avocado is stuck with podman container > fedora:38

[richtja]

Describe the bug
Avocado execution is stuck when podman spawner is used and container image is fedora:39 or higher.

Steps to reproduce
$ avocado run --spawner=podman --spawner-podman-image=fedora:39 examples/tests/passtest.py

Expected behavior

JOB ID     : fdf4b4829524a9b3c8991ae54d0e6caf6cc29d3f
JOB LOG    : /home/janrichter/avocado/job-results/job-2025-02-04T14.48-fdf4b48/job.log
 (1/1) examples/tests/passtest.py:PassTest.test: STARTED
 (1/1) examples/tests/passtest.py:PassTest.test: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2025-02-04T14.48-fdf4b48/results.html

Current behavior

JOB ID     : 727b87f5af76f230974bc4a7495ada7290926b08
JOB LOG    : /home/janrichter/avocado/job-results/job-2025-02-04T14.49-727b87f/job.log

Additional information

It looks like the container is properly configured, and avocado image is installed, but it crashes during the Task execution.

2025-02-04 14:49:09,580 avocado.core.task.statemachine statemachine     L0176 DEBUG| <Worker spawner="<avocado.plugins.spawners.podman.PodmanSpawner object at 0x7f2fed1920f0>" max_triaging=12 max_running=1 task_timeout=None> has been initialized
2025-02-04 14:49:09,582 avocado.core.task.statemachine statemachine     L0230 DEBUG| Task "1-examples/tests/passtest.py:PassTest.test": requested -> triaging
2025-02-04 14:49:09,582 avocado.core.task.statemachine statemachine     L0252 DEBUG| Task "1-examples/tests/passtest.py:PassTest.test": requirements OK (will proceed to check dependencies)
2025-02-04 14:49:09,582 avocado.core.task.statemachine statemachine     L0372 DEBUG| Task "1-examples/tests/passtest.py:PassTest.test": about to be spawned with "<avocado.plugins.spawners.podman.PodmanSpawner object at 0x7f2fed1920f0>"
2025-02-04 14:49:09,583 avocado.utils.podman podman           L0186 DEBUG| Executing ('run', '--rm', '--entrypoint=["/usr/bin/env", "python3", "-c", "import sys; print(sys.version_info.major, sys.version_info.minor, sys.executable)"]', 'fedora:39')
2025-02-04 14:49:22,794 avocado.utils.podman podman           L0195 DEBUG| Return code: 0
2025-02-04 14:49:22,794 avocado.utils.podman podman           L0196 DEBUG| Stdout: 3 12 /usr/bin/python3

2025-02-04 14:49:22,794 avocado.utils.podman podman           L0197 DEBUG| Stderr: Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:39...
Getting image source signatures
Copying blob sha256:e39ecf0219034e4dde7889b117de47a4f052dcf6a8105346412a9c2507bbf3e6
Copying config sha256:c29ab042bd6da559d0cab51dd117b7ad8579ac6cd4661035df8a5ef2734be6e1
Writing manifest to image destination

2025-02-04 14:49:22,794 avocado.utils.asset asset            L0398 INFO | Fetching asset https://github.com/avocado-framework/setuptools/releases/download/v59.2.0/setuptools-59.2.0-py3.12.egg
2025-02-04 14:49:22,795 avocado.utils.asset asset            L0353 DEBUG| No hash provided. Cannot check the asset file integrity.
2025-02-04 14:49:22,795 avocado.utils.asset asset            L0474 INFO | Asset already exists in cache.
2025-02-04 14:49:22,795 avocado.utils.asset asset            L0398 INFO | Fetching asset https://github.com/avocado-framework/avocado/releases/download/109.0/avocado_framework-109.0-py3.12.egg
2025-02-04 14:49:22,795 avocado.utils.asset asset            L0353 DEBUG| No hash provided. Cannot check the asset file integrity.
2025-02-04 14:49:22,795 avocado.utils.asset asset            L0474 INFO | Asset already exists in cache.
2025-02-04 14:49:22,795 avocado.utils.podman podman           L0186 DEBUG| Executing ('create', '--privileged', '-v', '/var/tmp/avocado_ncfh28te/.status_server.sock:/tmp/.status_server.sock', '-v', '/home/janrichter/avocado/job-results/job-2025-02-04T14.49-727b87f/test-results/1-examples_tests_passtest.py_PassTest.test:/home/janrichter/avocado/job-results/job-2025-02-04T14.49-727b87f/test-results/1-examples_tests_passtest.py_PassTest.test', '-v', '/home/janrichter/Avocado/avocado/examples/tests/passtest.py:/tmp/examples/tests/passtest.py:ro', '--entrypoint=["/usr/bin/python3", "-m", "avocado.plugins.runners.avocado_instrumented", "task-run", "-i", "1-examples/tests/passtest.py:PassTest.test", "-j", "727b87f5af76f230974bc4a7495ada7290926b08", "-k", "avocado-instrumented", "-u", "/tmp/examples/tests/passtest.py:PassTest.test", "-c", "{\\"run.test_parameters\\": [], \\"datadir.paths.cache_dirs\\": [\\"/home/janrichter/avocado/data/cache\\"], \\"core.show\\": {\\"__encoded_set__\\": [\\"app\\"]}, \\"job.output.loglevel\\": \\"DEBUG\\", \\"job.run.store_logging_stream\\": [], \\"runner.identifier_format\\": \\"{uri}\\"}", "tags=json:{\\"fast\\": null}", "output_dir=/home/janrichter/avocado/job-results/job-2025-02-04T14.49-727b87f/test-results/1-examples_tests_passtest.py_PassTest.test", "-s", "/tmp/.status_server.sock"]', '-e=PYTHONPATH=/tmp/setuptools-59.2.0-py3.12.egg:/tmp/avocado_framework-109.0-py3.12.egg', 'fedora:39')
2025-02-04 14:49:22,862 avocado.utils.podman podman           L0195 DEBUG| Return code: 0
2025-02-04 14:49:22,862 avocado.utils.podman podman           L0196 DEBUG| Stdout: 681dae552e76ff88fd352d79da970ec4908fdb84938fcfa8e5446fb3f3dea611

2025-02-04 14:49:22,862 avocado.utils.podman podman           L0197 DEBUG| Stderr: 
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0398 INFO | Fetching asset https://github.com/avocado-framework/setuptools/releases/download/v59.2.0/setuptools-59.2.0-py3.12.egg
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0353 DEBUG| No hash provided. Cannot check the asset file integrity.
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0474 INFO | Asset already exists in cache.
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0398 INFO | Fetching asset https://github.com/avocado-framework/avocado/releases/download/109.0/avocado_framework-109.0-py3.12.egg
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0353 DEBUG| No hash provided. Cannot check the asset file integrity.
2025-02-04 14:49:22,862 avocado.utils.asset asset            L0474 INFO | Asset already exists in cache.
2025-02-04 14:49:22,862 avocado.utils.podman podman           L0186 DEBUG| Executing ('cp', '/home/janrichter/avocado/data/cache/by_location/130600b3b754d1efc861345b6beb7cb841440479/setuptools-59.2.0-py3.12.egg', '681dae552e76ff88fd352d79da970ec4908fdb84938fcfa8e5446fb3f3dea611:/tmp/setuptools-59.2.0-py3.12.egg')
2025-02-04 14:49:23,861 avocado.utils.podman podman           L0195 DEBUG| Return code: 0
2025-02-04 14:49:23,862 avocado.utils.podman podman           L0196 DEBUG| Stdout: 
2025-02-04 14:49:23,862 avocado.utils.podman podman           L0197 DEBUG| Stderr: 
2025-02-04 14:49:23,862 avocado.utils.podman podman           L0186 DEBUG| Executing ('cp', '/home/janrichter/avocado/data/cache/by_location/5d538fe8782dd0459cb2eb00fb8070d7ec1e3406/avocado_framework-109.0-py3.12.egg', '681dae552e76ff88fd352d79da970ec4908fdb84938fcfa8e5446fb3f3dea611:/tmp/avocado_framework-109.0-py3.12.egg')
2025-02-04 14:49:24,081 avocado.utils.podman podman           L0195 DEBUG| Return code: 0
2025-02-04 14:49:24,081 avocado.utils.podman podman           L0196 DEBUG| Stdout: 
2025-02-04 14:49:24,081 avocado.utils.podman podman           L0197 DEBUG| Stderr: 
2025-02-04 14:49:24,081 avocado.utils.podman podman           L0186 DEBUG| Executing ('start', '681dae552e76ff88fd352d79da970ec4908fdb84938fcfa8e5446fb3f3dea611')
2025-02-04 14:49:24,221 avocado.utils.podman podman           L0195 DEBUG| Return code: 0
2025-02-04 14:49:24,222 avocado.utils.podman podman           L0196 DEBUG| Stdout: 681dae552e76ff88fd352d79da970ec4908fdb84938fcfa8e5446fb3f3dea611

2025-02-04 14:49:24,222 avocado.utils.podman podman           L0197 DEBUG| Stderr: 
2025-02-04 14:49:24,222 avocado.core.task.statemachine statemachine     L0379 DEBUG| Task "1-examples/tests/passtest.py:PassTest.test": spawned successfully
2025-02-04 14:49:24,261 avocado.core.task.statemachine statemachine     L0399 DEBUG| Task "1-examples/tests/passtest.py:PassTest.test" is alive at monitor phase
2025-02-04 14:58:18,458 avocado.core.task.statemachine statemachine     L0176 DEBUG| <Worker spawner="<avocado.plugins.spawners.podman.PodmanSpawner object at 0x7f2fed1920f0>" max_triaging=12 max_running=1 task_timeout=None> has been initialized
2025-02-04 14:58:19,154 avocado.job runner_nrunner   L0372 INFO | 

[richtja]

The container fails with:

/usr/bin/python3: Error while finding module specification for 'avocado.plugins.runners.avocado_instrumented' (ModuleNotFoundError: No module named 'avocado')