Pipelines cannot update pip modules with fedora-40-test

[makubacki]

Describe the bug

Permissions are denied when Linux pipelines attempt to update pip modules with the fedora-40-test container:

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/opt/venv/lib64/python3.12/site-packages/regex-2024.7.24.dist-info/'
Check the permissions.

Example: https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=174261&view=logs&j=9701361e-2546-5093-77e5-58c75d95b7ce&t=2574abe6-1618-5f94-9ee2-669d1a5df774

To Reproduce

Steps to reproduce the behavior:

1. Update pip modules in pip-requirements.txt so new pip modules are installed during the pipeline run.
2. Run a Linux pipeline with the change that uses the container image

Expected behavior

The pipeline step to install pip modules should succeed.

Execution environment

• Container: ghcr.io/tianocore/containers/fedora-40-test:c98ff99

Additional context

The pipeline initializes the container in the "Initialize containers" step. It creates a new group and user:

/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 sh -c "command -v bash"
whoami 
vsts
id -u vsts
1001
id -g vsts
128
id -gn vsts
docker
Try to create a user with UID '1001' inside the container.
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 bash -c "getent passwd 1001 | cut -d: -f1 "
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 bash -c "if [[ -e '/etc/alpine-release' ]]; then echo 'Alpine-based image detected.'; fi"
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 groupadd -g 128 docker_azpcontainer
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 useradd -m -g docker_azpcontainer -u 1001 vsts_azpcontainer
Grant user 'vsts_azpcontainer' SUDO privilege and allow it run any command without authentication.
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 groupadd azure_pipelines_sudo
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 usermod -a -G azure_pipelines_sudo vsts_azpcontainer
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 su -c "echo '%azure_pipelines_sudo ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers"
Allow user 'vsts_azpcontainer' run any docker command without SUDO.
stat -c %g /var/run/docker.sock
128
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 bash -c "cat /etc/group"
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 usermod -a -G docker_azpcontainer vsts_azpcontainer
/usr/bin/docker exec  f27462233b1df931b19776b92422d92f1e4cccacf6ca7ccca8510023af1ddd29 bash -c "/__a/externals/node20_1/bin/node -v"

Permissions need to be arranged so the pip modules can be updated in the pipeline.

tianocore/edk2#10795 is blocked by this.

[osteffenrh]

Should be fixed by #10797 (pipeline still running).