images/krte: add Rootless Docker (and systemd as a dependency)

This commit adds the following runtime variables to support running Rootless Docker:
- `KRTE_SYSTEMD=true` (default: `false`): enable systemd
- `KRTE_SYSTEMD_ROOTLESS=true` (default: `false`): switch to a non-root user via systemd.
   The KRTE container itself still has to be run as the root,
   so DO NOT specify `securityContext.runAsUser`.

e.g.,
```console
$ docker build -t krte .

$ docker run -t --rm --privileged \
  -e KRTE_SYSTEMD=true \
  -e KRTE_SYSTEMD_ROOTLESS=true \
  -e DOCKER_IN_DOCKER_ENABLED=true \
  krte \
  docker info
wrapper.sh] [INFO] Re-executing in systemd: `docker info`
...
wrapper.sh] [INFO] Switching to rootless: `docker info`
...
wrapper.sh] [TEST] Running Test Command: `docker info`
...
Cgroup Driver: systemd
 Cgroup Version: 2
...
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
...
```

This is planned to be used for testing the `KubeletInUserNamespace` feature gate (KEP 2033),
by running Kubernetes inside Rootless Docker.

[FAQ: Why is systemd needed?]
For delegating cgroup controllers to a non-root user.
https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation

systemd is still opt-in for the default "rootful" mode, so this change will not affect the existing test jobs.

Signed-off-by: Akihiro Suda <[email protected]>

images/krte/Dockerfile

@@ -25,6 +25,12 @@ ARG IMAGE_ARG
 # empty value specifies the latest version.
 ARG GO_VERSION
 
+# args for the user to run rootless docker (for KRTE_SYSTEMD_ROOTLESS mode)
+ARG ROOTLESS_USER=rootless
+ARG ROOTLESS_UID=5000
+ARG ROOTLESS_SUBUID_BEGIN=100000
+ARG ROOTLESS_SUBUID_COUNT=1048576
+
 # add envs:
 # - so we can debug with the image name:tag
 # - adding gsutil etc. to path (where we will install them)
@@ -54,6 +60,8 @@ RUN echo "Installing Packages ..." \
             build-essential \
             ca-certificates \
             curl \
+            dbus \
+            dbus-user-session \
             file \
             git \
             gnupg2 \
@@ -65,7 +73,11 @@ RUN echo "Installing Packages ..." \
             procps \
             python3 \
             rsync \
+            slirp4netns \
             software-properties-common \
+            sudo \
+            systemd \
+            uidmap \
             unzip \
         && rm -rf /var/lib/apt/lists/* \
     && echo "Installing Go ..." \
@@ -92,17 +104,31 @@ RUN echo "Installing Packages ..." \
             "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
             tee /etc/apt/sources.list.d/docker.list > /dev/null \
         && apt-get update \
-        && apt-get install -y --no-install-recommends docker-ce docker-buildx-plugin \
+        && apt-get install -y --no-install-recommends docker-ce docker-buildx-plugin docker-ce-rootless-extras \
+        && systemctl disable docker \
         && rm -rf /var/lib/apt/lists/* \
         && sed -i 's/cgroupfs_mount$/#cgroupfs_mount\n/' /etc/init.d/docker \
     && echo "Ensuring Legacy Iptables ..." \
         && update-alternatives --set iptables /usr/sbin/iptables-legacy \
-        && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+        && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy \
+    && echo "Setting up the systemd delegation for rootless" \
+        && mkdir -p /etc/systemd/system/[email protected] \
+        && echo "[Service]" > /etc/systemd/system/[email protected]/delegate.conf \
+        && echo "Delegate=yes" >> /etc/systemd/system/[email protected]/delegate.conf \
+    && echo "Setting up the rootless user" \
+        && useradd --create-home --home-dir /home/${ROOTLESS_USER} --uid ${ROOTLESS_UID} -G systemd-journal ${ROOTLESS_USER} \
+        && echo "${ROOTLESS_UID}:${ROOTLESS_SUBUID_BEGIN}:${ROOTLESS_SUBUID_COUNT}" >/etc/subuid \
+        && cp -f /etc/subuid /etc/subgid \
+        && mkdir -p /home/${ROOTLESS_USER}/.local/share/docker \
+        && chown -R ${ROOTLESS_USER} /home/${ROOTLESS_USER}
 
 # copy in image utility scripts
+ADD --chmod=0755 https://raw.githubusercontent.com/AkihiroSuda/containerized-systemd/47070057becab3f44771b676e9c400289405b0cd/docker-entrypoint.sh \
+  /usr/local/bin/containerized-systemd.sh
 COPY wrapper.sh /usr/local/bin/
 
 # entrypoint is our wrapper script, in Prow you will need to explicitly re-specify this
 ENTRYPOINT ["wrapper.sh"]
 # volume for docker in docker, use an emptyDir in Prow
 VOLUME ["/var/lib/docker"]
+VOLUME ["/home/${ROOTLESS_USER}/.local/share/docker"]

images/krte/README.md

@@ -8,3 +8,11 @@ is maintained for the sole purpose of testing Kubernetes with KIND.
 ## WARNING
 
 This image is _not_ supported for other use cases. Use at your own risk.
+
+## Build-time variables
+See the `ARG` instructions in [`Dockerfile`](./Dockerfile).
+
+## Run-time variables
+- `KRTE_SYSTEMD=true` (default: `false`): enable systemd
+- `KRTE_SYSTEMD_ROOTLESS=true` (default: `false`): switch to a non-root user via systemd.
+  The KRTE container itself still has to be run as the root, so DO NOT specify `securityContext.runAsUser`.

images/krte/wrapper.sh

@@ -30,6 +30,26 @@ set -o errexit
 set -o pipefail
 set -o nounset
 
+: "${KRTE_SYSTEMD_ROOTLESS:=false}"
+: "${KRTE_SYSTEMD_ROOTLESS_USER:=rootless}"
+: "${KRTE_SYSTEMD:=${KRTE_SYSTEMD_ROOTLESS}}"
+
+if [ "${KRTE_SYSTEMD}" = "true" ] && [ "$$" = "1" ]; then
+  >&2 echo "wrapper.sh] [INFO] Re-executing in systemd: \`$*\`"
+  exec /usr/local/bin/containerized-systemd.sh "$0" "$@"
+fi
+if [ "${KRTE_SYSTEMD_ROOTLESS}" = "true" ] && [ "$(id -u)" = "0" ]; then
+  >&2 echo "wrapper.sh] [INFO] Waiting for the systemd user session to start up"
+  loginctl enable-linger ${KRTE_SYSTEMD_ROOTLESS_USER}
+  sleep 3
+  >&2 echo "wrapper.sh] [INFO] Switching to rootless: \`$*\`"
+  rootless_uid=$(id -u ${KRTE_SYSTEMD_ROOTLESS_USER})
+  exec sudo -E -H -u ${KRTE_SYSTEMD_ROOTLESS_USER} \
+    XDG_RUNTIME_DIR=/run/user/${rootless_uid} \
+    DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/${rootless_uid}/bus \
+    "$0" "$@"
+fi
+
 >&2 echo "wrapper.sh] [INFO] Wrapping Test Command: \`$*\`"
 >&2 echo "wrapper.sh] [INFO] Running in: ${KRTE_IMAGE}"
 >&2 echo "wrapper.sh] [INFO] See: https://github.com/kubernetes/test-infra/blob/master/images/krte/wrapper.sh"
@@ -40,7 +60,11 @@ cleanup(){
   if [[ "${DOCKER_IN_DOCKER_ENABLED:-false}" == "true" ]]; then
     >&2 echo "wrapper.sh] [CLEANUP] Cleaning up after Docker in Docker ..."
     docker ps -aq | xargs -r docker rm -f || true
-    service docker stop || true
+    if [ "$(id -u)" = "0" ]; then
+      service docker stop || true
+    else
+      systemctl --user stop docker || true
+    fi
     >&2 echo "wrapper.sh] [CLEANUP] Done cleaning up after Docker in Docker."
   fi
 }
@@ -93,7 +117,11 @@ export DOCKER_IN_DOCKER_ENABLED=${DOCKER_IN_DOCKER_ENABLED:-false}
 if [[ "${DOCKER_IN_DOCKER_ENABLED}" == "true" ]]; then
   >&2 echo "wrapper.sh] [SETUP] Docker in Docker enabled, initializing ..."
   # If we have opted in to docker in docker, start the docker daemon,
-  service docker start
+  if [ "$(id -u)" = "0" ]; then
+    service docker start
+  else
+    dockerd-rootless-setuptool.sh install
+  fi
   # the service can be started but the docker socket not ready, wait for ready
   WAIT_N=0
   while true; do