Multistage poetry build (#480)

* Build doesn't require requirements.txt anymore

* fixing Dockerfile

* Tweaked Dockerfile, better python version handling

* fix: fixed pythonpath setting

Apparently cannot do command substitution in ENV

* ci: Remove compile stage in CI, and whitespace in Dockerfile

---------

Co-authored-by: Alan Christie <[email protected]>

.github/workflows/build-dev.yaml

@@ -80,14 +80,6 @@ jobs:
       run: |
         pip install --requirement build-requirements.txt
         pre-commit run --all-files
-    - name: Compile requirements.txt
-      # We need to put the file in $HOME
-      # - a simple way to ensure it's
-      # available between steps in the same job.
-      run: |
-        pip install --upgrade pip
-        pip install poetry==1.7.1
-        poetry export --without-hashes --without dev --output requirements.txt
     - name: Docker build
       uses: docker/build-push-action@v4
       with:

.github/workflows/build-production.yaml

@@ -127,11 +127,6 @@ jobs:
       run: |
         pip install --requirement build-requirements.txt
         pre-commit run --all-files
-    - name: Compile requirements.txt
-      run: |
-        pip install --upgrade pip
-        pip install poetry==1.7.1
-        poetry export --without-hashes --without dev --output requirements.txt
     - name: Build
       uses: docker/build-push-action@v4
       with:

.github/workflows/build-staging.yaml

@@ -149,11 +149,6 @@ jobs:
       run: |
         pip install --requirement build-requirements.txt
         pre-commit run --all-files
-    - name: Compile requirements.txt
-      run: |
-        pip install --upgrade pip
-        pip install poetry==1.7.1
-        poetry export --without-hashes --without dev --output requirements.txt
     - name: Docker build
       uses: docker/build-push-action@v4
       with:

Dockerfile

@@ -1,10 +1,4 @@
-FROM python:3.11.7-slim-bullseye
-
-# We rely on the presence of a requirements.txt file in project root.
-# This is achieved prior to a container build using poetry
-# and is the responsibility of the developer or CI process: -
-#
-#   poetry export --without-hashes --without dev --output requirements.txt
+FROM python:3.11.7-slim-bullseye  AS python-base
 
 ENV PYTHONUNBUFFERED 1
 ENV PYTHONDONTWRITEBYTECODE 1
@@ -23,14 +17,33 @@ RUN apt-get update -y && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+# another stage for poetry installation. this ensures poetry won't end
+# up in final image where it's not needed
+FROM python-base AS poetry-base
+
+ARG POETRY_VERSION=1.7.1
+RUN pip install --no-cache-dir poetry==${POETRY_VERSION}
+
+WORKDIR /
+COPY poetry.lock pyproject.toml /
+
+# POETRY_VIRTUALENVS_IN_PROJECT tells poetry to create the venv to
+# project's directory (.venv). This way the location is predictable
+RUN POETRY_VIRTUALENVS_IN_PROJECT=true poetry install --no-root --only main --no-directory
+
+# final stage. only copy the venv with installed packages and point
+# paths to it
+FROM python-base as final
+
+COPY --from=poetry-base /.venv /.venv
+
+ENV PYTHONPATH="${PYTHONPATH}:/.venv/lib/python3.11/site-packages/"
+ENV PATH=/.venv/bin:$PATH
+
 WORKDIR /srv/logs
 WORKDIR /code/logs
 WORKDIR /code
 
-COPY requirements.txt ./
-RUN pip install --no-cache-dir --upgrade pip && \
-    pip install --no-cache-dir --requirement requirements.txt
-
 COPY nginx.conf /etc/nginx/nginx.conf
 COPY django_nginx.conf /etc/nginx/sites-available/default.conf
 COPY proxy_params /etc/nginx/frag_proxy_params

README.md

@@ -66,31 +66,13 @@ installs/updates new packages to local venv. It's equivalent to running
 `poetry lock && poetry install`, so if you're not interested in local environment and
 just want to update the lockfile, you can run just `poetry lock`.
 
->   **TODO**: at the time of writing (2023-07-21), this project is still on Python 3.7,
-    which means some packages that are needed for development (specifically `pre-commit`),
-    have not been added to `pyproject.toml` due to version conflict. Once Python has
-    been updated to more recent version they can be added to project requirements where
-    they belong; until that time, instructions in section [Pre-commit](#pre-commit)
-    are still relevant.
 
 ## Building and running (local)
-The backend is a Docker container image and can be build and deployed locally using
-`docker-compose`.
-
-The build relies on a legacy `requirements.txt` file (to avoid polluting the run-time
-container with material and packages it does not need), and you need to create this
-file using `poetry`. After any package dependency changes, and before any container
-image build you must **export** the poetry environment as a requirements.txt file
-using its `export` command. For example: -
-
-    poetry export --without-hashes --without dev --output requirements.txt
-
-This is done automatically in the CI process but you need to do it at least once
-prior to building locally. Once done you can then build the container image
-with `docker-compose``: -
+The backend is a Docker container image and can be build and deployed locally using `docker-compose`: -
 
     docker-compose build
 
+
 To run the application (which wil include deployment of the postgres and neo4j databases)
 run: -