chore: update development mode scripts to containerize Poetry and other relevant dependencies

[nicomiguelino]

Docs

• https://github.com/nicomiguelino/Anthias/blob/update-dev-mode-script/docs/developer-documentation.md#dockerized-development-environment

Alternative Strategy

• We can do something line docker run --rm -v $(pwd):/code <dockerfile>, which will generate the required Dockerfiles.
• Afterwards, we can run the Docker Compose command for building and startup of the containers.
• We don't want to pollute the host machine with local dependencies.

[salmanfarisvp]

@nicomiguelino, I tried to build containers locally ( By following the "Dockerized development environment") on Mac but got errors.

Machine details

MacBook Air
Chip: Apple M1
Memmory: 8 GB
macOS : Sequoia 15.1.1

Error log:

 ./bin/start_development_server.sh
Skipping installation of APT packages. Detected macOS.
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################################################################################################################################################### 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
##O=- #     #
Warning: [email protected] 3.11.10 is already installed and up-to-date.
To reinstall 3.11.10, run:
  brew reinstall [email protected]
Retrieving Poetry metadata

The latest version (1.8.4) is already installed.
Installing dependencies from lock file

No dependencies to install or update
Skipping test service for x86...
[+] Building 11.2s (9/17)                                                                                                                                                     docker-container:confident_liskov
 => [anthias-server internal] load build definition from Dockerfile.server                                                                                                                                 0.0s
 => => transferring dockerfile: 1.53kB                                                                                                                                                                     0.0s
 => [anthias-server] resolve image config for docker-image://docker.io/docker/dockerfile:1.4                                                                                                               1.8s
 => [anthias-server auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                            0.0s
 => CACHED [anthias-server] docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                         0.0s
 => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                     0.0s
 => [anthias-server] local://context                                                                                                                                                                       0.0s
 => => transferring context: 442B                                                                                                                                                                          0.0s
 => [anthias-server internal] load metadata for docker.io/library/debian:bookworm                                                                                                                          1.1s
 => [anthias-server auth] library/debian:pull token for registry-1.docker.io                                                                                                                               0.0s
 => CACHED [anthias-server] docker-image://docker.io/library/debian:bookworm@sha256:10901ccd8d249047f9761845b4594f121edef079cfd8224edebd9ea726f0a7f6                                                       0.0s
 => => resolve docker.io/library/debian:bookworm@sha256:10901ccd8d249047f9761845b4594f121edef079cfd8224edebd9ea726f0a7f6                                                                                   0.0s
 => [anthias-server] local://context                                                                                                                                                                       0.0s
 => => transferring context: 41.39kB                                                                                                                                                                       0.0s
 => [anthias-server] /bin/sh -c apt-get update &&     apt-get -y install --no-install-recommends         build-essential         cec-utils         curl         ffmpeg         git         git-core        8.2s
 => => #   dbus libdecor-0-plugin-1-cairo | libdecor-0-plugin-1 libfile-fcntllock-perl
 => => #   liblocale-gettext-perl libgdk-pixbuf2.0-bin libglib2.0-data xdg-user-dirs
 => => #   javascript-common krb5-locales libldap-common pocketsphinx-en-us
 => => #   publicsuffix librsvg2-common libsasl2-modules va-driver-all | va-driver
 => => #   vdpau-driver-all | vdpau-driver mesa-vulkan-drivers | vulkan-icd cppzmq-dev
 => => #   file nodejs-doc node-tap netbase python3-olefile

108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
108.2 W: Tried to start delayed item http://deb.debian.org/debian bookworm/main arm64 libwbclient0 arm64 2:4.17.12+dfsg-0+deb12u1, but failed
.....
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/z/zvbi/libzvbi-common_0.2.41-1_all.deb  Could not connect to deb.debian.org:80 (199.232.106.132), connection timed out [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/z/zvbi/libzvbi0_0.2.41-1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/f/ffmpeg/libavcodec59_5.1.6-0%2bdeb12u1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/a/alsa-lib/libasound2-data_1.2.8-1_all.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/a/alsa-lib/libasound2_1.2.8-1%2bb1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/libr/libraw1394/libraw1394-11_2.1.2-2_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/liba/libavc1394/libavc1394-0_0.5.4-5_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/liba/libass/libass9_0.17.1-1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/libu/libudfread/libudfread0_1.1.2-1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/libb/libbluray/libbluray2_1.3.4-1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/c/chromaprint/libchromaprint1_1.5.1-2%2bb1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80
......
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/p/pygobject/python3-gi_3.42.2-3%2bb1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/p/pillow/python3-pil_9.4.0-1.1%2bdeb12u1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/s/setuptools/python3-setuptools_66.1.1-1_all.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/w/wheel/python3-wheel_0.38.4-2_all.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python-pip/python3-pip_23.0.1%2bdfsg-1_all.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/s/simplejson/python3-simplejson_3.18.3-1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Failed to fetch http://deb.debian.org/debian/pool/main/s/sqlite3/sqlite3_3.40.1-2%2bdeb12u1_arm64.deb  Unable to connect to deb.debian.org:80: [IP: 199.232.106.132 80]
108.2 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
failed to solve: process "/bin/sh -c apt-get update &&     apt-get -y install --no-install-recommends         build-essential         cec-utils         curl         ffmpeg         git         git-core         ifupdown         libcec-dev          libffi-dev         libssl-dev         libzmq3-dev         libzmq5-dev         libzmq5         lsb-release         mplayer         net-tools         procps         psmisc         python3-dev         python3-gi         python3-pil         python3-pip         python3-setuptools         python3-simplejson         python-is-python3         sudo         sqlite3         nodejs

[nicomiguelino]

@salmanfarisvp, it looks like a network issue inside Docker containers. Can you run the following?

# Start the container.
docker run -itd --name debian-bookworm-test debian:bookworm

# Get inside the container.
docker exec -it debian-bookworm-latest bash

# Run the following inside the container.
apt-get update -y
apt-get install -y --no-install-recommends python3-dev

Alternatively, you can try enabling Rosetta on your mac.

[salmanfarisvp]

@nicomiguelino Thanks, it's worked.

 ./bin/start_development_server.sh
Skipping installation of APT packages. Detected macOS.
Warning: [email protected] 3.11.10 is already installed and up-to-date.
To reinstall 3.11.10, run:
  brew reinstall [email protected]
Retrieving Poetry metadata

The latest version (1.8.4) is already installed.
Installing dependencies from lock file

No dependencies to install or update
Skipping test service for x86...
[+] Building 591.4s (57/57) FINISHED                                                                                                                                          docker-container:confident_liskov
 => [anthias-server internal] booting buildkit                                                                                                                                                             6.7s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                         6.0s
 => => creating container buildx_buildkit_confident_liskov0                                                                                                                                                0.7s
 => [anthias-server internal] load build definition from Dockerfile.server                                                                                                                                 0.0s
 => => transferring dockerfile: 1.53kB                                                                                                                                                                     0.0s
 => [anthias-nginx] resolve image config for docker-image://docker.io/docker/dockerfile:1.4                                                                                                                5.0s
 => [anthias-server auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                            0.0s
 => CACHED [anthias-nginx] docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                          0.0s
 => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                     0.0s
 => [anthias-server] local://context                                                                                                                                                                       0.0s
 => => transferring context: 442B                                                                                                                                                                          0.0s
 => [anthias-nginx internal] load metadata for docker.io/library/debian:bookworm                                                                                                                           4.5s
 => [anthias-server auth] library/debian:pull token for registry-1.docker.io                                                                                                                               0.0s
 => CACHED [anthias-nginx] docker-image://docker.io/library/debian:bookworm@sha256:10901ccd8d249047f9761845b4594f121edef079cfd8224edebd9ea726f0a7f6                                                        0.1s
 => => resolve docker.io/library/debian:bookworm@sha256:10901ccd8d249047f9761845b4594f121edef079cfd8224edebd9ea726f0a7f6                                                                                   0.0s
 => [anthias-server] local://context                                                                                                                                                                       0.0s
 => => transferring context: 41.39kB                                                                                                                                                                       0.0s
 => [anthias-server] /bin/sh -c apt-get update &&     apt-get -y install --no-install-recommends         build-essential         cec-utils         curl         ffmpeg         git         git-core      225.6s
 => [anthias-server] /bin/sh -c c_rehash                                                                                                                                                                   6.1s
 => [anthias-server] /bin/sh -c pip3 install --upgrade pip --break-system-packages &&     pip3 install wheel --break-system-packages                                                                       6.5s
 => [anthias-server] copy /requirements/requirements.txt /tmp/requirements.txt                                                                                                                             0.0s
 => [anthias-server] /bin/sh -c pip3 install -r /tmp/requirements.txt --break-system-packages                                                                                                             65.7s
 => [anthias-server] /bin/sh -c mkdir -p /usr/src/app                                                                                                                                                      0.2s
 => [anthias-server] copy / /usr/src/app/                                                                                                                                                                  0.1s
 => [anthias-server] mkdir /usr/src/app                                                                                                                                                                    0.0s
 => [anthias-server] exporting to docker image format                                                                                                                                                    100.8s
 => => exporting layers                                                                                                                                                                                   33.9s
 => => exporting manifest sha256:a8beccf3885aefbf2f9f7a6cb417bb0486d903afdbb5bf3ef2c2feedd2497901                                                                                                          0.0s
 => => exporting config sha256:537304f177f116668ac5bac9da5541e7bc9772592d37fc043e8e1a2b91b8bb0b                                                                                                            0.0s
 => => sending tarball                                                                                                                                                                                    66.8s
 => [anthias-server] importing to docker                                                                                                                                                                  43.6s
 => => loading layer 37fb38f2a0c7 421.13MB / 430.40MB                                                                                                                                                     43.5s
 => => loading layer 6a0afb2d1d6c 7.34kB / 7.34kB                                                                                                                                                          5.2s
 => => loading layer 585783230e30 65.54kB / 5.84MB                                                                                                                                                         5.1s
 => => loading layer b5f14de746d4 603B / 603B                                                                                                                                                              4.9s
 => => loading layer d12ef393d2ca 60.72MB / 89.08MB                                                                                                                                                        4.9s
 => => loading layer db68e7bb68b8 133B / 133B                                                                                                                                                              0.5s
 => => loading layer 5b472325901c 32.77kB / 2.62MB                                                                                                                                                         0.3s
 => => loading layer 5f70bf18a086 32B / 32B                                                                                                                                                                0.2s
 => [anthias-server] resolving provenance for metadata file                                                                                                                                                0.0s
 => [anthias-celery internal] load build definition from Dockerfile.celery                                                                                                                                 0.0s
 => => transferring dockerfile: 1.62kB                                                                                                                                                                     0.0s
 => [anthias-websocket internal] load build definition from Dockerfile.websocket                                                                                                                           0.0s
 => => transferring dockerfile: 1.50kB                                                                                                                                                                     0.0s
 => [anthias-websocket auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                         0.0s
 => [anthias-websocket] local://context                                                                                                                                                                    0.0s
 => => transferring context: 442B                                                                                                                                                                          0.0s
 => [anthias-websocket auth] library/debian:pull token for registry-1.docker.io                                                                                                                            0.4s
 => [anthias-celery] local://context                                                                                                                                                                       0.0s
 => => transferring context: 442B                                                                                                                                                                          0.0s
 => [anthias-celery] local://context                                                                                                                                                                       0.2s
 => => transferring context: 4.96MB                                                                                                                                                                        0.2s
 => [anthias-websocket] local://context                                                                                                                                                                    0.1s
 => => transferring context: 25.54kB                                                                                                                                                                       0.1s
 => CACHED [anthias-websocket] /bin/sh -c apt-get update &&     apt-get -y install --no-install-recommends         build-essential         cec-utils         curl         ffmpeg         git         git-  0.0s
 => CACHED [anthias-websocket] /bin/sh -c c_rehash                                                                                                                                                         0.0s
 => CACHED [anthias-websocket] /bin/sh -c pip3 install --upgrade pip --break-system-packages &&     pip3 install wheel --break-system-packages                                                             0.0s
 => [anthias-websocket] copy /requirements/requirements-websocket.txt /tmp/requirements.txt                                                                                                                0.1s
 => [anthias-websocket] /bin/sh -c pip3 install -r /tmp/requirements.txt --break-system-packages                                                                                                          18.6s
 => [anthias-celery] copy /requirements/requirements.txt /tmp/requirements.txt                                                                                                                             0.0s
 => [anthias-celery] /bin/sh -c pip3 install -r /tmp/requirements.txt --break-system-packages                                                                                                             82.4s
 => [anthias-websocket] /bin/sh -c mkdir -p /usr/src/app                                                                                                                                                   0.8s
 => [anthias-websocket] mkdir /usr/src/app                                                                                                                                                                 0.2s
 => [anthias-websocket] copy / /usr/src/app/                                                                                                                                                               1.5s
 => [anthias-websocket] exporting to docker image format                                                                                                                                                 113.1s
 => => exporting layers                                                                                                                                                                                   62.4s
 => => exporting manifest sha256:d05d03b2bf8f5196051b94e4a8ed98fb646dd7063a6665d2d6bd860e7fdd2a62                                                                                                          0.0s
 => => exporting config sha256:8c99f02ad11c0c4e8943bac0f71990b879caf5f7e65bcdaf750f25c6565556a6                                                                                                            0.0s
 => => sending tarball                                                                                                                                                                                    50.6s
 => [anthias-celery] /bin/sh -c mkdir -p /usr/src/app                                                                                                                                                      0.6s
 => [anthias-celery] mkdir /usr/src/app                                                                                                                                                                    0.1s
 => [anthias-celery] copy / /usr/src/app/                                                                                                                                                                  0.3s
 => [anthias-celery] exporting to docker image format                                                                                                                                                     85.2s
 => => exporting layers                                                                                                                                                                                   45.8s
 => => exporting manifest sha256:4db3f7ee8df1e20561bcbfaf203ba1a5bbfdf7421a9880384c7555e65a56411e                                                                                                          0.0s
 => => exporting config sha256:827be85a9da33d20d0fa13872ea1cd7afb4b045d2620b47d5ffc55e562e002db                                                                                                            0.0s
 => => sending tarball                                                                                                                                                                                    39.4s
 => [anthias-websocket] importing to docker                                                                                                                                                               27.7s
 => => loading layer 70e068aeb744 379.36MB / 395.88MB                                                                                                                                                     27.7s
 => => loading layer e56babb1d9e3 7.32kB / 7.32kB                                                                                                                                                          5.0s
 => => loading layer 72ee0cd99f1f 65.54kB / 5.84MB                                                                                                                                                         5.0s
 => => loading layer fe1d0d45ce19 227B / 227B                                                                                                                                                              4.1s
 => => loading layer d02fe1f94c77 23.33MB / 25.42MB                                                                                                                                                        4.0s
 => => loading layer e92e4abb7977 132B / 132B                                                                                                                                                              0.8s
 => => loading layer 5f70bf18a086 32B / 32B                                                                                                                                                                0.7s
 => => loading layer 11eff76af01d 32.77kB / 2.62MB                                                                                                                                                         0.5s
 => [anthias-websocket] resolving provenance for metadata file                                                                                                                                             0.0s
 => [anthias-nginx internal] load build definition from Dockerfile.nginx                                                                                                                                   0.1s
 => => transferring dockerfile: 513B                                                                                                                                                                       0.0s
 => [anthias-nginx] local://context                                                                                                                                                                        0.1s
 => => transferring context: 442B                                                                                                                                                                          0.0s
 => [anthias-nginx] local://context                                                                                                                                                                        0.2s
 => => transferring context: 226B                                                                                                                                                                          0.2s
 => [anthias-nginx] /bin/sh -c apt-get update &&     apt-get -y install --no-install-recommends     nginx                                                                                                 20.9s
 => [anthias-celery] importing to docker                                                                                                                                                                  23.3s
 => => loading layer 3963b1dddbaa 603B / 603B                                                                                                                                                             23.2s
 => => loading layer 2244c25c68c6 84.12MB / 89.08MB                                                                                                                                                       23.0s
 => => loading layer ae0f79213186 133B / 133B                                                                                                                                                              0.5s
 => => loading layer 5f70bf18a086 32B / 32B                                                                                                                                                                0.5s
 => => loading layer 547c0046c188 32.77kB / 2.62MB                                                                                                                                                         0.4s
 => [anthias-nginx] copy /docker/nginx/nginx.development.conf /etc/nginx/sites-enabled/                                                                                                                    0.1s
 => [anthias-nginx] /bin/sh -c rm -f /etc/nginx/sites-enabled/default                                                                                                                                      0.4s
 => [anthias-nginx] exporting to docker image format                                                                                                                                                       5.3s
 => => exporting layers                                                                                                                                                                                    2.1s
 => => exporting manifest sha256:c3b03979e75a7446465081e08471723c5a1aec19428b64ac98869f3c0b280e1b                                                                                                          0.0s
 => => exporting config sha256:d02f6dfefa9e244d1dfda4a5775fa4a6664d4c30e26507b3e00d3d0c9053dbf2                                                                                                            0.0s
 => => sending tarball                                                                                                                                                                                     3.1s
 => [anthias-nginx] importing to docker                                                                                                                                                                    0.9s
 => => loading layer 652f7d8a4a6c 229.38kB / 21.82MB                                                                                                                                                       0.9s
 => => loading layer a01230847c9a 732B / 732B                                                                                                                                                              0.1s
 => => loading layer 1477e28840ec 176B / 176B                                                                                                                                                              0.1s
 => [anthias-nginx] resolving provenance for metadata file                                                                                                                                                 0.1s
 => [anthias-celery] resolving provenance for metadata file                                                                                                                                                0.0s
[+] Running 7/7
 ✔ Network anthias_default                Created                                                                                                                                                          0.1s
 ✔ Volume "anthias_anthias-data"          Created                                                                                                                                                          0.0s
 ✔ Container anthias-redis-1              Started                                                                                                                                                          2.5s
 ✔ Container anthias-anthias-server-1     Started                                                                                                                                                          2.5s
 ✔ Container anthias-anthias-websocket-1  Started                                                                                                                                                          2.5s
 ✔ Container anthias-anthias-celery-1     Started                                                                                                                                                          2.5s
 ✔ Container anthias-anthias-nginx-1      Started

[salmanfarisvp]

Tested and working on MAC

MacBook Air
Chip: Apple M1
Memmory: 8 GB
macOS : Sequoia 15.1.1

[vpetersson]

I don't like this approach. It's bad practice to pollute the host system. There's really no reason why you can't run the entire development environment in Docker.

[vpetersson]

Much cleaner.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[nicomiguelino]

@salmanfarisvp, in the meantime, you can try this on your Debian/Ubuntu machine to see if it works.