Error running a container

[AadarshBhalerao]

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I have created a Dockerfile

# Use the official Airflow image as the base
FROM apache/airflow:2.9.3

# Switch to root to install PostgreSQL and create directories with proper permissions
USER root

# Install PostgreSQL and other necessary packages
RUN apt-get update && \
    apt-get install -y postgresql postgresql-contrib && \
    apt-get clean

# Create a new user for PostgreSQL
RUN service postgresql start && \
    sudo -u postgres psql -c "CREATE USER airflow WITH PASSWORD 'airflow';" && \
    sudo -u postgres psql -c "CREATE DATABASE airflow OWNER airflow;"

# Expose the PostgreSQL port
EXPOSE 5432

# Switch back to the airflow user
USER airflow

# Set environment variables to configure Airflow to use PostgreSQL
ENV AIRFLOW__CORE__EXECUTOR=LocalExecutor
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost/airflow
ENV AIRFLOW__CORE__FERNET_KEY=FB0o_zt4e3Ziq3LdUUO7F2Z95cvFFx16hU8jTeR1ASM=
ENV AIRFLOW__CORE__LOAD_EXAMPLES=False
ENV AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth
ENV AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK=True

# Copy your dags, plugins, and other configuration files if needed
COPY ./dags /opt/airflow/dags
COPY ./plugins /opt/airflow/plugins
COPY ./config /opt/airflow/config
COPY ./logs /opt/airflow/logs

# Command to start PostgreSQL, initialize Airflow database, create admin user, start scheduler, and webserver
CMD bash -c "service postgresql start && \
airflow db init && \
airflow users create --role Admin --username admin --password admin --email [email protected] --firstname admin --lastname admin && \
airflow scheduler & airflow webserver"

I created an image docker build -t my-airflow .

And then a container docker run --name airflow-container -d -p 8080:8080 -p 5432:5432 -v /path/to/dags:/opt/airflow/dags -v /path/to/logs:/opt/airflow/logs -v /path/to/plugins:/opt/airflow/plugins my-airflow

When I opened logs in docker desktop I see these errors

2024-07-23 19:38:00 ....................
2024-07-23 19:38:00 ERROR! Maximum number of retries (20) reached.
2024-07-23 19:38:00 
2024-07-23 19:38:00 Last check result:
2024-07-23 19:38:00 $ airflow db check
2024-07-23 19:38:00 /home/airflow/.local/lib/python3.12/site-packages/airflow/configuration.py:764 FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
2024-07-23 19:38:00 Unable to load the config, contains a configuration error.
2024-07-23 19:38:00 Traceback (most recent call last):
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
2024-07-23 19:38:00     os.mkdir(self, mode)
2024-07-23 19:38:00 FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2024-07-23'
2024-07-23 19:38:00 
2024-07-23 19:38:00 During handling of the above exception, another exception occurred:
2024-07-23 19:38:00 
2024-07-23 19:38:00 Traceback (most recent call last):
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/logging/config.py", line 581, in configure
2024-07-23 19:38:00     handler = self.configure_handler(handlers[name])
2024-07-23 19:38:00               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/logging/config.py", line 854, in configure_handler
2024-07-23 19:38:00     result = factory(**kwargs)
2024-07-23 19:38:00              ^^^^^^^^^^^^^^^^^
2024-07-23 19:38:00   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/log/file_processor_handler.py", line 53, in __init__
2024-07-23 19:38:00     Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/pathlib.py", line 1315, in mkdir
2024-07-23 19:38:00     self.parent.mkdir(parents=True, exist_ok=True)
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
2024-07-23 19:38:00     os.mkdir(self, mode)
2024-07-23 19:38:00 PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'
2024-07-23 19:38:00 
2024-07-23 19:38:00 The above exception was the direct cause of the following exception:
2024-07-23 19:38:00 
2024-07-23 19:38:00 Traceback (most recent call last):
2024-07-23 19:38:00   File "/home/airflow/.local/bin/airflow", line 5, in <module>
2024-07-23 19:38:00     from airflow.__main__ import main
2024-07-23 19:38:00   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__init__.py", line 74, in <module>
2024-07-23 19:38:00     settings.initialize()
2024-07-23 19:38:00   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/settings.py", line 531, in initialize
2024-07-23 19:38:00     LOGGING_CLASS_PATH = configure_logging()
2024-07-23 19:38:00                          ^^^^^^^^^^^^^^^^^^^
2024-07-23 19:38:00   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 74, in configure_logging
2024-07-23 19:38:00     raise e
2024-07-23 19:38:00   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 69, in configure_logging
2024-07-23 19:38:00     dictConfig(logging_config)
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/logging/config.py", line 920, in dictConfig
2024-07-23 19:38:00     dictConfigClass(config).configure()
2024-07-23 19:38:00   File "/usr/local/lib/python3.12/logging/config.py", line 588, in configure
2024-07-23 19:38:00     raise ValueError('Unable to configure handler '
2024-07-23 19:38:00 ValueError: Unable to configure handler 'processor'
2024-07-23 19:38:00

This is my folder structure

What you think should happen instead?

No response

How to reproduce

Please follow the files shared above

Operating System

WSL

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[AadarshBhalerao]

Hi @potiuk can someone from your team please help me with this issue. I can share more details if needed. Thanks!

[potiuk]

Just to clarify. I have no "team" - there are maintainers who can approve changes but ANYONE in the community can answer and help other users raising an issue or discussion - and pointing out one person to be responsible for it to happen is not the way it works. You need to patiently wait until somoene who decided to spend some of their free time to help you - and sometimes you will not get help - this is an open-source project and people contribute and help others here voluntarly - when they see they can help.

So please avoid passing responsibility on somoene else to take care of finding someone who can help you - especially - that I am individual contributor here and the team is not mine.

[potiuk]

BTW. converted it to a discussion - because it's not really Airflow issue, it's your troubleshooting

[potiuk]

And clearly - you have a permission issue here. Apparently - your user that you use for running your container does not have permission to write and create files in the folder it shoudl have /opt/airflow otherwise pointed to by AIRFLOW_HOME variable. You should make sure that the deployment you have allows to do it - but how - it depends on how you run container, what volumes you mount and what permissions you have - but there is no way to tell what you do - because you have not explained your deployment here.

[potiuk]

Specificallhy - what files you mounted there, which filesystem and which permissions it has etc. etc. Generally speaking - when you have WSL, you should make sure you use the "linux" part of it exclusively. We explain that in development guides (because WSL should only possibly be used by development). Airflow makes certain assumptions about it's environement being "linux" and if you mix them (for example by using Windows volumes, things like this might happen.

[andreabisellonamirial]

I have the same problem, using the standard aiflow docker compose.

Following https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml

1. I have created an empty folder
2. I have downloaded https://airflow.apache.org/docs/apache-airflow/2.10.4/docker-compose.yaml
3. I have executed docker compose up airflow-init
4. I have executed docker compose up , and everything is up and running5.

but accessing the web interface, I obtain the error

2025-01-16 16:42:53 PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'
2025-01-16 16:42:53
2025-01-16 16:42:53 The above exception was the direct cause of the following exception:
2025-01-16 16:42:53
2025-01-16 16:42:53 Traceback (most recent call last):
2025-01-16 16:42:53 File "/home/airflow/.local/bin/airflow", line 5, in
2025-01-16 16:42:53 from airflow.main import main
2025-01-16 16:42:53 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/init.py", line 74, in
2025-01-16 16:42:53 settings.initialize()
2025-01-16 16:42:53 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/settings.py", line 785, in initialize
2025-01-16 16:42:53 LOGGING_CLASS_PATH = configure_logging()
2025-01-16 16:42:53 ^^^^^^^^^^^^^^^^^^^
2025-01-16 16:42:53 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 74, in configure_logging
2025-01-16 16:42:53 raise e
2025-01-16 16:42:53 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/logging_config.py", line 69, in configure_logging
2025-01-16 16:42:53 dictConfig(logging_config)
2025-01-16 16:42:53 File "/usr/local/lib/python3.12/logging/config.py", line 942, in dictConfig
2025-01-16 16:42:53 dictConfigClass(config).configure()
2025-01-16 16:42:53 File "/usr/local/lib/python3.12/logging/config.py", line 615, in configure
2025-01-16 16:42:53 raise ValueError('Unable to configure handler '
2025-01-16 16:42:53 ValueError: Unable to configure handler 'processor'
2025-01-16 16:42:53

So now i'm trying to solve the problem with the permissions ... struggling with chmod that can lock the folder, etc.

It is normal that the official docker compose file is giving to me this kind of problems?

[potiuk]

It is normal that the official docker compose file is giving to me this kind of problems?

This docker compose is not "official" compose. It's definitely not intended to work in all cicumstances on all environments and with all configuraitons and when you decide to use docker-compose, you need to know quite a bit of details how it works and how to debug any problems with it.

This docker compose is a quick-start that allows you to get up and running with docker-compose - but it's neither comprehensive nor guaranteed in all kind of cases. Specifically, if you follow all the steps includng https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user - it has been tested and works in one of the configurations of docker - specifically on Linux Bookworm and standard docker installation.

If you have any other OS's, docker-composes, volume configuraitons etc. it might or might not work. Generally docker compose is a tool for developers to manage their multiple containers and know how to modify the compose and debug issues with containers.

For example a number of container engines on Mac OS on Windows have different ways of sharing their volumes with virtual machines and they might cause some permission problems. It's usually something that can be searched for and fixed by configuring things better.

So - answering your questions - yes in a number of cases and environments, this docker compose MAY not work.

[andreabisellonamirial]

Ok clear. i was able to make it work on windows 11 adding the .env file, and also creating the folders before running the script. I'm gonna try to understand which one was the solution, and then by a pull request update the documentation for the community. thankyou.