Scheduler keeps dying - Killing DAGFileProcessorProcess

[gabrielboehme]

Apache Airflow version

2.4.3

What happened

I'm running airflow on Docker (inside Windows WSL2) with Celery Executor. Suddenly, from one day to another, my airflow scheduler started to die. I have this specific dag file that it creates dynamically 53 dags from a db. Even if I comment out this file and leave the remaining ones, airflow will kill the scheduler.

This behavior occurs when all containers are healthy after start up.

Error:
{processor.py:233} WARNING - Killing DAGFileProcessorProcess (PID=<PID>)

I checked and that process exists in the container before it's killed.

=== EDIT===
I saw that uncommenting that dag, makes the scheduler not being killed. The thing is: I wasnt getting this behavior until today. Nothing changed (including airflow cfg, amount of dags, etc).

What you think should happen instead

My airflow scheduler should be running as usual, without errors.

How to reproduce

You can see this dynamic dag that I think its breaking everything here in this Gist: https://gist.github.com/gabrielboehme/1e62cb5e4b26633130882544d59ad058

Also, the dockerfile, docker-compose and .env file are in the gist above.

Operating System

Ubuntu 20.04 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==4.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-docker==3.1.0
apache-airflow-providers-elasticsearch==4.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-azure==4.2.0
apache-airflow-providers-mysql==3.2.0
apache-airflow-providers-odbc==3.1.1
apache-airflow-providers-postgres==5.2.1
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==4.0.0
apache-airflow-providers-slack==5.1.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.1.0

Deployment

Docker-Compose

Deployment details

Docker-compose, Dockerfile, and .env can be found here:
https://gist.github.com/gabrielboehme/32ada28b251d758ef5ce334444e0d433

Docker version:
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 3056208
Built: Tue Oct 25 18:00:04 2022
OS/Arch: linux/amd64

Docker Compose version v2.12.2

Anything else

Seems that every 40 or 60 seconds the airflow-scheduler gets killed.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

This is at most a discussion, not airflow issue. You need to look further in your docker compose who and why kills it. Maybe you run out of resources? Disk? maybe your DAG started to use more memory - there all kinds of reasons why it coudl have failed but this is not an airflow issue, but something you need to investigate. Look at prerequisites, cleanup your docker etc. etc.

[gabrielboehme]

Things I've tried to mitigate the issue:

• Uninstalled/installed docker and docker-compose.
• When I change logging to DEBUG instead of info, no relevant information is provided about WHY it's getting killed. -
• I've added 50% more memory to docker since the begging of the issue and nothing changed. As I commented, it's not about resources being used heavily used, since it was running perfectly fine and the DB didn't change.

Based on the provided airflow env variables, can you (@potiuk) point me to which ones should help to eliminate this?

[potiuk]

No @gabrielboehme I can't.

[potiuk]

There is simply nnot enough information. SOMETHING must have changed - new update of windows, new docker version etc. Your dag, If you have not changed anything - it must be something that changed - environmentally, for example. But what it was - you are the only person who can posibly know. All the other people might just guess.

[potiuk]

I am not a wizard. I have no crystal ball simply :(