[Critical Issue] Session Table Auto-Increment Mechanism Leads to ID Exhaustion in Airflow Database #45794
-
Apache Airflow version2.10.4 If "Other Airflow 2 version" selected, which one?No response What happened?airflow-web failed with error logs as follow： [SQL: INSERT INTO session (session_id, data, expiry) VALUES (%(session_id)s, %(data)s, %(expiry)s) RETURNING session.id] airflow_2=> SELECT MAX(id) FROM session;
|
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 10 replies
-
|
Through this issue, I believe there is a problem with the use of the auto-increment mechanism for the session table in Airflow's database. The specific reasons are as follows: I already have a scheduled task to execute from airflow.utils.db_cleanup import run_cleanup to clean up the database cache. |
Beta Was this translation helpful? Give feedback.
-
|
As already discussed in #45782 (comment) -> you are doing something terribly wrong to exceed 2 B sessions. This is impractical to achieve in reality As workaround - you should delete all entries from session database. |
Beta Was this translation helpful? Give feedback.
-
|
But you should also find out how it happened that your number of sessions is such huge (impractical to achieve in normal circumstances). |
Beta Was this translation helpful? Give feedback.
-
|
I have figured out why there are so many API requests. It's due to the clb-health requests from our internal CLB component (there is indeed an issue with the requests being too frequent). The request is as follows: [20/Jan/2025:10:50:38 +0800] "HEAD / HTTP/1.1" 302 0 "-" "clb-healthcheck" The issue on my environment has been fixed. I would like to suggest that the Airflow project take this issue into consideration. |
Beta Was this translation helpful? Give feedback.
-
|
Also - since you are already in the situation that your sequence like has such high value you might need to look at which sequence is used in session db and reset it |
Beta Was this translation helpful? Give feedback.
-
|
In fact, due to the abnormal scenario where the API calls from my surrounding load balancing component were excessively frequent (the frequent health checks by the load balancing component are a real-world issue we encountered in our production environment, not a test), I indeed reached this limit after Airflow had been running stably for a long time. This caused the Airflow web component to crash. |
Beta Was this translation helpful? Give feedback.
-
Yes. And this is one of those situations we are not going to handle with cleanup. You need to fix the problem yourself as deployment manager and prevent it from happening in the future. |
Beta Was this translation helpful? Give feedback.
-
|
Just learn from it and move on. |
Beta Was this translation helpful? Give feedback.
-
|
Understood. Thank you for your patient explanation. |
Beta Was this translation helpful? Give feedback.
-
|
It's just impractical to expect that all kinds of abnormal situations will be handled by Airlfow maintainers. As a deployment manager of Airflow it's your responsibilty to handle problems that are caused by misconfiguraiton of your system. There are about a million ways where wrongly configured deployment of Airlfow might cause it to crash - this is just one of them and it's your responsibility to make sure your deployment is not causing it. Also - your problem with too frequent load balancing caused not only auto-increment field exhaution but al zo excessive, unnecessary load on the database. So the fact that your DB eventually crashed, was a GOOD thing because it allowed you to detect a deployment issue you had and excessively used resources of your company. |
Beta Was this translation helpful? Give feedback.
-
|
But really - my sincere advice is to look at what you are doing to create 2B sessions, there is something really terribly wrong in your situation to reach such high sequence value. This mechanims (via |
Beta Was this translation helpful? Give feedback.
-
@claude - I don't believe this is practical - you are the first of many thousands of people who ever hit that limit . But if you feel like it - this is something you might suggest to Here where it is defined: Feel free to open issue in flask-session and describe your case. |
Beta Was this translation helpful? Give feedback.
@claude - I don't believe this is practical - you are the first of many thousands of people who ever hit that limit . But if you feel like it - this is something you might suggest to
flask-sessionproject. This is where both session table and size of the fleld is Integer (which in Postgres is up to 2B: https://www.postgresql.org/docs/current/datatype-numeric.htmlHere where it is defined:
https://github.com/pallets-eco/flask-session/blob/development/src/flask_session/sqlalchemy/sqlalchemy.py#L27
Feel free to open issue in flask-session and describe your case.