Exception method for using Airflow with error handling in Slack #28739
Unanswered
emincanoguz11
asked this question in
General
Replies: 5 comments 3 replies
-
|
Could you provide a sample DAG code. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Seems like a discussion/troubleshooting rather than issue - and the description is very difficult to understand. I do not even understand what your problem is @emincanoguz11. Converting to a discussion - and if you need help add the DAG in question and please explain clearly what the problem is. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi again, My problem is basically I want to see reason of dag error in Slack but I can't this. I use following code will see the error. But I see only None parameter as a screenshot. Also I change context.get('exception') to sys.exc_info() and the result is same. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
There is still not enough information but I assume that you set callback on DAG level and not Task level. import pendulum
from airflow import DAG
from airflow.operators.python import PythonOperator
TMP_DIR = "/tmp"
def task_callback(context):
with open(f"{TMP_DIR}/task_callback.log", "w") as fp:
for key, value in context.items():
msg = f"Context key {key!r}, value type {type(value).__name__}, string representation {str(value)!r}"
fp.write(msg + "\n")
print(msg)
def dag_callback(context):
with open(f"{TMP_DIR}/dag_callback.log", "w") as fp:
for key, value in context.items():
msg = (
f"Context key {key!r}, "
f"value type {type(value).__name__}, "
f"string representation {str(value)!r}"
)
fp.write(msg + "\n")
print(msg)
with DAG(
dag_id="example_on_failure_callbacks",
schedule=None,
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
tags=["on_failure", "callbacks"],
on_failure_callback=dag_callback
):
PythonOperator(task_id="failed-task", python_callable=lambda: 1 / 0, on_failure_callback=task_callback)And outputs: dag_callback.logtask_callback.log |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Yes. I want to see dag level log and I want to catch dag level error. |
Beta Was this translation helpful? Give feedback.
-
You could find "DAG level logs" into scheduler logs, something like
There is no DAG level error, see my sample |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for giving information. I want to to see error logs for about dag level. I think context.get('exception') is working with tasks. If tasks are done, context.get('exception') is giving None. When I research for this problem, I can't see any this type of information for variables. Maybe Airflow isn't support to getting dag level logs. I don't know. Also I don't know how I manage this problem. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Airlfow works on task level. Yes. And when you have task failing - you can get dag_id from the task instance. This is as far as it can go. If you have "Slack" tasks - they will have completely different exceptions than non-slack tasks. You should focus on that aspect. You can also write your own on_failure_callback and use it in all your tasks if if you want to make a common way of handling it. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Apache Airflow version
2.5.0
What happened
Hi everyone,
I have a problem with getting error logs in Slack.
I followed this steps in the order.
I take cURL in logs (I get curl scripts in network at Google Chrome) and I see that logs. After than I take jquery and grep for see only errors locally.
I take in dag for this curl but also I see only numbers. I use os.system(). I don’t know that this situation because of I try locally it is works but I don’t know why it is not worked. (I take os.system() also log in Python script at localhost)
So that I researched another way and I found Exception parameter in Airflow but it is not worked because I see only None.
I excepted that the second step is success for getting error logs in Airflow. But it is not worked in Airflow
What you think should happen instead
No response
How to reproduce
Operating System
MacOs
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions