Airflow task finished successfully according to logs but task status was never updated

[lozuwa]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow 2.2.4

We have a task that runs a slack notification which usually takes 1 to 3 seconds to finish. However, we noticed it was stuck on queued state for some hours. Below I provide some logs of the scheduler, there it can be seen the task was correctly queued and sent to the worker. However, when it came back as successful, it didn't update the task state. As can be seen in the latest log below.

Also, even though the task says successful, it didn't execute. The slack notification was never sent.

Nov 17, 2022 @ 13:02:16.203 | [2022-11-17 17:02:16,203] {scheduler_job.py:570} INFO - TaskInstance Finished: dag_id=***, task_id=***, run_id=scheduled__2022-11-17T16:00:00+00:00, run_start_date=None, run_end_date=None, run_duration=None, state=queued, executor_state=failed, try_number=1, max_tries=1, job_id=None, pool=default_pool, queue=default, priority_weight=1, operator=SlackAPIPostOperator
-- | --

  | Nov 17, 2022 @ 13:02:15.943 | [2022-11-17 17:02:15,942] {scheduler_job.py:473} INFO - Sending TaskInstanceKey(dag_id='***', task_id='***', run_id='scheduled__2022-11-17T16:00:00+00:00', try_number=1) to executor with priority 1 and queue default

  | Nov 17, 2022 @ 13:02:15.943 | [2022-11-17 17:02:15,943] {base_executor.py:82} INFO - Adding to queue: ['airflow', 'tasks', 'run', '***', '***', 'scheduled__2022-11-17T16:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/***.py']

What you think should happen instead

The task updates the status to completed and doesn't hang in queued state.

How to reproduce

This happens randomly.

Operating System

Official docker image

Versions of Apache Airflow Providers

None, custom code.

Deployment

Official Apache Airflow Helm Chart

Deployment details

Helm, kubernetes, celery executor.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ephraimbuddy]

Please can you try the dag on the latest airflow and give feedback? Can't reproduce

[potiuk]

Waiting for user to confirm - converted to discussion.