Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catchup=FALSE but it still run for missing Task? #45274

Open
2 tasks done
amzar96 opened this issue Dec 30, 2024 · 1 comment
Open
2 tasks done

Catchup=FALSE but it still run for missing Task? #45274

amzar96 opened this issue Dec 30, 2024 · 1 comment
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@amzar96
Copy link

amzar96 commented Dec 30, 2024

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.2

What happened?

Hi All,

Why does my DAG still behave like catchup=True and run for the missing DAG run when I change the schedule interval? I already set it to False, but why is it still running for the missing DAG run?

DAG code:

with DAG(
    dag_id=dag_id,
    schedule=schedule_value,
    start_date=datetime(2024, 12, 1),
    catchup=False,
    tags=["xxx"],
    default_args=default_args,
) as dag:
    schemas = variable_mapping["data"]

    with TaskGroup(
        group_id="compare",
        tooltip="vs",
    ) as column_compare_tg:
        for schema in list(schemas.keys()):
            create_task(
                env_vars, job_name=job_name, redshift_schema=schema, _type="compare"
            )

    send_alert = create_task(env_vars, job_name=job_name, _type="alert")

    column_compare_tg >> send_alert

What you think should happen instead?

When I changed the schedule interval, the DAG should skip the missing run and run only one task.

How to reproduce

  1. Create a DAG with schedule_value as a Airflow variable
  2. First time: set the schedule as 2 time a day
  3. Second time: set the schedule as 5 time a day

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.28.0
apache-airflow-providers-celery==3.8.1
apache-airflow-providers-cncf-kubernetes==8.4.1
apache-airflow-providers-common-compat==1.2.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.16.0
apache-airflow-providers-docker==3.13.0
apache-airflow-providers-elasticsearch==5.5.0
apache-airflow-providers-fab==1.3.0
apache-airflow-providers-ftp==3.11.0
apache-airflow-providers-google==10.22.0
apache-airflow-providers-grpc==3.6.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.0
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure==10.4.0
apache-airflow-providers-mysql==5.7.0
apache-airflow-providers-odbc==4.7.0
apache-airflow-providers-openlineage==1.11.0
apache-airflow-providers-postgres==5.12.0
apache-airflow-providers-redis==3.8.0
apache-airflow-providers-sendgrid==3.6.0
apache-airflow-providers-sftp==4.11.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-snowflake==5.7.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

helm, on AWS EKS

Anything else?

No.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@amzar96 amzar96 added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Dec 30, 2024
Copy link

boring-cyborg bot commented Dec 30, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Development

No branches or pull requests

1 participant