Getting 400 when using airflow rest api to trigger airflow dag

[vimalgalani28]

I am using airflow version 2.9.2 and trying to trigger dag using airflow rest api. PFA curl:

curl --request POST \
  --url http://localhost:9000/api/v1/dags/test_dag/dagRuns \
  --header 'Authorization: Basic token' \
  --header 'Content-Type: application/json' \
  --data '{
	"conf": {
		"version": "1"
	}
}'

Getting response as below:

{
	"detail": "{'run_id': ['Missing data for required field.'], 'dag_id': ['Missing data for required field.'], 'run_type': ['Missing data for required field.'], 'logical_date': ['Unknown field.'], 'dag_run_id': ['Unknown field.']}",
	"status": 400,
	"title": "Bad Request",
	"type": "https://airflow.apache.org/docs/apache-airflow/2.9.2/stable-rest-api-ref.html#section/Errors/BadRequest"
}

I checked airflow server logs. It gives me error as below:
│ <unknown>:160 SyntaxWarning: invalid escape sequence '\;'

Checked airflow docs. The only required field is conf and dag id in request param. Am I missing something?

[potiuk]

I think you need to show the whole stacktrace if you want to get help, it's hard to say who and where is causing this syntax error. My guess it's something about your DAG definition of parameters, but it's just a guess.

[vimalgalani28]

@potiuk Which stacktrace are you referring to? scheduler or web server?

[potiuk]

Whatever will help those who are looking at it loose less time when they are trying to help people like you in their free time

[potiuk]

It's basically on you to provide enough information to get helped - you are asking for help here.

[nrobinson-intelycare]

I was looking forward to using this to allow GitHub Actions to trigger DAG runs from the Airflow REST API, but I am getting the same error as @vimalgalani28 on Airflow v2.10.4, using Basic authentication:
https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/api-authentication.html#basic-authentication

Request with cURL:

curl --request POST \
--url http://AIRFLOW_HOST:8080/api/v1/dags/manual_dbt_airflow_run_batch/dagRuns \
--user 'API_USER:API_PASSWORD' \
-H 'Content-Type: application/json' \
-d '{}'

Response:

{
  "detail": "{'run_id': ['Missing data for required field.'], 'dag_id': ['Missing data for required field.'], 'run_type': ['Missing data for required field.'], 'logical_date': ['Unknown field.'], 'dag_run_id': ['Unknown field.']}",
  "status": 400,
  "title": "Bad Request",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.10.4/stable-rest-api-ref.html#section/Errors/BadRequest"
}

Here are the only logs I see for the request:

Jan 14 16:35:57 ip-10-105-6-62.ec2.internal airflow-webserver[172469]: [2025-01-14T16:35:57.526+0000] {override.py:1679} INFO - Updated user API User
Jan 14 16:35:57 ip-10-105-6-62.ec2.internal airflow-webserver[172469]: 100.64.0.3 - API_USER [14/Jan/2025:16:35:57 +0000] "POST /api/v1/dags/manual_dbt_airflow_run_batch/dagRuns HTTP/1.1" 400 396 "-" "curl/8.7.1"

@potiuk Could we please reopen this discussion?

[nrobinson-intelycare]

@potiuk

I have a suspicion this is a poorly handled error from dag.create_dagrun:
https://github.com/apache/airflow/blame/main/airflow/api_fastapi/core_api/routes/public/dag_run.py#L371

Switching up the request body yields some interesting responses.

curl \
--url http://AIRFLOW_HOST:8080/api/v1/dags/manual_dbt_airflow_run_batch/dagRuns \
--user 'API_USER:API_PASSWORD' \
-H 'Content-Type: application/json' \
-d '{"run_id":"", "dag_id":"", "run_type":""}'

{
  "detail": "Property is read-only - 'dag_id'",
  "status": 400,
  "title": "Bad Request",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.10.4/stable-rest-api-ref.html#section/Errors/BadRequest"
}

curl \
--url http://AIRFLOW_HOST:8080/api/v1/dags/manual_dbt_airflow_run_batch/dagRuns \
--user 'API_USER:API_PASSWORD' \
-H 'Content-Type: application/json' \
-d '{"run_id":""}'

{
  "detail": "{'dag_id': ['Missing data for required field.'], 'run_type': ['Missing data for required field.'], 'logical_date': ['Unknown field.'], 'dag_run_id': ['Unknown field.']}",
  "status": 400,
  "title": "Bad Request",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.10.4/stable-rest-api-ref.html#section/Errors/BadRequest"
}

curl \
--url http://AIRFLOW_HOST:8080/api/v1/dags/manual_dbt_airflow_run_batch/dagRuns \
--user 'API_USER:API_PASSWORD' \
-H 'Content-Type: application/json' \
-d '{"run_id":null, "logical_date":"2025-01-15T21:31:12Z"}'

{
  "detail": "{'run_id': ['Field may not be null.'], 'dag_id': ['Missing data for required field.'], 'run_type': ['Missing data for required field.'], 'logical_date': ['Unknown field.'], 'dag_run_id': ['Unknown field.']}",
  "status": 400,
  "title": "Bad Request",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.10.4/stable-rest-api-ref.html#section/Errors/BadRequest"
}

Is this a known issue on the roadmap?

[potiuk]

We don't have "roadmap" for issues. This is open source. If someone wants to take on an issue, they do. You can contribute if you want, or you can create a detailed issue describing it and label it with "good first issue" and maybe someone will pick it up eventually and solve.

[potiuk]

Generally speaking - all is open - if you search for issues and do not find it created, then you can safely create a new one - you have as much information as anyone else - the issues here are the ... database of issues that are known. If someone asked to be assigned to an issue, we assign them - which might, or might not mean they are working on it.

However, the most certain way of making sure an issue is addressed is to contribute a PR fixing it, otherwise it really depends on whether someone picks an interest and starts working on it - and it's a decision of contributing individual.

[nrobinson-intelycare]

Understood, but I imagine this is a popular API endpoint, and I'm curious if other users have encountered this same behavior @vimalgalani28 and I have seen.

I'll do some more digging on my end, but as a stopgap, I've opted to run airflow dags trigger over SSH, which works pretty well.

Do I need to be running airflow dags trigger on the same machine as the scheduler? As long as the Airflow database is reachable, airflow dags trigger should work right?

[potiuk]

Open an issue please with all the details I think it's just a bad error description in documentation with a specifc content of your API call. And yes it's a popular API and it generally works for others, so I am not sure what the problem is in your case. And yes if you have machine configured same way and with same credentials and access to the DB, it shold work from any machine.