S3 keys provided by aws/log/s3_task_handler.py to s3 hook contain the prefix s3://, causing minio bad request.

[spelufo]

Apache Airflow version

2.3.1 (latest released)

What happened

We've been trying to configure remote_logging to minio with the s3_task_handler, as described here. We test on a local k8s kind cluster, with the airflow helm chart. After some debugging and verifying that the connection can be made and authenticated, we see the following from the minio logs:

2022-05-26T16:50:42:000 [400 Bad Request] s3.HeadObject minio:9000/airflow-logs/s3%3A//airflow-logs/hello_world_dag/print_date/2022-05-26T19%3A50%3A08.676103%2B00%3A00/1.log 192.168.189.192    40.828ms     ↑ 131 B ↓ 293 B

You'll note that the url contains s3%3A//. I think this is the cause of the bad request. Running the following snippet from the airflow webserver pod in a python shell reproduces the bad request, while doing the same without the prefix succeeds.

from airflow.configuration import conf
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

remote_conn_id = conf.get('logging', 'REMOTE_LOG_CONN_ID')
hook = S3Hook(remote_conn_id, transfer_config_args={"use_threads": False})
c = hook.get_conn()
c.head_object(Bucket='airflow-logs', Key='airflow-logs/airflow/logs')  # ok
c.head_object(Bucket='airflow-logs', Key='s3://airflow-logs/airflow/logs')  # raises bad request error

In this test I see that airflow might think having the prefix in the key is fine. The moto mock probably handles it, perhaps s3 does as well. Minio doesn't seem to handle it. The S3 documentation for object keys seem to imply that the prefix is not part of the key.

We've tried removing the prefix from the remote_base_log_folder which reading through the code looks like it is the source of the value of key, but then airflow has no way of knowing what logging backend to use, and running migrations fails.

What you think should happen instead

Airflow shouldn't pass s3 keys with the s3:// prefix to head_object.

How to reproduce

No response

Operating System

Kind cluster, kubernetes 1.20.7

Versions of Apache Airflow Providers

apache-airflow-providers-airbyte==2.1.1
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==3.0.0
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[uranusjr]

It would be a good idea to check with Minio maintainers and potentially Amazon about this. Would you be interested in doing this?

[spelufo]

I tested it against S3, and what happens is that it does work, but it interprets the key as a path, creating folders called s3:/, /, ..., not what should be expected to happen. Unless I'm configuring airflow wrong somehow I think this behavior is a bug in airflow. I think I configured everything in accordance with the docs, but let me know how I can help or if you can't reproduce the issue

I don't think it's necessary to contact minio about the airflow issue. They might want to do the same that S3 does with this kind of key instead of responding with bad request, but after testing against S3 I think that's a separate issue from airflow sending keys with the s3://{bucket_name} prefix. I'll open a minio issue linking here to let them know this.

[spelufo]

Update: This only happens when the connection is specified using the URI format (through environment variables). When using the json encoding everything works fine.

[potiuk]

nio:9000/airflow-logs/s3%3A//airflow-logs/hello_world_dag/print_date/2022-05-26T19%3A50%3A08.676103%2B00%3A00/1.log 192.168.189.192 40.828ms ↑ 131 B ↓ 293 B

Could you type in exact example of the URL/variable you use ?
I believe it might be that you need DOUBLE PERCENT ESCAPING. 😱 😱 😱 in this case.

[o-nikolas]

Hey @spelufo,

As @potiuk suggested, can you provide screenshots or text snippet of exactly how you're configuring the connection for logging as well as the config necessary for remote logging (i.e. how you're specifying remote_base_log_folder).

I tested on my end specifying remote_base_log_folder in airflow.cfg and exporting as an env var and both worked correctly for me (creating an s3 path which does not include the s3:// URI prefix)

@eladkal, @kaxil
Can you add the pending-response label to this issue, thanks!

[potiuk]

Since there was no answer from the user when asked twice, converting it into discussion.