Airflow: Airflow support for S3 compatible storages

[vaddisrinivas]

Hi,

Curious to know about the support for S3 compatible storages like DELL ECS, MINIO ETC

Thanks

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[dimon222]

Yes, it works as long as you specify endpoint url. However, it seems to be broken to serving logs in UI for some users on latest release 1.10.10 (works fine in 1.10.9)

[vaddisrinivas]

It doesnt work for me even as I continue to provide HOST in the connection and also other relevant parameters.
If there is any alternative for the same, please help with that.
Also @dimon222, can you please share a sample connection screenshot/ instruction along with configurations thus enabled to facilitate remote logging?
Thanks.

[dispensable]

We are currently using Ceph RGW as our airflow cluster logging backend. It works perfect. But setting the connection is a little bit tricky U should not set host in the host form field otherwise set host in the Extra fields with {"host": "http://YOUR_S3_URL:PORT"}, just leave the host/port/schema form fields blank.

[vaddisrinivas]

hi @dispensable , will try this and update the ticket asap!

[vaddisrinivas]

hi @dispensable / all,

How do I pass the access_key_id and other parameters to connect to that? can someone please help on that?

[dimon222]

hi @dispensable / all,

How do I pass the access_key_id and other parameters to connect to that? can someone please help on that?

Username/Password fields in respective connection
Extra args I believe not supported apart of mentioned above host.

[vaddisrinivas]

hi @dispensable/@dimon222, thanks for your help.

I have SUCCESSFULLY enabled pushing airflow logs to s3 compatible bucket by following what was mentioned above,
in brief :

• created a new connection with -> {"host": "http://myhost:myport", "aws_access_key_id" :"myaccesskey/username"
  ,"aws_secret_access_key": "myreallybigsecretkey"}

• then I went ahead, modified the configuration by adding
  AIRFLOW__CORE__REMOTE_LOGGING: True,
  AIRFLOW__CORE__REMOTE_LOG_CONN_ID: connectionthatwassetearlier,
  AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER: "s3://bucketname/pathorfolderonBucket",

• restarted Airflow!

[Asgoret]

Everywhere is HTTP, but what about HTTPS? airflow doesn't support HTTPS S3 endpoints? I've got a very odd error (An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records) when use HTTPS-based endpoint. In dev with HTTP, all works perfectly.
cc @dimon222

ok, how it works.
Connection in GUI:

Name: Some name
Type: S3
Host: <empty>
Schema: <empty>
Login: Your ID from minio
Password: You password key
Port: <empty>
Extras: {"host": "https://domain"}