Airflow: Add SecureString Decryption to AWS Secrets Backend

[pcandoalmeida]

Description

Add AWS Secrets Manager secrets backend support for SecureString parameters and include role based authentication when connecting to AWS via the SecretsManagerBackend class.

Use case / motivation

PR #8186 added support for AWS Secrets Manager as a secrets backend within Airflow. SecureStrings currently do not get decrypted. From what I can also tell from the source code, the current implementation requires an AWS profile (and therefore, I am assuming a key/secret).

I therefore wanted to see if I could add the ability to retrieve and make use of decrypted SecureString parameters and use non credentials based authentication (using a role, for example, to authenticate the connection to AWS SSM).

This change would help my use of Airflow within my present company.

The first issue I think is known and discussed in the above PR. The latter could perhaps do with some discussion as I could be wrong about the use of profile within SecretsManagerBackend.

Thank you!

Related Issues

• #8186

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Sounds like really useful feature! Maybe you would like to contribute it yourself :)? It does not seem too complex and we are happy to guide with the contribution process @pcandoalmeida :)

[pcandoalmeida]

Hi @potiuk sure, I'd love to.

[potiuk]

You can start with https://github.com/apache/airflow/blob/master/CONTRIBUTING.rstt

We also run "First time contributor's" workshops (next one will be likely in September) but you can take a look at the https://cwiki.apache.org/confluence/display/AIRFLOW/First+time+contributor%27s+workshop (a little outdated but still good).
For the development environment there is recently refreshed movie - you can find all information in https://github.com/apache/airflow/blob/master/BREEZE.rst .

I will be on holidays next week, but there are other commiters around who can help if you have any questions.

[kaxil]

Assigned that issue to you @pcandoalmeida , let me know if you need any help :)

[kaxil]

To authenticate you can either supply a profile name to reference aws profile, e.g. defined in ~/.aws/config or set
environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.

You could also pass aws_access_key_id, aws_secret_access_key (all the keyword arguments mentioned in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session can be pass to backend_kwargs)

https://airflow.readthedocs.io/en/latest/howto/use-alternative-secrets-backend.html#aws-secrets-manager-backend

[pcandoalmeida]

Hi @kaxil thanks for this. I think in the way that I was trying to use it, I didn't have an access key or secret as we've moved away from them in my present company. I did try and pass a role ARN into ~/.aws/config but for some reason I was getting some issues. I pinged across a quick question on Slack on the new users channel to see if I was using the backend correctly, but I'm not sure I've had a reply just yet. In any case, I'll get started on this and see how I get on!

[pcandoalmeida]

Closing this issue as a fix has been applied in 1.10.11 😀

[potiuk]

Cool ! Thanks for closing 👍