Aws-cli: [v2] Add the ability to export AWS environment variables in a shell

[rnhurt]

Some CLI tools (ie. awslogs, Ansible) require (or at least work better) with the standard AWS_ environment variables. It would be nice to be able to export these env vars in a terminal in a quick and easy fashion.

For example, I want to use awslogs to watch my CloudWatch log stream. It supports "profiles" but not the new aws2 profiles. So, something like this won't work:

$ awslogs groups --profile myaccount-adminaccess
You've found a bug! Please, raise an issue attaching the following traceback 
 https://github.com/jorgebastida/awslogs/issues/new
 ....

However, if I were to be able to export the env_vars from aws2, I could use these tools in a more native fashion:

 $ aws2 --profile myaccount-adminaccess export
   AWS_ACCESS_KEY_ID=afsdasfasfasfasfasfasf
   AWS_SECRET_ACCESS_KEY=23432423234244342
   AWS_SESSION_TOKEN=234lj234jkl234lkj24ljk234klj234
 $ awslogs group
 /aws/codebuild/test
 /aws/lambda/mylambda
 ....

[pcolmer]

Or, as I've just suggested in #4676, add an exec command to aws2 so that you could do this:

aws2 --profile myaccount-adminaccess exec awslogs group

aws2 would set the environment variables then run everything after exec as a subprocess so that whatever gets run can retrieve the values.

[pcolmer]

Until the functionality does get added to aws2, this might help:

https://github.com/linaro-its/aws2-wrap

[pcolmer]

I've updated the script so that it is now installable:

pip install aws2-wrap

[benbridts]

Some more context on why this would help simplify things for customers:

• There are a lot of tools, leading to a very fragmented experience: see this tweet about installers and the related issue about conflicting packages.
• Customers are missing an unified experience both in the authentication process too: background about the reason for this, some context about how this got out of hand before
• There is an abundance of tools that do something similar, almost every organization ends up either choosing an existing one or writing their own: tweet and tweet.

I would also really like the exec command

• Some of those tools try to use the existing config files and add extra commands on top of them, anecdotally aws-vault seems one of the most popular ones. It implements exactly that (among other things)
• It also allows developers to not have to care about profiles in their code. If you always use the default credential chain, you can use environment variables to test your program locally and the metadata service when running in AWS.

[benbridts]

Additionally, I'd recommend using -- with exec, it will make it easier to know which flags belong to aws-cli and wich are for the program:

These two should be the same command:

aws2 --profile myaccount-adminaccess --debug exec -- awslogs group
aws2 --profile myaccount-adminaccess exec --debug -- awslogs group

while this one is a different one:

aws2 --profile myaccount-adminaccess exec -- awslogs group --debug

[joguSD]

Marking this as a feature request. I think this could be a useful idea. I think we're more inclined to take the export approach rather than exec. Another idea would be to output the credentials as a JSON document, maybe even in the credential_process format:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html

This would mean any SDK that implements support for credential_process could get support for the CLI's entire credential chain by proxy.

[benkehoe]

This would mean any SDK that implements support for credential_process could get support for the CLI's entire credential chain by proxy.

This is a separate ticket, but I don't think integration in SDKs with e.g., AWS SSO sign in should be happening through credential_process. It would be papering over the lack of support for various configuration options in certain SDKs. Please don't push the work onto us users by requiring us to configure credential_process in our configs; put in the work to bring the SDKs up to par.

[bondsbw]

@joguSD I am trying to use credential_process as you are describing. It doesn't appear to work for some SDKs. Specifically, Terraform fails where the AWS CLI v2 succeeds.

Steps

1. aws sso configure (or aws sso login later)
2. Use credential_process to call a script which:
   
   
   
   • Executes aws sso get-role-credentials
   
   
   • Reformats the JSON to make it compatible with credential_process
   
   
3. Run terraform plan against a module referencing that profile
4. Run an AWS CLI v2 command referencing that profile

Result

Terraform fails with Error: No valid credential sources found for AWS Provider.

AWS CLI v2 succeeds.

Comparison

I can manually create a profile in .aws/config with the same aws_access_key_id, aws_secret_access_key, and aws_session_token. Terraform succeeds.

I already had a script which parses .aws/config to load credentials from a particular profile (originally built to merge multiple config files). I took this code and the manually-generated profile from just above, and piped that back into credentials_process in the same .aws/config file. This process also fails.

Thoughts

This looks like a bug with reading the longer session tokens. The final script I mentioned above works with tokens generated by AWS STS in the AWS SDK for Javascript v2.7.5, and with session tokens generated by aws sts assume-role in CLI v2. These tokens are significantly shorter than those generated by aws sso get-role-credentials.

[pcolmer]

@bondsbw FWIW we are successfully using aws2-wrap (mentioned above) with Terraform. For example:

pipenv run aws2-wrap --profile SSO_Production_TerraformAssumeAllRoles --exec "/opt/terraform/terraform-0.11.14/terraform plan -out TF_Update.plan"

[bondsbw]

@pcolmer Thanks, I'll take a look. But some of my modules require multiple profiles.

[bondsbw]

Update: I found the source of the issue. ~It is described in this PR on the Go SDK: https://github.com/aws/aws-sdk-go/pull/3238~ (Current version of SDK already fixed this issue, but the Terraform AWS provider relies on an older SDK.)

The session token is now long enough that the standard-format JSON is above the maximum buffer size of 1024 bytes. ~The PR increases the maximum buffer size to 2048 bytes.~

A workaround is to use minified JSON, which reduces the total size to 1010 bytes and fits in the buffer.

---

Edit: Created an issue in aws-sdk-go-base.

[benkehoe]

I wrote a simple tool to export credentials using botocore, so it will work with whatever credentials botocore understands (role assumption, credential process, EC2 instance metadata, etc.): aws-export-credentials.

It doesn't work with AWS SSO credentials, because botocore in PyPI is still v1. But I made aws-sso-credential-process for that purpose, and it allows SDKs that understand credential_process to use SSO credentials with no further modifications, so hopefully with that, exporting the credentials and injecting them otherwise isn't necessary.

@joguSD we still need proper AWS SSO credential support, including dispatching authentication to the browser, in all the SDKs. Customers should never have to build these kinds of tools for ourselves.

[benkehoe]

@joguSD Another reason not to use credential_process as a way of providing AWS SSO support to the SDKs is that the SDK captures stdout and stderr, so the tool is not generally able to show any output to the user during the authentication process. I encountered this with aws-sso-credential-process where the normal display of the verificationUri and userCode as a backup to the browser opening can't be shown.