Aws-cli: Client is not using AWS_PROFILE correctly

[garrethmcdaid-zd]

aws-cli/1.15.12 Python/2.7.10 Darwin/16.7.0 botocore/1.10.12

If AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are set as env variables, the client appears to ignore the AWS_PROFILE environment variable.

Steps to re-produce:

export AWS_ACCESS_KEY_ID='yyy'
export AWS_SECRET_ACCESS_KEY='xxx'
unset AWS_PROFILE

aws configure list

Name                    Value             Type    Location
----                    -----             ----    --------
profile                <not set>             None    None

export AWS_PROFILE=foo

aws configure list
Name                    Value             Type    Location
----                    -----             ----    --------
profile                  foo           manual    --profile

Expected:

aws configure list
Name                    Value             Type    Location
----                    -----             ----    --------
profile                  foo           env       AWS_PROFILE

Profile is only set when --profile parameter is used. Profile does not change when AWS_PROFILE is set/updated.

[garrethmcdaid-zd]

This behaviour occurs when AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are set in your env.

When these env vars are unset, the AWS_PROFILE environment variable works as expected.

[joguSD]

I wasn't able to reproduce this.
With all 3 env vars set I got output like this:

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                     foo           manual    --profile
access_key     ****************XXXX              env    
secret_key     ****************XXXX              env    
    region                us-east-1      config-file    ~/.aws/config

It picks up on the profile, but still sources credentials from the environment variables, which is the expected behavior.
With all 3 set and providing a --profile:

$ aws configure list --profile foo
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                    foo             manual    --profile
access_key     ****************XXXX shared-credentials-file    
secret_key     ****************XXXX shared-credentials-file    
    region                us-east-1      config-file    ~/.aws/config

As for why it says manual --profile I believe this is just an issue with how the configure list customization is determining if it come from the --profile arg or from the env:
https://github.com/aws/aws-cli/blob/develop/awscli/customizations/configure/list.py#L57-L62

The credentials and profiles should still behave correctly, the list command is just slightly inaccurate. Is this what you're seeing as well?

[garrethmcdaid-zd]

I can still re-produce the issue:

Current environment:

15278-gmcdaid:~ gmcdaid$ env | grep AWS
AWS_PROFILE=staging
AWS_DEFAULT_OUTPUT=json
AWS_DEFAULT_REGION=us-west-1
AWS_SECRET_ACCESS_KEY=*****
AWS_ACCESS_KEY_ID=******

Verify AWS_PROFILE is set

15278-gmcdaid:~ gmcdaid$ echo $AWS_PROFILE
staging
````

Test identity with --profile, verifying thaty "staging" profile is returned:

15278-gmcdaid:~ gmcdaid$ aws sts get-caller-identity --profile staging
{
"Account": “*6847",
"UserId": "AROAIUCUJYYZG67XJELOY:botocore-session-*2088",
"Arn": "arn:aws:sts::*6847:assumed-role/Admin/botocore-session-**2088"
}

Verify AWS_PROFILE still set:

15278-gmcdaid:~ gmcdaid$ echo $AWS_PROFILE
staging

Test identity without --profile, "staging" profile should be returned, but is not.

15278-gmcdaid:~ gmcdaid$ aws sts get-caller-identity
{
"Account": “*8374",
"UserId": "**REBRS",
"Arn": "arn:aws:iam::**8374:user/gmcdaid"
}

Unset KEY and SECRET and verify AWS_PROFILE still set

15278-gmcdaid:~ gmcdaid$ unset AWS_SECRET_ACCESS_KEY
15278-gmcdaid:~ gmcdaid$ unset AWS_ACCESS_KEY_ID

Add key and secret to ~/.aws/ credentials

AWS_PROFILE still set:

15278-gmcdaid:~ gmcdaid$ echo $AWS_PROFILE
staging

Retest now that KEY and SECRET vars no longer exist. This time correct profile is returned with out without the --profile switch (ie AWS_PROFILE is used).

15278-gmcdaid:~ gmcdaid$ aws sts get-caller-identity --profile staging
{
"Account": “*6847",
"UserId": "***JELOY:botocore-session-*2088",
"Arn": "arn:aws:sts::*6847:assumed-role/Admin/botocore-session-**2088"
}

15278-gmcdaid:~ gmcdaid$ echo $AWS_PROFILE
staging

15278-gmcdaid:~ gmcdaid$ aws sts get-caller-identity
{
"Account": “*6847",
"UserId": "***JELOY:botocore-session-*2088",
"Arn": "arn:aws:sts::*6847:assumed-role/Admin/botocore-session-*2088"
}
```

[joguSD]

@garrethmcdaid-zd This is the expected behavior.

The credentials in the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID will be used if present (whether or not AWS_PROFILE is present). If a --profile is explicitly passed when invoking the command that profile will be used (it doesn't matter if the env vars are set or not).

[garrethmcdaid-zd]

My understanding is that AWS_PROFILE should be used if it is set, unless --profile is called at run time.

However, it is not being used when AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are set as environment variables.

There seems to be different behaviour when AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are set as global ernvironment variables rather than just shell variables, which I am looking in to.

[garrethmcdaid-zd]

Confirmed that same behaviour exists regardless of whether Shell or Env variables are used.

[joguSD]

@garrethmcdaid-zd The credentials from the profile will not be used when all three of those env vars are set.

https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#id1

[garrethmcdaid-zd]

@joguSD Agreed, but that isn't the issue.

It makes sense that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables will override any values set in AWS_PROFILE.

But the issue here is that when AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set in the environment (shell or globally), the value of the AWS_PROFILE itself variable is ignored.

[menocomp]

@joguSD Thanks for sharing the precedence of the 3 keys. Personally, I found this paragraph a bit confusing:

If AWS_PROFILE environment variable is set and the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, then the credentials provided by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override the credentials located in the profile provided by AWS_PROFILE.

But anyway, that solved my problem by unsetting both AWS_SECRET_ACCESS_KEY && AWS_ACCESS_KEY_ID

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY

[garrethmcdaid-zd]

I can't see why you should not be able to use all 3 environment variables at the same time:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_PROFILE

Its still the case that you can't use AWS_PROFILE when the other 2 are set.

[SunMar]

@joguSD AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY key should not be mutually exclusive with AWS_PROFILE, to me that seems like an oversight

In my credentials file I have my permanent credentials, but we require the use of MFA, also via CLI. So if you want to perform CLI commands you need to get a temporary session via aws sts get-session-token which is stored in AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.

The profile for me defines a role switch in ~/.aws/config (there are no credentials in ~/.aws/credentials for this profile) to assume a role in a different AWS account:

[profile myprofile]
credential_source=Environment
region = myregion
role_arn = arn:aws:iam::0123456789:role/myrole

So I need to be able to use all environment variables at the same time, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the temporary session credentials and AWS_PROFILE to make the account/role I want to access the default when performing commands (without having to add --profile to every aws command).

[SunMar]

Another option which maybe makes more sense is that AWS_PROFILE should override the environment AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, not the other way around. Then if the profile has credential_source=Environment it will read them from the environment.

[Krijger]

This also means that setting up an environment like described in https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html (but with a profile other than default) does not work. CLI commands do not have the region specified if you export AWS_PROFILE and run a command

[pikeas]

My use case is the same as @SunMar. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY determine base permissions, then a profile is used to assume a role/region/account for doing work.

This works with --profile but fails with AWS_PROFILE. By convention, environment variables that map to CLI args should have identical behavior, so this inconsistency seems to be a bug.

@joguSD This isn't about using the profile's credentials, it's about using the rest of the profile's values, like region and role. It's equivalent to wanting a dictionary merge (profile values + credentials) vs clobber (profile values ignored, only credentials used).

[pikeas]

@joguSD Can this issue be reopened? Or would you prefer a new issue?

[joguSD]

@pikeas While it is common in many tools for environment variables that map to CLI arguments to have exactly the same behavior, this isn't the pattern the AWS CLI has. In general, we treat explicit arguments to the CLI with a higher priority than settings in the environment. In this case, setting credentials via environment variables results in the profile set in the environment being ignored entirely (not just the credentials from it).

This interaction between AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_PROFILE was defined before assume role configuration was added. For backwards compatibility reasons we can't change the way those three environment variables interact.

This is unfortunate as @SunMar's use case seems reasonable to me, it just isn't possible for legacy reasons.

[pikeas]

@joguSD Thanks for your reply. It sounds like we're all on the same page about this behavior being surprising (credentials overriding profile).

It would be great if this could be solved in a legacy-compatible way. Are you open to the possibility of adding a new env var which could be used to accomplish the goal of "merge env credentials into profile" as AWS_CONTEXT_PROFILE? Or a change to the config file adding support for a new type of non-credential profile?

Basically, is this a WONTFIX, or is it a matter of creating the right proposal?

Meanwhile, it would be great if this could be called out in the docs, preferably in bold!

[thyme-87]

@pikeas While it is common in many tools for environment variables that map to CLI arguments to have exactly the same behavior, this isn't the pattern the AWS CLI has. In general, we treat explicit arguments to the CLI with a higher priority than settings in the environment. In this case, setting credentials via environment variables results in the profile set in the environment being ignored entirely (not just the credentials from it).

This interaction between AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_PROFILE was defined before assume role configuration was added. For backwards compatibility reasons we can't change the way those three environment variables interact.

This is unfortunate as @SunMar's use case seems reasonable to me, it just isn't possible for legacy reasons.

Could you please explain, how awscli handles the case, when I define credential_source=Environment for the target profile and then provide AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?
My assumption was, that in that case awscli has enough context and would use the AWS_ACCESS_KEY_IDand AWS_SECRET_ACCESS_KEY together with the role_arn I defined in ~/.aws/config to assume the role and store the resulting credentials internally for further usage.
That however, didn't work so that I had to fall back to provide the credentials in ~/.aws/config to make it work. (This solves my immediate problem, but I would prefer if it would be possible to keep the credenitals encrypted on a per account/organization basis, so that at any given time only one pair of credentials must be exposed)

[tomassommar]

It is confusing that the AWS_PROFILE is ignored when credentials are set as environments variables. As @pikeas points out, it is not about using the profiles credentials, but rather using the profiles values like region and role.

I understand that this can be hard to solve in a backwards compatible way, but as @pikeas points out, adding a documented env variable called AWS_CONTEXT_PROFILE could perhaps be a viable backwards compatible solution?

[kyleha-flexera]

I'm here to say this is obviously a bug.

$ aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
$ aws --profile $AWS_PROFILE s3 ls
[correct output]

...and that it cost me several hours.

Please fix it.

[geotheory]

My cli was working perfectly just recently. Now $AWS_PROFILE is ignored such that aws s3 ls returns different account to aws --profile $AWS_PROFILE s3 ls - same as @kyleha-flexera.

if [ -z $AWS_ACCESS_KEY_ID ]; then echo "unset"; fi returns '_unset_'.

Not sure if related but I recently installed python virtualenv, and subsequently removed it when I learned conda has its own env implementaton.

[robinvalk]

This is still an issue. We'd like to setup the CI/CD pipelines so that per deployment environment we can assume a different role with access to different accounts. Obviously we don't want include the credentials config in the code repo so we need to use the env variables.

Turns out the cli has this bug and there is no other way of doing it then to run AWS CLI setup code for each deployment environment in the pipeline 🤦‍♂

Does anyone know if it's fixed in cli v2?

[robinvalk]

This is still an issue. We'd like to setup the CI/CD pipelines so that per deployment environment we can assume a different role with access to different accounts. Obviously we don't want include the credentials config in the code repo so we need to use the env variables.

Turns out the cli has this bug and there is no other way of doing it then to run AWS CLI setup code for each deployment environment in the pipeline 🤦‍♂

Does anyone know if it's fixed in cli v2?

Just checked with v2, still the same issue.

[bartvanraalte]

A hack that works for me is to set an alias that always adds the profile via the command line parameter:
alias aws='aws --profile ${AWS_PROFILE:-default}'

[robinvalk]

If you want the cli to act like nothing happened if there is no AWS_PROFILE set, just give it an empty string.

alias aws='aws --profile ${AWS_PROFILE:-""}'