Aws-cli: Assume role with profile throws KeyError: 'aws_access_key_id'

[pbudzon]

Hi,

I'm trying to assume a role through a cli profile, as described here: http://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html

I have an EC2 instance with an instance role with permissions to assume a role on another account.
AWS CLI version:
aws-cli/1.9.2 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.2

My .aws/config:

[default]
region = eu-west-1

[profile somerole-1234]
source_profile=default
role_arn=arn:aws:iam::1234:role/somerole

Running aws sts assume-role --role-arn arn:aws:iam::1234:role/somerole --role-session-name test returns credentials information as intended.
But if I run aws --profile somerole-1234 --debug ... I get the following error:

2015-11-02 16:04:39,685 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.9.2 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.2
2015-11-02 16:04:39,685 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['rds', 'describe-db-snapshots', '--profile', 'cnidev-800540593336', '--debug']
2015-11-02 16:04:39,685 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f90452f29b0>
2015-11-02 16:04:39,685 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7f904554b050>
2015-11-02 16:04:39,717 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.rds: calling handler <function register_retries_for_service at 0x7f9045afca28>
2015-11-02 16:04:39,717 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: rds
2015-11-02 16:04:39,719 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <function _building_command_table at 0x7f9045621500>
2015-11-02 16:04:39,719 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.rds: calling handler <function register_retries_for_service at 0x7f9045afca28>
2015-11-02 16:04:39,720 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: rds
2015-11-02 16:04:39,720 - MainThread - botocore.hooks - DEBUG - Event building-command-table.rds: calling handler <function add_waiters at 0x7f904554bde8>
2015-11-02 16:04:39,721 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'db-instance-identifier', <awscli.arguments.CLIArgument object at 0x7f9044f65f90>), (u'db-snapshot-identifier', <awscli.arguments.CLIArgument object at 0x7f9044f65fd0>), (u'snapshot-type', <awscli.arguments.CLIArgument object at 0x7f9044f65cd0>), (u'filters', <awscli.arguments.ListArgument object at 0x7f9044f7b050>), (u'max-records', <awscli.arguments.CLIArgument object at 0x7f9044f7b090>), (u'marker', <awscli.arguments.CLIArgument object at 0x7f9044f7b0d0>)])
2015-11-02 16:04:39,721 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.describe-db-snapshots: calling handler <function add_streaming_output_arg at 0x7f90456ecb90>
2015-11-02 16:04:39,722 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.describe-db-snapshots: calling handler <function add_cli_input_json at 0x7f904553d410>
2015-11-02 16:04:39,722 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.describe-db-snapshots: calling handler <function unify_paging_params at 0x7f904568dd70>
2015-11-02 16:04:39,722 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: DescribeDBSnapshots
2015-11-02 16:04:39,722 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.rds.describe-db-snapshots: calling handler <function add_generate_skeleton at 0x7f904553d758>
2015-11-02 16:04:39,723 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.rds.describe-db-snapshots: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f9044f7b110>>
2015-11-02 16:04:39,723 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.rds.describe-db-snapshots: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f9044f7b410>>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.rds.describe-db-snapshots: calling handler <functools.partial object at 0x7f9044f5a520>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.db-instance-identifier: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.db-snapshot-identifier: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.snapshot-type: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.filters: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.max-records: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.marker: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,724 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.cli-input-json: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.starting-token: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.page-size: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.max-items: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.rds.describe-db-snapshots.generate-cli-skeleton: calling handler <function uri_param at 0x7f90456d5c08>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event calling-command.rds.describe-db-snapshots: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f9044f7b410>>
2015-11-02 16:04:39,725 - MainThread - botocore.hooks - DEBUG - Event calling-command.rds.describe-db-snapshots: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f9044f7b110>>
2015-11-02 16:04:39,725 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2015-11-02 16:04:39,725 - MainThread - awscli.customizations.assumerole - DEBUG - Retrieving credentials via AssumeRole.
2015-11-02 16:04:39,725 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 184, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 369, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 538, in __call__
    call_parameters, parsed_globals)
  File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 655, in invoke
    verify=parsed_globals.verify_ssl)
  File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 754, in create_client
    credentials = self.get_credentials()
  File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 413, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 602, in load_credentials
    creds = provider.load()
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/assumerole.py", line 186, in load
    return self._load_creds_via_assume_role()
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/assumerole.py", line 205, in _load_creds_via_assume_role
    creds, response = self._retrieve_temp_credentials()
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/assumerole.py", line 305, in _retrieve_temp_credentials
    client = self._create_client_from_config(config)
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/assumerole.py", line 296, in _create_client_from_config
    'sts', aws_access_key_id=source_cred_values['aws_access_key_id'],
KeyError: 'aws_access_key_id'
2015-11-02 16:04:39,726 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

'aws_access_key_id'

I'm trying to assume the role on another account to perform an action on this account. This works correctly when access key and secret to a user account are placed inside the .aws/credentials file in the default profile. I have moved the policies from the user account to the instance role as it seemed like a better solution that storing the credentials on the machine.

My .aws/credentials file is currently empty. Please let me know if I'm doing something completely unsupported here! Although the documentation (http://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html) and the output of the command seem to indicate that I'm on the right track here :)

[pbudzon]

I think this is a duplicate of #1604
I don't understand why this is working when access keys are specified, but not when the access is granted via EC2 instance role? Especially since the documentation suggests this is possible.

[kyleknap]

@pbudzon
In order to take advantage of the assume role functionality you need to specify the aws access key id and secret access key in the .aws/credentials or .aws/config files.

If you want to use the EC2 instance role, you would need to make sure that you are not specifying a --profile. The CLI uses EC2 instance metdata to pull for temporary credentials for that instance role. However, instance metadata is the last source that is checked in the credential chain and by setting--profile explicitly, it tells the CLI that the creds will be in the config or credential file, which they are not.

Let me know if you have any more questions and if this helps.

[pbudzon]

Thanks for the answer.

So in order to assume a role with instance profile, I'd have to call sts assume-role myself and use the access key and secret it returns to authenticate against future calls (probably by exporting them as env variables) - which is more or less what CLI is doing by itself when --profile is used? In my opinion, from the user's point of view, it shouldn't really matter where the original permission to assume the role came from.

If this is not possible, then I have to say it makes the instance profile much less useful than it could be. Storing the access keys on the ec2 machine seems like a much worse solution than using instance profiles. I think this should be considered as a feature to add - or if not, then the documentation should be updated (including here: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence) to state that instance profiles have limits like this, as so far the documentation suggests all ways of authenticating are equal and can be used interchangeably.

The documentation here: http://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html does not mention at all that this is possible only with access keys stored in the credentials file, so it would be great if it was amended as well.

[kyleknap]

No for instance profiles, if you launch your instance with an instance profile, the CLI can automatically detect that the instance was launched with a role and so it will automatically grap temporary credentials from the EC2 metadata. To use instance profile with the CLI, you do not need credentials stored on your instance. As a matter of fact, if you launched the instance with a role, you should be able to completely wipe out your credentials and config files and you should be able to make calls with the CLI.

If you do specify --profile in the command you ran, the CLI will not look at the EC2 metadata to grab the temporary credentials because --profile has a higher precedence in the credential chain then checking the EC2 metadata.

As to the documentation about the order of precedence, it does state that the CLI will look for credentials in an order of preference and in this list --profile will fall under category 1 and instance profile falls under category 5, hence --profile is checked before the EC2 metadata.

We also have topic guides about IAM roles here. Let me know if you have any more questions or anything else that you find misleading about the docs.

[kyleknap]

Never mind I just reread your comment. Misunderstood the use case. You are trying to use temprorary credentials from EC2 metdata to assume role using with the CLI's assume role provider. So we do not have support for that. We have an open issue requesting that: https://github.com/aws/aws-cli/issues/1390.

[kyleknap]

Yeah this makes much more sense now. Closing this issue as it is a duplicate of the referenced feature request. I think what we will have to do is add a value for the source_profile or a flag of some sort to indicate that we need to query the EC2 metadata. Thanks for the feedback!