Aws-cli: Bad error message if instance profile assume role policy is wrong

[rvandegrift]

We had an instance profile with a misconfigured role - the assume role policy didn't permit ec2.amazonaws.com. awscli gets very confusing:

ubuntu@i-7b4dc7eb:~$ aws --region us-east-1 ec2 describe-instances

'AccessKeyId'
ubuntu@i-7b4dc7eb:~$

It'd be nice if awscli could return the helpful error message from the meta-data API:

ubuntu@i-7b4dc7eb:~$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/testing-node-role
{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role testing-node-role.  Please see documentation at http://docs.amazonwebservices.com/IAM/latest/UserGuide/RolesTroubleshooting.html.",
  "LastUpdated" : "2016-07-11T18:09:39Z"
}

Thanks,
Ross

[kyleknap]

I can confirm this. Here is my debug logs:

$ aws s3 ls --debug
2016-07-12 19:30:37,299 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.10.33 Python/2.7.10 Linux/4.4.11-23.53.amzn1.x86_64 botocore/1.4.23
2016-07-12 19:30:37,299 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fefe98bec80>
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fefe9c73f50>
2016-07-12 19:30:37,300 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2016-07-12 19:30:37,300 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7fefe98c4578>
2016-07-12 19:30:37,301 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,301 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x7fefe98c4578>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fefe98c94d0>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,302 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,303 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fefe98c94d0>
2016-07-12 19:30:37,303 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <function uri_param at 0x7fefe9c9b410>
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2016-07-12 19:30:37,303 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2016-07-12 19:30:37,307 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2016-07-12 19:30:37,308 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 23
2016-07-12 19:30:37,310 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2016-07-12 19:30:37,311 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/DataPipelineDefaultRole HTTP/1.1" 200 267
2016-07-12 19:30:37,312 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/awscli/clidriver.py", line 186, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 438, in _run_main
    super(ListCommand, self)._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 424, in _run_main
    parsed_globals.verify_ssl)
  File "/usr/lib/python2.7/dist-packages/awscli/customizations/s3/subcommands.py", line 417, in get_client
    config=config)
  File "/usr/lib/python2.7/dist-packages/botocore/session.py", line 808, in create_client
    credentials = self.get_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/session.py", line 442, in get_credentials
    'credential_provider').load_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/credentials.py", line 1017, in load_credentials
    creds = provider.load()
  File "/usr/lib/python2.7/dist-packages/botocore/credentials.py", line 473, in load
    metadata = fetcher.retrieve_iam_role_credentials()
  File "/usr/lib/python2.7/dist-packages/botocore/utils.py", line 198, in retrieve_iam_role_credentials
    'access_key': data[role_name]['AccessKeyId'],
KeyError: 'AccessKeyId'
2016-07-12 19:30:37,493 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

'AccessKeyId'

Looks like we should check if there is an error message if the access or secret access keys are missing.

[rsalmond]

Based on the above I would suggest this bug should be raised against botocore rather than this repo, since that's the component which cycles through various AWS auth methods trying and then ultimately failing to get access. awscli is just relaying the exception.

[jobwat]

For the next guy coming here with the same error and no clue what's happening:
For me the badly setup IAM Role part was the "Trust entities".
I had rds.amazonaws.com instead of ec2.amazonaws.com
Good luck! :)

[srvvr]

thanks @jobwat - that saved my butt today!!!

[rdonkin]

Since this is such a poor error, here's how to get the underlying error - use your IAM role name in the curl command from this AWS troubleshooting section. In my case, the EC2 service (ec2.amazonaws.com) was not authorized as a principal allowed to assume the IAM role.