Aws-cli: Authentication errors when using aws cli, but not s3cmd

[rgooler]

When running any command with aws cli (installed on osx Yosemite today via pip), I fail to authenticate to all services. When using other tools, like s3cmd, with the same keys, everything works fine.

Example (censored for my proection):

$ s3cmd ls | head
2014-08-22 18:46  s3://xxxxxxxxxxxxxxx
2013-04-29 23:12  s3://xxxxxxx
2014-07-02 21:03  s3://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2016-02-11 09:12  s3://xxxxxxxxxxxxxxxxx
2014-03-29 06:50  s3://xxxxxxxxxxxxxxxxxxx
2015-08-04 20:45  s3://xxxxxxxxxxxxxxx
2015-09-29 22:56  s3://xxxxxxxxxxxxxxx
2014-10-31 00:26  s3://xxxxxxxxxxxxxxxxxxxxxxx
2013-01-21 07:30  s3://xxxxxxx
2014-07-22 00:25  s3://xxxxxxxxxxxxx


$ aws s3 ls | head

A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

Debug output:

$ aws s3 ls --debug
2016-03-23 15:14:35,560 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.10.15 Python/2.7.10 Darwin/14.5.0 botocore/1.4.6
2016-03-23 15:14:35,560 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2016-03-23 15:14:35,561 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x10385d758>
2016-03-23 15:14:35,561 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x103838488>
2016-03-23 15:14:35,561 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2016-03-23 15:14:35,561 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x103838b18>
2016-03-23 15:14:35,562 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,562 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler <function add_waiters at 0x103838b18>
2016-03-23 15:14:35,563 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,563 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,563 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthand object at 0x1038bb8d0>
2016-03-23 15:14:35,564 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,564 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,564 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthand object at 0x1038bb8d0>
2016-03-23 15:14:35,564 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <function uri_param at 0x1035b2398>
2016-03-23 15:14:35,564 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2016-03-23 15:14:35,564 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2016-03-23 15:14:35,598 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2016-03-23 15:14:35,602 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x1032f1cf8>
2016-03-23 15:14:35,602 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x1032f1500>
2016-03-23 15:14:35,672 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2016-03-23 15:14:35,673 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListBuckets: calling handler <function validate_bucket_name at 0x103312500>
2016-03-23 15:14:35,673 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListBuckets: calling handler <function add_expect_header at 0x103312938>
2016-03-23 15:14:35,673 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x103e9ce50> (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/', 'headers': {'User-Agent': 'aws-cli/1.10.15 Python/2.7.10 Darwin/14.5.0 botocore/1.4.6'}, 'query_string': '', 'url_path': u'/', 'method': u'GET'}
2016-03-23 15:14:35,674 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x103eadbd0>>
2016-03-23 15:14:35,674 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListBuckets: calling handler <function fix_s3_host at 0x103152d70>
2016-03-23 15:14:35,674 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2016-03-23 15:14:35,674 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2016-03-23 15:14:35,675 - MainThread - botocore.auth - DEBUG - StringToSign:
GET


Wed, 23 Mar 2016 22:14:35 GMT
/
2016-03-23 15:14:35,681 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2016-03-23 15:14:35,681 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): s3.amazonaws.com
2016-03-23 15:14:36,413 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET / HTTP/1.1" 403 None
2016-03-23 15:14:36,414 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'GA0AAjpgE6lQzW3fviCwcKCZeEREJGMw0iGfV5X+iZe9BkWQFM7CdxBZT/6cGdqGDQ3InTnzjkM=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'B17F8061A221DBBB', 'date': 'Wed, 23 Mar 2016 22:14:35 GMT', 'content-type': 'application/xml'}
2016-03-23 15:14:36,414 - MainThread - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId></AWSAccessKeyId><RequestId>B17F8061A221DBBB</RequestId><HostId>GA0AAjpgE6lQzW3fviCwcKCZeEREJGMw0iGfV5X+iZe9BkWQFM7CdxBZT/6cGdqGDQ3InTnzjkM=</HostId></Error>
2016-03-23 15:14:36,416 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x103e9c1d0>
2016-03-23 15:14:36,416 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2016-03-23 15:14:36,416 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <function enhance_error_msg at 0x103830b90>
2016-03-23 15:14:36,416 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListBuckets: calling handler <awscli.errorhandler.ErrorHandler object at 0x1038bb9d0>
2016-03-23 15:14:36,417 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2016-03-23 15:14:36,417 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 186, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/Library/Python/2.7/site-packages/awscli/customizations/commands.py", line 190, in __call__
    parsed_globals)
  File "/Library/Python/2.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/Library/Python/2.7/site-packages/awscli/customizations/s3/subcommands.py", line 449, in _run_main
    self._list_all_buckets()
  File "/Library/Python/2.7/site-packages/awscli/customizations/s3/subcommands.py", line 508, in _list_all_buckets
    response_data = self.client.list_buckets()
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 228, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 488, in _make_api_call
    model=operation_model, context=request_context
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 226, in emit
    return self._emit(event_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/hooks.py", line 209, in _emit
    response = handler(**kwargs)
  File "/Library/Python/2.7/site-packages/awscli/errorhandler.py", line 70, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.
2016-03-23 15:14:36,424 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

[JordonPhillips]

How are you passing your keys to each tool? This sort of thing is often the result of a bad copy/paste, so I would double check that they're both being given the same value. If you used aws config to set up your credentials, they will be located in ~/.aws/credentials.

[rgooler]

The keys in ~/.aws/credentials and ~/.s3cfg/ are identical. There are no trailing whitespace or anything.

[default]
aws_access_key_id = AKXXXXXXXXXXXXXXXXXX
aws_secret_access_key = 0mqXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

[kyleknap]

Interesting. I wonder if you are pulling your credentials from an unintended source. Could you run:
$ aws configure list. This will show you what credentials are being used and where they are being pulled from.

[rgooler]

Thank you - there were some errant environment variables set that caused the issue. Everything working fine now.

As a side note, this happened cause the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY were both set to empty strings - maybe skip using any credential that is an empty string would be a good enhancement?

As far as I'm concerned, though, this is resolved for me.

[kyleknap]

I see where you are coming from in the case that they were accidentally set to nothing. The only issue is that if we have this behavior for environment variables, we would have to add this check for all instances where you could set credentials (like in the shared credentials file and config file), which would substantially complicate that code path as we are no longer checking for if the credentials are set but also if they follow a valid format. Furthermore, since the other SDKs share sources for credentials, we also share how credentials get resolved. Adding this behavior may cause us to potentially deviate from the rest of the SDKs which is not ideal for individuals who use multiple SDKs.

I still think it something we can consider in the future, but will require much broader thought and discussion across SDKs.

Otherwise, I am glad it is now working for you.

[frank-lzh]

Event I set the environment variable AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the same error comes out. Finally I go with directly passing the credentials to the API: import boto3 client = boto3.client( 's3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, aws_session_token=SESSION_TOKEN)