botocore 1.9.13 resulting in 400s on s3 HeadObject if region is wrong

[cwharland]

Problem

Upon upgrading to awcli 1.14.60 we started to get:

fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

when invoking aws s3 cp. It turns out that if our default region was not set correctly (i.e. it didn't match the region the file was actually stored in) this happens. We can fix by specifying --region but that means we have to know the underlying region of each s3 key.

Our current fix is to pin awscli==1.14.59 which uses botocore 1.9.12 and avoids all errors. This is happening across all our local boxes and CircleCi builds.

Suspicion is that https://github.com/boto/botocore/pull/1409 might have something to do with it.

Repro

Assume your_file.txt is in us-west-2
aws s3 cp s3://your_bucket/your_file.txt --region us-east-1 will fail with above error
aws s3 cp s3://your_bucket/your_file.txt --region us-west-2 will succeed

Note: We use sse liberally so I'm not sure if this repros on a key without encryption.

System info

$ venv/bin/aws --version
aws-cli/1.14.60 Python/3.6.2 Darwin/17.4.0 botocore/1.9.13

[jdirwin]

Just confirming that we are seeing this exact bug as well.

[joguSD]

Can you provide debug logs for a request that fails?

[cwharland]

Will do. Have to obfuscate key and bucket names. Will take a second.

[kyleknap]

I am seeing this as well:

$ aws s3api get-bucket-location --bucket knapp-us-west-2
{
    "LocationConstraint": "us-west-2"
}
$ aws s3 cp vpc.sh s3://knapp-us-west-2
$ aws s3 cp s3://knapp-us-west-2/vpc.sh . --region us-east-1
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

These are my logs:

2018-03-21 16:56:16,646 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.14.60 Python/2.7.10 Darwin/15.6.0 botocore/1.9.13
2018-03-21 16:56:16,646 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://knapp-us-west-2/vpc.sh', '.', '--region', 'us-east-1', '--debug']
2018-03-21 16:56:16,647 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x10c5e4500>
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10bfb9cf8>
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,647 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2018-03-21 16:56:16,648 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,648 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x10c1d8500>
2018-03-21 16:56:16,648 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,648 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x10c5ea668>
2018-03-21 16:56:16,649 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,649 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x10c5ea668>
2018-03-21 16:56:16,653 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,653 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.request-payer: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,654 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,655 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,656 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,657 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,658 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,658 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <function uri_param at 0x10be4c2a8>
2018-03-21 16:56:16,658 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10c5f0a50>
2018-03-21 16:56:16,658 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,658 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-03-21 16:56:16,658 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,658 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-03-21 16:56:16,658 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2018-03-21 16:56:16,658 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2018-03-21 16:56:16,658 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2018-03-21 16:56:16,659 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2018-03-21 16:56:16,659 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/endpoints.json
2018-03-21 16:56:16,698 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,698 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10bbc05f0>
2018-03-21 16:56:16,701 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /Users/kyleknap/GitHub/botocore/botocore/data/s3/2006-03-01/service-2.json
2018-03-21 16:56:16,741 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10bb89b90>
2018-03-21 16:56:16,742 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10bb899b0>
2018-03-21 16:56:16,752 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-03-21 16:56:16,753 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-03-21 16:56:16,753 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-03-21 16:56:16,753 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,753 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,754 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-03-21 16:56:16,754 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,754 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-03-21 16:56:16,754 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,754 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10bbc05f0>
2018-03-21 16:56:16,757 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10bb89b90>
2018-03-21 16:56:16,757 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10bb899b0>
2018-03-21 16:56:16,760 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-03-21 16:56:16,762 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-03-21 16:56:16,763 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-03-21 16:56:16,763 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,763 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-03-21 16:56:16,764 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,764 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-03-21 16:56:16,764 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2018-03-21 16:56:16,764 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10bbc05f0>
2018-03-21 16:56:16,766 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10bb89b90>
2018-03-21 16:56:16,767 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10bb899b0>
2018-03-21 16:56:16,770 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2018-03-21 16:56:16,771 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2018-03-21 16:56:16,771 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2018-03-21 16:56:16,772 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608
2018-03-21 16:56:16,772 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x10bbc0c08>
2018-03-21 16:56:16,772 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x10bbc0b90>
2018-03-21 16:56:16,772 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x10d69ca10>>
2018-03-21 16:56:16,772 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x10bbc0848>
2018-03-21 16:56:16,773 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x10bbc4050>
2018-03-21 16:56:16,773 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x10d69ca10>>
2018-03-21 16:56:16,773 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) (verify_ssl=True) with params: {'body': '', 'url': u'https://s3.amazonaws.com/knapp-us-west-2/vpc.sh', 'headers': {'User-Agent': 'aws-cli/1.14.60 Python/2.7.10 Darwin/15.6.0 botocore/1.9.13'}, 'context': {'auth_type': None, 'client_region': u'us-east-1', 'signing': {'bucket': u'knapp-us-west-2'}, 'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x10d64f910>}, 'query_string': {}, 'url_path': u'/knapp-us-west-2/vpc.sh', 'method': u'HEAD'}
2018-03-21 16:56:16,774 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10d64f850>>
2018-03-21 16:56:16,774 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x10d6351d0>>
2018-03-21 16:56:16,774 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x10bbc0758>
2018-03-21 16:56:16,774 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x10ba30758>
2018-03-21 16:56:16,774 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/knapp-us-west-2/vpc.sh
2018-03-21 16:56:16,774 - MainThread - botocore.utils - DEBUG - URI updated to: https://knapp-us-west-2.s3.amazonaws.com/vpc.sh
2018-03-21 16:56:16,775 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-03-21 16:56:16,775 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/vpc.sh

host:knapp-us-west-2.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180321T235616Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-03-21 16:56:16,775 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180321T235616Z
20180321/us-east-1/s3/aws4_request
67818b2822c07393bf0b7c7c13dff4dd715e0ae9ae8e14e79a5e8edb66249a62
2018-03-21 16:56:16,775 - MainThread - botocore.auth - DEBUG - Signature:
531aba9bf9817e02f6ebc399fba731f5625e53d8eb92c354d2d50aa3f7cbd79b
2018-03-21 16:56:16,778 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [HEAD]>
2018-03-21 16:56:16,778 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): knapp-us-west-2.s3.amazonaws.com
2018-03-21 16:56:16,840 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "HEAD /vpc.sh HTTP/1.1" 400 0
2018-03-21 16:56:16,843 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '4Vzq3n1nUDlO+rSXksZWzxwsDQ99Jie/yyc6aGY/Os+WlZzKQ4slVAoaR/I05ojwHNOKa8FXr+o=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': '2CB9581A2A69322F', 'date': 'Wed, 21 Mar 2018 23:56:16 GMT', 'content-type': 'application/xml'}
2018-03-21 16:56:16,843 - MainThread - botocore.parsers - DEBUG - Response body:

2018-03-21 16:56:16,843 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x10d69c990>
2018-03-21 16:56:16,844 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2018-03-21 16:56:16,844 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x10d69ca10>>
2018-03-21 16:56:16,844 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x10c5e40c8>
2018-03-21 16:56:16,848 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: An error occurred (400) when calling the HeadObject operation: Bad Request
Traceback (most recent call last):
  File "/Users/kyleknap/GitHub/aws-cli/awscli/customizations/s3/s3handler.py", line 173, in call
    for fileinfo in fileinfos:
  File "/Users/kyleknap/GitHub/aws-cli/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
    for file_base in files:
  File "/Users/kyleknap/GitHub/aws-cli/awscli/customizations/s3/filegenerator.py", line 142, in call
    for src_path, extra_information in file_iterator:
  File "/Users/kyleknap/GitHub/aws-cli/awscli/customizations/s3/filegenerator.py", line 318, in list_objects
    yield self._list_single_object(s3_path)
  File "/Users/kyleknap/GitHub/aws-cli/awscli/customizations/s3/filegenerator.py", line 355, in _list_single_object
    response = self._client.head_object(**params)
  File "/Users/kyleknap/GitHub/botocore/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/kyleknap/GitHub/botocore/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request
fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request
2018-03-21 16:56:16,852 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

I think we may need to revert this and re-release.

[cwharland]

Our debug log is nearly identical to @kyleknap

[jamesls]

Yep I can confirm here. Let's revert this.

[kyleknap]

PR to revert this is here: https://github.com/boto/botocore/pull/1414

[cwharland]

Thanks for the fast response and fix :1st_place_medal:

[joguSD]

As a note, this will fail to redirect only if the client is in the us-east-1 region. If you set the region to anything else it will successfully attempt to redirect the request.

[cwharland]

Will there a be a re-release or should we stick with our version freezes?

[joguSD]

There will be another release going out shortly that reverts the above PR. We'll need to take some time to figure out how to tighten when we redirect while still supporting redirects for clients using us-east-1.

[pkmnfrk]

Hi, for what it's worth, I'm seeing this behaviour in us-west-2, although my own debug logs show us-east-1 for the region. My logs look exactly like the one @kyleknap posted.

[joguSD]

@pkmnfrk Yep, it's the region the aws cli is configured to use (for example by setting --region).
So if you have a bucket in us-west-2 but have configured the CLI to use us-east-1 we will fail to redirect to us-west-2.

[kyleknap]

We just released a fix. Upgrade to 1.14.61 of the CLI and it should be working again. Everything works for me when I upgrade. Can others on the thread chime in that it works for them as well?

[pkmnfrk]

Looks good on my end! Thanks for the quick turn around!

[kyleknap]

No problem. Glad it works again. Closing out issue.