botocore 1.9.x doesn't raise ExpiredToken error

[smtakeda]

s3client.Object's upload_file method does no longer raise ExpiredToken error if the given token expires in botocore 1.9.x. I have this code and catch ClientError such that the upper stream can renew the token. This works fine before 1.9.0:

try
...
            akey = meta[u'client'].Object(s3location.bucket_name, s3path)
            akey.upload_file(
                data_file,
                Callback=meta[u'put_callback'](
                    data_file,
                    os.path.getsize(data_file),
                    output_stream=meta[u'put_callback_output_stream']) if
                        meta[u'put_callback'] else None,
                ExtraArgs={
                    u'Metadata': s3_metadata,
                    u'ContentEncoding': u'gzip',
                },
                Config=TransferConfig(
                    multipart_threshold=SnowflakeS3Util.DATA_SIZE_THRESHOLD,
                    max_concurrency=max_concurrency,
                    num_download_attempts=10,
                )
            )
...
        except botocore.exceptions.ClientError as err:
            if err.response[u'Error'][u'Code'] == 'ExpiredToken':
                logger.debug(u"AWS Token expired. Renew and retry")
                meta[u'result_status'] = ResultStatus.RENEW_TOKEN
                return

But the new botocore 1.9.0+ returns HTTP 400 but doesn't raise ExpiredToken error code.

Is there any way to detect the token expiration?

[smtakeda]

Here is a reproducible test case, which will not raise an error but _hang_ where the authentication use a valid key and secret and invalid token using boto3==1.6.4 and botocore==1.9.4:

import boto3

# specify AWS key, secret and an invalid token to get HTTP 400
client = boto3.resource(
    u's3',
    region_name='us-west-2',
    aws_access_key_id='VALIDKEY'
    aws_secret_access_key='VALIDSECRET',
    aws_session_token='INVALIDTOKEN',
)

akey = client.Object('bucketname', '/path/file')
akey.load()

But if you run boto3=1.5.36 and botocore==1.8.50, it will raise an error:

botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Looks like new botocore retries automatically for HTTP 400, but never success. In theory, the error indicates the request is not valid, so it has to change the input and retry.

[dstufft]

I can reproduce this using the above code, my initial guess is that it's related to https://github.com/boto/botocore/pull/1387 somehow.

[smtakeda]

Removing 400 from this line address the issue:
https://github.com/boto/botocore/blob/develop/botocore/utils.py#L899

Is it possible to check the content of HTTP response to decide whether it is retryable?

There are several cases that return 400:

https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

[smtakeda]

Just to clarify my specific use case is error_code== ExpiredToken. Testing code that excludes the case for retry.

[smtakeda]

Verified PR #1409 worked for my use case (not just above test case). Thank you @joguSD !

[smtakeda]

ah, looks like #1409 was reverted by #1414. can somebody reopen this?

[dstufft]

Done.

[smtakeda]

1.9.18 works for my test cases. Thanks!

[joguSD]

Good to hear!

[fiendish]

This is happening to me with boto3 1.9.47 + botocore 1.12.47 calling download_fileobj with an expired profile.

boto3.session.Session(profile_name='saml').resource('s3').Object(bucket, key).download_fileobj(dest_obj)
  File "venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 760, in object_download_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "venv/lib/python3.7/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "venv/lib/python3.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "venv/lib/python3.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "venv/lib/python3.7/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "venv/lib/python3.7/site-packages/s3transfer/download.py", line 353, in _submit
    **transfer_future.meta.call_args.extra_args
  File "venv/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "venv/lib/python3.7/site-packages/botocore/client.py", line 624, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

The call works if I re-auth.

reopen with new title?