Aws-cli: S3-RequestTimeout still not working properly

[rhasson]

I'm using aws-cli/1.10.1 Python/2.6.6 Linux/2.6.32-573.18.1.el6.x86_64 botocore/1.3.23

This issue is related to https://github.com/aws/aws-cli/issues/401

In https://github.com/aws/aws-sdk-js/issues/281#issuecomment-43798354 it was mentioned that this possibly is related to the Content-Length header being out of sync with the actual number of bytes being sent on the wire. If #401 above fixed the issue by properly rewinding the stream, is the Content-Length header not being properly updated?

The above mentioned 2 issues were closed but did not completely resolve the issue. I'm still seeing these failures happen on uploads.

<Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>XXX</RequestId><HostId>XXX</HostId></Error>
    proxies=self.proxies, timeout=self.timeout)
    proxies=self.proxies, timeout=self.timeout)
ReadTimeout: HTTPSConnectionPool(host='xxx.s3.amazonaws.com', port=443): Read timed out. (read timeout=60)
2016-04-19 09:44:54,113 - Thread-5 - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='xxx.s3.amazonaws.com', port=443): Read timed out. (read timeout=60)
ReadTimeout: HTTPSConnectionPool(host='xxx.s3.amazonaws.com', port=443): Read timed out. (read timeout=60)
    proxies=self.proxies, timeout=self.timeout)

[pouya-cognitiv]

Same issue using:
aws-cli/1.2.9 Python/3.4.3 Linux/3.13.0-85-generic

Using mc in order to be able to restart downloads:
mc: Failed to copy ‘https://s3.amazonaws.com/blah.tar.gz’. net/http: request canceled (Client.Timeout exceeded while reading body)

[JordonPhillips]

@rhasson Could you post --debug logs for an affected copy? Also, is there anything special about the files?

[rhasson]

This was an S3 sync of a directory containing Oracle DB backup files, 5.6TB in total. I can't upload the entire debug log since it's over 500MB but here is a snippet of a couple errors:

2016-04-20 23:31:31,204 - Thread-12 - botocore.vendored.requests.packages.urllib
3.connectionpool - DEBUG - "PUT /IRMREP/yyy?partNumber=354&uploadId=TcsY2HSzEiUeIPPoISeA2jydNI8jNHego1aGXpzRlhyg9Emm2kK40B9Z6J7AG698geNlATG1v5F2LBpQE8wLKQ.dj6lz.jMiRsCgRwzrqzVWIgD1_uAkL.ai1MQ_cUPd HTTP/1.1" 400 None
2016-04-20 23:31:31,208 - Thread-12 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'SwqIatAXcrp4LM3OVkJoW505+ke9V8Aw0ZbhRMf27hMWSdrZrJmn2a0+o1nNIRkX6C/l+sg8KQ0=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': 'AECCC4EDF04B4114', 'date': 'Thu, 21 Apr 2016 03:30:58 GMT', 'content-type': 'application/xml'}
2016-04-20 23:31:31,208 - Thread-12 - botocore.parsers - DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>AECCC4EDF04B4114</RequestId><HostId>SwqIatAXcrp4LM3OVkJoW505+ke9V8Aw0ZbhRMf27hMWSdrZrJmn2a0+o1nNIRkX6C/l+sg8KQ0=</HostId></Error>
2016-04-20 23:31:31,208 - Thread-12 - botocore.hooks - DEBUG - Event needs-retry.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x28f8d90>
2016-04-20 23:31:31,208 - Thread-12 - botocore.retryhandler - DEBUG - retry needed: matching HTTP status and error code seen: 400, RequestTimeout
2016-04-20 23:31:31,208 - Thread-12 - botocore.retryhandler - DEBUG - Retry needed, action of: 0.168099706037
2016-04-20 23:31:31,209 - Thread-12 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.168099706037 seconds
2016-04-20 23:31:31,377 - Thread-12 - botocore.awsrequest - DEBUG - Rewinding stream: <awscli.customizations.s3.utils.ReadFileChunk object at 0x2cd2450>
2016-04-20 23:31:31,377 - Thread-12 - botocore.hooks - DEBUG - Event request-created.s3.UploadPart: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x29e2d50>>
2016-04-20 23:31:31,377 - Thread-12 - botocore.hooks - DEBUG - Event before-sign.s3.UploadPart: calling handler <function fix_s3_host at 0x1aba5f0>
2016-04-20 23:31:31,377 - Thread-12 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/xxx/IRMREP/yyy?partNumber=354&uploadId=TcsY2HSzEiUeIPPoISeA2jydNI8jNHego1aGXpzRlhyg9Emm2kK40B9Z6J7AG698geNlATG1v5F2LBpQE8wLKQ.dj6lz.jMiRsCgRwzrqzVWIgD1_uAkL.ai1MQ_cUPd
2016-04-20 23:31:31,378 - Thread-12 - botocore.utils - DEBUG - URI updated to: https://xxx.s3.amazonaws.com/IRMREP/yyy?partNumber=354&uploadId=TcsY2HSzEiUeIPPoISeA2jydNI8jNHego1aGXpzRlhyg9Emm2kK40B9Z6J7AG698geNlATG1v5F2LBpQE8wLKQ.dj6lz.jMiRsCgRwzrqzVWIgD1_uAkL.ai1MQ_cUPd
2016-04-20 23:31:31,378 - Thread-12 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2016-04-20 23:31:31,378 - Thread-12 - botocore.auth - DEBUG - HTTP request method: PUT
2016-04-20 23:31:31,378 - Thread-12 - botocore.auth - DEBUG - StringToSign:
PUT
AIYsCRbgl+SGWNUfQDmxaQ==

Thu, 21 Apr 2016 03:31:31 GMT
x-amz-security-token:AQoDYXdzEPn...cUPd
2016-04-20 23:31:31,379 - Thread-12 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2016-04-20 23:31:31,379 - Thread-12 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Resetting dropped connection: xxx.s3.amazonaws.com
2016-04-20 23:31:31,379 - Thread-12 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (8583): xxx.s3.amazonaws.com
2016-04-20 23:31:31,413 - Thread-12 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2016-04-20 23:31:31,447 - Thread-12 - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.

*******************************************************

2016-04-20 23:39:43,344 - Thread-12 - botocore.endpoint - DEBUG - ConnectionErro
r received when sending HTTP request.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/botocore/endpoint.py", line 168, in _ge
t_response
    proxies=self.proxies, timeout=self.timeout)
  File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/sessions.py"
, line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/botocore/vendored/requests/adapters.py"
, line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine('',))
2016-04-20 23:39:43,345 - Thread-12 - botocore.hooks - DEBUG - Event needs-retry
.s3.UploadPart: calling handler <botocore.retryhandler.RetryHandler object at 0x
28f8d90>
2016-04-20 23:39:43,345 - Thread-12 - botocore.retryhandler - DEBUG - retry need
ed, retryable exception caught: Connection was closed before we received a valid
 response from endpoint URL: "https://xxx.s3.amazonaws.com/IRMREP/yyy?partNumber=359&uploadId=SYNR
jVlcRTdj68yAS.dhFMa9gtGe0TMNd3Gk.J2aX_C0TTBj3dU_KmDOzkO6FhpS43MWAKrnCmtaQtgAGAwH
JU9elzswNrXQS6ZBURVV.FkH2s40pg3hWaRr30hmz06.".
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/botocore/retryhandler.py", line 265, in
 _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/lib/python2.6/site-packages/botocore/retryhandler.py", line 313, in __call__
    caught_exception)
  File "/usr/lib/python2.6/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(attempt_number, caught_exception)
  File "/usr/lib/python2.6/site-packages/botocore/retryhandler.py", line 355, in _check_caught_exception
    raise caught_exception
ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://xxx.s3.amazonaws.com/IRMREP/yyy?partNumber=359&uploadId=SYNRjVlcRTdj68yAS.dhFMa9gtGe0TMNd3Gk.J2aX_C0TTBj3dU_KmDOzkO6FhpS43MWAKrnCmtaQtgAGAwHJU9elzswNrXQS6ZBURVV.FkH2s40pg3hWaRr30hmz06.".
2016-04-20 23:39:43,346 - Thread-12 - botocore.retryhandler - DEBUG - Retry needed, action of: 0.846044759328
2016-04-20 23:39:43,346 - Thread-12 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.846044759328 seconds

[JordonPhillips]

It looks like it was retrying the request as it should. Are you getting the errors consistently? If so, you can increase the read timeout with the --read-timeout argument. The default is 60, and 0 would be no timeout. Could you see if increasing the timeout solves the problem?

[JordonPhillips]

Closing due to inactivity.

[ghost]

This problem still exists. In multiple amazon SDKs. Closing the issue from inactivity lol....cmon

[voycey]

Yeah I am still having this problem when trying to do a non-multipart upload....

[banjoman]

+1, managed about 700MB/16000MB of image transfers to local before it timed out.

[blackbaud-sumitpandey]

i am using aws-cli/1.14.32 Python/2.7.6 Linux/3.4.0+ botocore/1.8.36 version of AWS-CLI
i am using us-east-1 region.
getting time out error when hitting the command " aws ec2 describe-regions --profile bb-lonxt-shared"

HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)

i tried to expand the log by hitting command in --debug mode and getting below logs _

sumit@NAG6SUMITPAN02:~/.aws$ aws ec2 describe-regions --profile bb-lonxt-shared --debug
2018-02-06 05:45:30,086 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.14.32 Python/2.7.6 Linux/3.4.0+ botocore/1.8.36
2018-02-06 05:45:30,088 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'describe-regions', '--profile', 'bb-lonxt-shared', '--debug']
2018-02-06 05:45:30,088 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2018-02-06 05:45:30,088 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,088 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2018-02-06 05:45:30,088 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,088 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2018-02-06 05:45:30,089 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2018-02-06 05:45:30,090 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,090 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2018-02-06 05:45:30,091 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,091 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2018-02-06 05:45:30,092 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,092 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2018-02-06 05:45:30,093 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2018-02-06 05:45:30,093 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,094 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,094 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-06 05:45:30,102 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/dist-packages/botocore/data/ec2/2016-11-15/service-2.json
2018-02-06 05:45:30,227 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler
2018-02-06 05:45:30,229 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: ec2
2018-02-06 05:45:30,241 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler
2018-02-06 05:45:30,242 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance
2018-02-06 05:45:30,243 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume
2018-02-06 05:45:30,244 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler
2018-02-06 05:45:30,351 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/dist-packages/botocore/data/ec2/2016-11-15/waiters-2.json
2018-02-06 05:45:30,357 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'filters', ), (u'region-names', ), (u'dry-run', ), (u'no-dry-run', )])
2018-02-06 05:45:30,358 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,359 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,359 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,360 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,360 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,363 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,405 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/dist-packages/botocore/data/ec2/2016-11-15/paginators-1.json
2018-02-06 05:45:30,406 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-regions: calling handler
2018-02-06 05:45:30,406 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-regions: calling handler >
2018-02-06 05:45:30,407 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-regions: calling handler >
2018-02-06 05:45:30,411 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.describe-regions: calling handler
2018-02-06 05:45:30,411 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-regions.filters: calling handler
2018-02-06 05:45:30,412 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-regions.region-names: calling handler
2018-02-06 05:45:30,413 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-regions.dry-run: calling handler
2018-02-06 05:45:30,414 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-regions.cli-input-json: calling handler
2018-02-06 05:45:30,417 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-regions.generate-cli-skeleton: calling handler
2018-02-06 05:45:30,417 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-regions: calling handler >
2018-02-06 05:45:30,418 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-regions: calling handler >
2018-02-06 05:45:30,419 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,419 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,420 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'us-east-1'.
2018-02-06 05:45:30,420 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,421 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-02-06 05:45:30,421 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,422 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-06 05:45:30,422 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2018-02-06 05:45:30,424 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/dist-packages/botocore/data/endpoints.json
2018-02-06 05:45:30,437 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,438 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler
2018-02-06 05:45:30,458 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ec2: calling handler
2018-02-06 05:45:30,459 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-02-06 05:45:30,468 - MainThread - botocore.endpoint - DEBUG - Setting ec2 timeout as (60, 60)
2018-02-06 05:45:30,470 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: ec2
2018-02-06 05:45:30,475 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeRegions: calling handler >
2018-02-06 05:45:30,477 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeRegions: calling handler
2018-02-06 05:45:30,478 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeRegions) (verify_ssl=True) with params: {'body': {'Action': u'DescribeRegions', 'Version': u'2016-11-15'}, 'url': u'https://ec2.us-east-1.amazonaws.com/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/1.14.32 Python/2.7.6 Linux/3.4.0+ botocore/1.8.36'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': }, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2018-02-06 05:45:30,480 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeRegions: calling handler >
2018-02-06 05:45:30,482 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeRegions: calling handler
2018-02-06 05:45:30,489 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,490 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,491 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'us-east-1'.
2018-02-06 05:45:30,492 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,494 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2018-02-06 05:45:30,495 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,500 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2018-02-06 05:45:30,502 - MainThread - botocore.session - DEBUG - Loading variable profile from instance vars with value u'bb-lonxt-shared'.
2018-02-06 05:45:30,503 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler
2018-02-06 05:45:30,507 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python2.7/dist-packages/botocore/data/sts/2011-06-15/service-2.json
2018-02-06 05:45:30,516 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sts: calling handler
2018-02-06 05:45:30,517 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2018-02-06 05:45:30,525 - MainThread - botocore.endpoint - DEBUG - Setting sts timeout as (60, 60)
2018-02-06 05:45:30,527 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: sts
2018-02-06 05:45:30,527 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sts.AssumeRole: calling handler
2018-02-06 05:45:30,529 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=AssumeRole) (verify_ssl=True) with params: {'body': {'Action': u'AssumeRole', 'RoleArn': 'arn:aws:iam::519810912671:role/platform-engineering', 'Version': u'2011-06-15', 'RoleSessionName': 'botocore-session-1517913930'}, 'url': u'https://sts.amazonaws.com/', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/1.14.32 Python/2.7.6 Linux/3.4.0+ botocore/1.8.36'}, 'context': {'auth_type': None, 'client_region': 'us-east-1', 'has_streaming_input': False, 'client_config': }, 'query_string': '', 'url_path': '/', 'method': u'POST'}
2018-02-06 05:45:30,531 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler >
2018-02-06 05:45:30,533 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler
2018-02-06 05:45:30,541 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-06 05:45:30,541 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20180206T104530Z

content-type;host;x-amz-date
824e9449a524d4587eed0f6a90888d5a0e30d58c35042ec027ea3fb468cdbcf5
2018-02-06 05:45:30,556 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180206T104530Z
20180206/us-east-1/sts/aws4_request
f20012ffdfc834c5be52b2c7bcc96d3fdd5e5de4222c337a2eb92d3da1969aa6
2018-02-06 05:45:30,565 - MainThread - botocore.auth - DEBUG - Signature:
5f5ff0f88c47684b2379923683a09d3de26d57a3dc28f2d995aa200c85585c5a
2018-02-06 05:45:30,589 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2018-02-06 05:45:30,591 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): sts.amazonaws.com
2018-02-06 05:46:30,734 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:46:30,765 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler
2018-02-06 05:46:30,767 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:46:30,799 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.338879448638
2018-02-06 05:46:30,806 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.338879448638 seconds
2018-02-06 05:46:31,147 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler >
2018-02-06 05:46:31,153 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler
2018-02-06 05:46:31,155 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-06 05:46:31,156 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20180206T104631Z

content-type;host;x-amz-date
824e9449a524d4587eed0f6a90888d5a0e30d58c35042ec027ea3fb468cdbcf5
2018-02-06 05:46:31,176 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180206T104631Z
20180206/us-east-1/sts/aws4_request
2eb0c726291a2d162a5d25df4dd1726e52e940f4f672bb926820123c14ded2a9
2018-02-06 05:46:31,190 - MainThread - botocore.auth - DEBUG - Signature:
08d95308e43a306d7de83716209549dfa5229245795ad7f0bcd64db1f0739231
2018-02-06 05:46:31,194 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2018-02-06 05:46:31,195 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (2): sts.amazonaws.com
2018-02-06 05:47:31,296 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:47:31,331 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler
2018-02-06 05:47:31,340 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:47:31,374 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 1.07551030122
2018-02-06 05:47:31,376 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 1.07551030122 seconds
2018-02-06 05:47:32,460 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler >
2018-02-06 05:47:32,467 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler
2018-02-06 05:47:32,470 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-06 05:47:32,471 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20180206T104732Z

content-type;host;x-amz-date
824e9449a524d4587eed0f6a90888d5a0e30d58c35042ec027ea3fb468cdbcf5
2018-02-06 05:47:32,501 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180206T104732Z
20180206/us-east-1/sts/aws4_request
60eb25862668d99aec308c11938b4d0523e3d29b9ae91d15fe61c83ef2f6958a
2018-02-06 05:47:32,517 - MainThread - botocore.auth - DEBUG - Signature:
5b15269df8072f3b19d3edf5712b23d7c8c99d20c2a315795ef3a767f3adb2e4
2018-02-06 05:47:32,527 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2018-02-06 05:47:32,528 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (3): sts.amazonaws.com
2018-02-06 05:48:32,733 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:48:32,752 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler
2018-02-06 05:48:32,754 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:48:32,785 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 2.40000928933
2018-02-06 05:48:32,788 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 2.40000928933 seconds
2018-02-06 05:48:35,190 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler >
2018-02-06 05:48:35,192 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler
2018-02-06 05:48:35,199 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-06 05:48:35,200 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20180206T104835Z

content-type;host;x-amz-date
824e9449a524d4587eed0f6a90888d5a0e30d58c35042ec027ea3fb468cdbcf5
2018-02-06 05:48:35,220 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180206T104835Z
20180206/us-east-1/sts/aws4_request
0aa3f9266f850137555852af2c463e3c04c3a40fdc6258f1a5b149c6c5a18a72
2018-02-06 05:48:35,235 - MainThread - botocore.auth - DEBUG - Signature:
ddb042cc220f5ac7e3273d03a5a00cabdc504d612e5290d29bd2a4e3f2a79a1d
2018-02-06 05:48:35,244 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2018-02-06 05:48:35,245 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (4): sts.amazonaws.com
2018-02-06 05:49:35,391 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:49:35,410 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler
2018-02-06 05:49:35,411 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:49:35,441 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 4.0502776938
2018-02-06 05:49:35,442 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 4.0502776938 seconds
2018-02-06 05:49:39,494 - MainThread - botocore.hooks - DEBUG - Event request-created.sts.AssumeRole: calling handler >
2018-02-06 05:49:39,500 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sts.AssumeRole: calling handler
2018-02-06 05:49:39,502 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-02-06 05:49:39,503 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20180206T104939Z

content-type;host;x-amz-date
824e9449a524d4587eed0f6a90888d5a0e30d58c35042ec027ea3fb468cdbcf5
2018-02-06 05:49:39,523 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20180206T104939Z
20180206/us-east-1/sts/aws4_request
05d9dc89227c985dd4b9aa69f673462fb8c014670c31d61cfebfcc08f0c0538c
2018-02-06 05:49:39,536 - MainThread - botocore.auth - DEBUG - Signature:
bad57897324354fb7b388a259fc7fa7ff53863e62d3ea05171f53fbd9afad3ea
2018-02-06 05:49:39,545 - MainThread - botocore.endpoint - DEBUG - Sending http request: 2018-02-06 05:49:39,546 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (5): sts.amazonaws.com
2018-02-06 05:50:39,654 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py", line 433, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:50:39,676 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sts.AssumeRole: calling handler
2018-02-06 05:50:39,677 - MainThread - botocore.credentials - WARNING - Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 431, in _protected_refresh
metadata = self._refresh_using()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 555, in fetch_credentials
return self._get_cached_credentials()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 565, in _get_cached_credentials
response = self._get_credentials()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 687, in _get_credentials
return client.assume_role(kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 602, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 143, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 172, in _send_request
success_response, exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 265, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:50:39,785 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 208, in main
return command_tableparsed_args.command
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 345, in __call__
return command_tableparsed_args.operation
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 517, in __call__
call_parameters, parsed_globals)
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 637, in invoke
client, operation_name, parameters, parsed_globals)
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 649, in _make_client_call
*parameters)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 602, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 143, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 168, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 152, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 148, in sign
auth = self.get_auth_instance(kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 227, in get_auth_instance
frozen_credentials = self._credentials.get_frozen_credentials()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 507, in get_frozen_credentials
self._refresh()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 415, in _refresh
self._protected_refresh(is_mandatory=is_mandatory_refresh)
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 431, in _protected_refresh
metadata = self._refresh_using()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 555, in fetch_credentials
return self._get_cached_credentials()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 565, in _get_cached_credentials
response = self._get_credentials()
File "/usr/local/lib/python2.7/dist-packages/botocore/credentials.py", line 687, in _get_credentials
return client.assume_role(kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 602, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 143, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 172, in _send_request
success_response, exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 265, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(*kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeout: HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)
2018-02-06 05:50:39,983 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

HTTPSConnectionPool(host='sts.amazonaws.com', port=443): Read timed out. (read timeout=60)

[blackbaud-sumitpandey]

Hey !! Anyone can help me on this??

[blackbaud-sumitpandey]

Please Update on the same

[yangyadi1993]

Got the same error. Is there anyone working on this?

[qianggenxiadeshuihu]

same issue in using awscli

aws version:
aws-cli/1.11.13 Python/3.5.2 Linux/4.15.0-39-generic botocore/1.4.70

debug info below:

2018-11-20 18:09:52,859 - Thread-9 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:52,860 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 112984064}) about to wait for the following futures []
2018-11-20 18:09:52,861 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 112984064}) done waiting for dependent futures
2018-11-20 18:09:52,861 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 112984064}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 112984064}
2018-11-20 18:09:52,865 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:53,150 - Thread-5 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 44826624}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:53,151 - Thread-5 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:53,151 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 44826624}) about to wait for the following futures []
2018-11-20 18:09:53,152 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 44826624}) done waiting for dependent futures
2018-11-20 18:09:53,152 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 44826624}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 44826624}
2018-11-20 18:09:53,157 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:54,009 - Thread-6 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 61603840}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:54,009 - Thread-6 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:54,010 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 61603840}) about to wait for the following futures []
2018-11-20 18:09:54,010 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 61603840}) done waiting for dependent futures
2018-11-20 18:09:54,010 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 61603840}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 61603840}
2018-11-20 18:09:54,015 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:54,025 - Thread-10 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 128712704}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:54,025 - Thread-10 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:54,026 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 128712704}) about to wait for the following futures []
2018-11-20 18:09:54,026 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 128712704}) done waiting for dependent futures
2018-11-20 18:09:54,027 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 128712704}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 128712704}
2018-11-20 18:09:54,031 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:54,720 - Thread-4 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 28573696}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:54,720 - Thread-4 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:54,721 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 28573696}) about to wait for the following futures []
2018-11-20 18:09:54,721 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 28573696}) done waiting for dependent futures
2018-11-20 18:09:54,722 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 28573696}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 28573696}
2018-11-20 18:09:54,726 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:55,416 - Thread-8 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 94109696}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:55,416 - Thread-8 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:55,417 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 94109696}) about to wait for the following futures []
2018-11-20 18:09:55,417 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 94109696}) done waiting for dependent futures
2018-11-20 18:09:55,418 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 94109696}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 94109696}
2018-11-20 18:09:55,422 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:55,603 - Thread-12 - s3transfer.futures - DEBUG - Submitting task IOWriteTask(transfer_id=0, {'offset': 163577856}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:09:55,603 - Thread-12 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:09:55,604 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 163577856}) about to wait for the following futures []
2018-11-20 18:09:55,604 - Thread-16 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 163577856}) done waiting for dependent futures
2018-11-20 18:09:55,604 - Thread-16 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 163577856}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7f0bed928f98>, 'offset': 163577856}
2018-11-20 18:09:55,608 - Thread-16 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:56,513 - Thread-14 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Read timed out.
2018-11-20 18:09:56,522 - Thread-14 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:56,897 - Thread-5 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:56,901 - Thread-7 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:59,106 - Thread-6 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:09:59,726 - Thread-4 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:00,354 - Thread-9 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:00,785 - Thread-10 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:01,715 - Thread-8 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:02,652 - Thread-12 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:05,288 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:10:08,193 - Thread-11 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:11:04,777 - Thread-13 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Read timed out.
2018-11-20 18:11:04,781 - Thread-13 - s3transfer.futures - DEBUG - Submitting task IORenameFileTask(transfer_id=0, {'final_filename': '/e/s3_dat/SpaceNet_Buildings_Competition_Round2_Sample.tar.gz'}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f0bee18a9e8> for transfer request: 0.
2018-11-20 18:11:04,781 - Thread-13 - s3transfer.utils - DEBUG - Acquiring 0
2018-11-20 18:11:04,781 - Thread-13 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-11-20 18:11:04,782 - Thread-16 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/e/s3_dat/SpaceNet_Buildings_Competition_Round2_Sample.tar.gz'}) about to wait for the following futures []
2018-11-20 18:11:04,782 - Thread-16 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/e/s3_dat/SpaceNet_Buildings_Competition_Round2_Sample.tar.gz'}) done waiting for dependent futures

[unacceptable]

@banjoman I believe that your issue is that you are doing a non-multipart upload for an object above 5GB.

[banjoman]

@ScriptMyJob good looks, I solved it a long time ago with an insane workaround, but that makes total sense.

[qianggenxiadeshuihu]

Just for those who had the similar issue with mine, here is my workaround:
use Amazon S3 chrome plugin to get the download link, then use XDM or FDM or others to download. By this way, I can download big files (>20G).
Hope it helps!