Botocore: S3UploadFailedError when i upgrade botocore,

[TreHack]

When I upgrade botocore from 1.5.47 to the latest version, I get the below error, Is there a bug in latest verion botocore? The version of 1.5.47 is ok.

S3UploadFailedError: Failed to upload I:\work\service-filesystem\yolanda\media\49973840-66cf-11e7-a216-1c1b0dc04405-H40-common-branch2-muzhiwan.apk to hdg1-testlab24/49973840-66cf-11e7-a216-1c1b0dc04405-H40-common-branch2-muzhiwan.apk: An error occurred (InvalidArgument) when calling the CreateMultipartUpload operation: Unknown

[JordonPhillips]

Could you share a snippet showing what you're doing?

[TreHack]

from boto3.session import Session
from botocore.client import Config
from boto3.s3.transfer import S3Transfer, TransferConfig

s3client = Session().client('s3', **connection_kwargs)
config = TransferConfig(**transfer_kwargs)
transfer = S3Transfer(s3client, config)
transfer.upload_file(file_path, settings.AWS_STORAGE_BUCKET_NAME, key)

the connection_kwargsandtransfer_kwargsis ok. the error happens when i call transfer.upload_file method.

[stealthycoin]

Can you provide us with debug logs for this?

[TreHack]

The traceback is below:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "I:\yolanda\apps\util\s3storage.py", line 32, in _do_wrap_errors
    return func(self, name, *args, **kwargs)
  File "I:\yolanda\apps\util\s3storage.py", line 155, in _save
    file_path, settings.AWS_STORAGE_BUCKET_NAME, name)
  File "I:\yolanda\env\lib\site-packages\boto3\s3\transfer.py", line 283, in upl
oad_file
    filename, '/'.join([bucket, key]), e))
S3UploadFailedError: Failed to upload I:\yolanda\media\862528ee-66dc-11e7-967b-0
05056c00008-lima.apk to hdg1-testlab24/862528ee-66dc-11e7-967b-005056c00008-lima
.apk: An error occurred (InvalidArgument) when calling the PutObject operation:
Unknown

>>> name = "862528ee-66dc-11e7-967b-005056c00008-lima.apk"
>>> file_path = "I:\yolanda\media\862528ee-66dc-11e7-967b-005056c00008-lima.apk"
>>> bucket = "hdg1-testlab24"
>>> s3.transfer.upload_file(file_path, bucket, name)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "I:\yolanda\env\lib\site-packages\boto3\s3\transfer.py", line 283, in upl
oad_file
    filename, '/'.join([bucket, key]), e))
S3UploadFailedError: Failed to upload I:\yolanda\media\862528ee-66dc-11e7-967b-0
05056c00008-lima.apk to hdg1-testlab24/862528ee-66dc-11e7-967b-005056c00008-lima
.apk: An error occurred (InvalidArgument) when calling the PutObject operation:
Unknown

[TreHack]

When I downgrade botocore to version 1.5.47, everything is ok!

>>> name = "862528ee-66dc-11e7-967b-005056c00008-lima.apk"
>>> file_path = "I:\yolanda\media\862528ee-66dc-11e7-967b-005056c00008-lima.apk"

>>> bucket = "hdg1-testlab24"
>>> s3.transfer.upload_file(file_path, bucket, name)
>>>

[joguSD]

It's difficult to pinpoint the issue from just the traceback.
Can you enable the botocore debug logs and post the output for us?
You can enable debug logs by adding a call to Session.set_debug_logger().

[TreHack]

set_debug_logger is not work?

>>> from boto3.session import Session
>>> Session.set_debug_logger()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
AttributeError: type object 'Session' has no attribute 'set_debug_logger'
>>> 

[joguSD]

Ah, the above is to enable botocore.session debug logs, not boto3.session. Sorry if that was unclear.

The following will enable boto3 and botocore debug logs:

import boto3
boto3.set_stream_logger('')

Try adding that and posting the output.

[TreHack]

The debug output is below:

>>> name = "862528ee-66dc-11e7-967b-005056c00008-lima.apk"
>>> file_path = "I:\yolanda\media\862528ee-66dc-11e7-967b-005056c00008-lima.apk"
>>> bucket = "hdg1-testlab24"
>>> s3.transfer.upload_file(file_path, bucket, name)

2017-07-21 11:36:26,141 s3transfer.utils [DEBUG] Acquiring 1
2017-07-21 11:36:26,142 s3transfer.tasks [DEBUG] UploadSubmissionTask(transfer_i
d=1, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x03953F10
>}) about to wait for the following futures []
2017-07-21 11:36:26,142 s3transfer.tasks [DEBUG] UploadSubmissionTask(transfer_i
d=1, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x03953F10
>}) done waiting for dependent futures
2017-07-21 11:36:26,144 s3transfer.tasks [DEBUG] Executing task UploadSubmission
Task(transfer_id=1, {'transfer_future': <s3transfer.futures.TransferFuture objec
t at 0x03953F10>}) with kwargs {'osutil': <s3transfer.utils.OSUtils object at 0x
037AF530>, 'client': <botocore.client.S3 object at 0x03B2EA70>, 'config': <boto3
.s3.transfer.TransferConfig object at 0x0379FF50>, 'transfer_future': <s3transfe
r.futures.TransferFuture object at 0x03953F10>, 'request_executor': <s3transfer.
futures.BoundedExecutor object at 0x03B2EF70>}
2017-07-21 11:36:26,144 s3transfer.futures [DEBUG] Submitting task PutObjectTask
(transfer_id=1, {'extra_args': {}, 'bucket': 'hdg1-testlab24', 'key': '862528ee-
66dc-11e7-967b-005056c00008-lima.apk'}) to executor <s3transfer.futures.BoundedE
xecutor object at 0x03B2EF70> for transfer request: 1.
2017-07-21 11:36:26,144 s3transfer.utils [DEBUG] Acquiring 1
2017-07-21 11:36:26,147 s3transfer.tasks [DEBUG] PutObjectTask(transfer_id=1, {'
extra_args': {}, 'bucket': 'hdg1-testlab24', 'key': '862528ee-66dc-11e7-967b-005
056c00008-lima.apk'}) about to wait for the following futures []
2017-07-21 11:36:26,147 s3transfer.utils [DEBUG] Releasing acquire 1/None
2017-07-21 11:36:26,147 s3transfer.tasks [DEBUG] PutObjectTask(transfer_id=1, {'
extra_args': {}, 'bucket': 'hdg1-testlab24', 'key': '862528ee-66dc-11e7-967b-005
056c00008-lima.apk'}) done waiting for dependent futures
2017-07-21 11:36:26,148 s3transfer.tasks [DEBUG] Executing task PutObjectTask(tr
ansfer_id=1, {'extra_args': {}, 'bucket': 'hdg1-testlab24', 'key': '862528ee-66d
c-11e7-967b-005056c00008-lima.apk'}) with kwargs {'extra_args': {}, 'client': <b
otocore.client.S3 object at 0x03B2EA70>, 'bucket': 'hdg1-testlab24', 'key': '862
528ee-66dc-11e7-967b-005056c00008-lima.apk', 'fileobj': <s3transfer.utils.ReadFi
leChunk object at 0x03B4F1D0>}
2017-07-21 11:36:26,151 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <function validate_ascii_metadata at 0x036EBC30>
2017-07-21 11:36:26,154 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <function sse_md5 at 0x036EB6B0>
2017-07-21 11:36:26,155 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <function convert_body_to_file_like_object at 0x036EBF
70>
2017-07-21 11:36:26,157 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <function validate_bucket_name at 0x036EB670>
2017-07-21 11:36:26,157 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <bound method S3RegionRedirector.redirect_from_cache o
f <botocore.utils.S3RegionRedirector object at 0x03B2EDF0>>
2017-07-21 11:36:26,157 botocore.hooks [DEBUG] Event before-parameter-build.s3.P
utObject: calling handler <function generate_idempotent_uuid at 0x036EB4B0>
2017-07-21 11:36:26,158 botocore.hooks [DEBUG] Event before-call.s3.PutObject: c
alling handler <function conditionally_calculate_md5 at 0x036EB630>
2017-07-21 11:36:26,178 botocore.hooks [DEBUG] Event before-call.s3.PutObject: c
alling handler <function add_expect_header at 0x036EB8B0>
2017-07-21 11:36:26,180 botocore.handlers [DEBUG] Adding expect 100 continue hea
der to request.
2017-07-21 11:36:26,183 botocore.hooks [DEBUG] Event before-call.s3.PutObject: c
alling handler <bound method S3RegionRedirector.set_request_url of <botocore.uti
ls.S3RegionRedirector object at 0x03B2EDF0>>
2017-07-21 11:36:26,184 botocore.endpoint [DEBUG] Making request for OperationMo
del(name=PutObject) (verify_ssl=True) with params: {'body': <s3transfer.utils.Re
adFileChunk object at 0x03B4F1D0>, 'url': u'http://s3.nie.netease.com/hdg1-testl
ab24/862528ee-66dc-11e7-967b-005056c00008-lima.apk', 'headers': {'Content-MD5':
u'qjHTtUIXlOZQNgYB20bmug==', 'Expect': '100-continue', 'User-Agent': 'Boto3/1.4.
4 Python/2.7.12 Windows/7 Botocore/1.5.85'}, 'context': {'auth_type': None, 'cli
ent_region': u'us-east-1', 'signing': {'bucket': 'hdg1-testlab24'}, 'has_streami
ng_input': True, 'client_config': <botocore.config.Config object at 0x03B2EAF0>}
, 'query_string': {}, 'url_path': u'/hdg1-testlab24/862528ee-66dc-11e7-967b-0050
56c00008-lima.apk', 'method': u'PUT'}
2017-07-21 11:36:26,186 botocore.hooks [DEBUG] Event request-created.s3.PutObjec
t: calling handler <function disable_upload_callbacks at 0x037600B0>
2017-07-21 11:36:26,187 botocore.hooks [DEBUG] Event request-created.s3.PutObjec
t: calling handler <bound method RequestSigner.handler of <botocore.signers.Requ
estSigner object at 0x03B2EAB0>>
2017-07-21 11:36:26,187 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject:
 calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <bo
tocore.client.ClientCreator object at 0x038C4530>>
2017-07-21 11:36:26,188 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject:
 calling handler <function set_operation_specific_signer at 0x036EB430>
2017-07-21 11:36:26,190 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: c
alling handler <function switch_to_virtual_host_style at 0x03619F30>
2017-07-21 11:36:26,190 botocore.utils [DEBUG] Checking for DNS compatible bucke
t for: http://s3.nie.netease.com/hdg1-testlab24/862528ee-66dc-11e7-967b-005056c0
0008-lima.apk
2017-07-21 11:36:26,190 botocore.utils [DEBUG] URI updated to: http://hdg1-testl
ab24.s3.nie.netease.com/862528ee-66dc-11e7-967b-005056c00008-lima.apk
2017-07-21 11:36:26,226 botocore.auth [DEBUG] Calculating signature using v4 aut
h.
2017-07-21 11:36:26,226 botocore.auth [DEBUG] CanonicalRequest:
PUT
/862528ee-66dc-11e7-967b-005056c00008-lima.apk

content-md5:qjHTtUIXlOZQNgYB20bmug==
host:hdg1-testlab24.s3.XXX.com
x-amz-content-sha256:9d85026283215e5cf4c0f0c87f1ecbca98367939b247d41e52360935a1b
df29c
x-amz-date:20170721T033626Z

content-md5;host;x-amz-content-sha256;x-amz-date
9d85026283215e5cf4c0f0c87f1ecbca98367939b247d41e52360935a1bdf29c
2017-07-21 11:36:26,229 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20170721T033626Z
20170721/us-east-1/s3/aws4_request
6b7aa7c4dc39f430ff915614f9475df65e8a26c22d79dc067d4402579dcbd050
2017-07-21 11:36:26,230 botocore.auth [DEBUG] Signature:
fe7ead248413c2a0bc326df55f77fca7950edf4141531d48604d03afcab32ac9
2017-07-21 11:36:26,230 botocore.hooks [DEBUG] Event request-created.s3.PutObjec
t: calling handler <function enable_upload_callbacks at 0x037600F0>
2017-07-21 11:36:26,233 botocore.endpoint [DEBUG] Sending http request: <Prepare
dRequest [PUT]>
2017-07-21 11:36:26,236 botocore.vendored.requests.packages.urllib3.connectionpo
ol [INFO] Resetting dropped connection: hdg1-testlab24.s3.nie.netease.com
2017-07-21 11:36:26,266 botocore.awsrequest [DEBUG] Waiting for 100 Continue res
ponse.
2017-07-21 11:36:26,292 botocore.awsrequest [DEBUG] 100 Continue response seen,
now sending request body.
2017-07-21 11:36:35,226 botocore.vendored.requests.packages.urllib3.connectionpo
ol [DEBUG] "PUT /862528ee-66dc-11e7-967b-005056c00008-lima.apk HTTP/1.1" 400 81
2017-07-21 11:36:35,230 botocore.parsers [DEBUG] Response headers: {'content-len
gth': '81', 'accept-ranges': 'bytes', 'server': 'openresty', 'connection': 'keep
-alive', 'x-amz-request-id': 'tx0000000000000028beb9e-00597176c2-2d5e2-hdg1', 'd
ate': 'Fri, 21 Jul 2017 03:36:34 GMT', 'content-type': 'application/xml'}
2017-07-21 11:36:35,230 botocore.parsers [DEBUG] Response body:
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code></Error
>
2017-07-21 11:36:35,233 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: c
alling handler <botocore.retryhandler.RetryHandler object at 0x03AFFEB0>
2017-07-21 11:36:35,233 botocore.retryhandler [DEBUG] No retry needed.
2017-07-21 11:36:35,233 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: c
alling handler <bound method S3RegionRedirector.redirect_from_error of <botocore
.utils.S3RegionRedirector object at 0x03B2EDF0>>
2017-07-21 11:36:35,234 s3transfer.tasks [DEBUG] Exception raised.
Traceback (most recent call last):
  File "I:\yolanda\env\lib\site-packages\s3transfer\tasks.py", line 126, in __ca
ll__
    return self._execute_main(kwargs)
  File "I:\yolanda\env\lib\site-packages\s3transfer\tasks.py", line 150, in _exe
cute_main
    return_value = self._main(**kwargs)
  File "I:\yolanda\env\lib\site-packages\s3transfer\upload.py", line 679, in _ma
in
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "I:\yolanda\env\lib\site-packages\botocore\client.py", line 310, in _api_
call
    return self._make_api_call(operation_name, kwargs)
  File "I:\yolanda\env\lib\site-packages\botocore\client.py", line 599, in _make
_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidArgument) when calling the PutObject oper
ation: Unknown
2017-07-21 11:36:35,236 s3transfer.utils [DEBUG] Releasing acquire 1/None
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "I:\yolanda\env\lib\site-packages\boto3\s3\transfer.py", line 283, in upl
oad_file
    filename, '/'.join([bucket, key]), e))
S3UploadFailedError: Failed to upload I:\yolanda\media\862528ee-66dc-11e7-967b-0
05056c00008-lima.apk to hdg1-testlab24/862528ee-66dc-11e7-967b-005056c00008-lima
.apk: An error occurred (InvalidArgument) when calling the PutObject operation:
Unknown
>>>

[joguSD]

Looks like you're using a custom endpoint. We recently switched to signature v4 to sign requests by default, which is possibly not supported by your endpoint. You could try setting the config signature_version to s3.

[TreHack]

yes, I'm using the custom endpoint that my company provided, thank you, It works.

[JordonPhillips]

Glad it worked out!