Aws-sdk-php: Multipart Upload with S3MultiRegionClient empty parts

Created on 7 Mar 2017  路  15Comments  路  Source: aws/aws-sdk-php

aws/aws-sdk-php version 3.20.3

I have some problems with using multipart upload with multi-region client. Steps to reproduce:

  1. Use upload method for file which > than file part option which calls multipart methods.
  2. It will successfully call createMultipartUpload
  3. Then calling a bunch of uploadPart methods with success response with ETags
  4. Then calling completeMultipartUpload with empty parts parameter and throw exception about failed validation of XML.

Why parts parameter is empty?

I fix it with getClientFromPool for region method upload wrapper above upload method of S3MultiRegionClient

bug
Source
picture quantizer
👍3

Most helpful comment

@imshashank
As I have already written in my first post in this issue, you should try to upload file with size bigger than 16 MB (value of constant ObjectUploader::DEFAULT_MULTIPART_THRESHOLD).

For about 2 weeks you have been trying to reproduce a bug for which I have posted the script which reproduces this bug with 100% probability. Have you ever run it?

So I need to repeat reproducing steps again.

How to reproduce

  1. Create a bucket bucket-in-ap-southeast-2 in the ap-southeast-2 region.
  2. Create AWS IAM key/secret pair and grant full access to this bucket.
  3. Create empty directory aws-test and install AWS SDK:
composer require 'aws/aws-sdk-php':'3.20.3'
  1. Create script test.php with the following content:
<?php
require __DIR__ . '/vendor/autoload.php';

$filename = tempnam(sys_get_temp_dir(), 'aws-test-');

file_put_contents($filename, str_repeat('a', 20 * 1048576));

$client = new \Aws\S3\S3MultiRegionClient([
    'credentials' => [
        'key' => 'AWS_KEY',
        'secret' => 'AWS_SECRET',
    ],
    'region' => 'us-east-1',
    'version' => 'latest',
]);

$client->upload('BUCKET', 'test/20mb', fopen($filename, 'r'));
  1. Replace in this script:
  2. AWS_KEY - put actual AWS key
  3. AWS_SECRET - put actual AWS secret
  4. BUCKET - put actual bucket name in ap-southeast-2 region
  5. Do not change anything else in the script.
  6. Run script:
php test.php

Actual result

Fatal error: Uncaught GuzzleHttp\Exception\ClientException: Client error: `POST https://s3-ap-southeast-2.amazonaws.com/BUCKET/test/20mb?uploadId=...` resulted in a `400 Bad Request` response:
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our publis (truncated...)
 in vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:111
Stack trace:
#0 vendor/guzzlehttp/guzzle/src/Middleware.php(65): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response))
#1 vendor/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 vendor/guzzlehttp/promises/src/Promise.php(156): Guzzl in vendor/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php on line 135

Expected result
There shouldn't be any errors and file should be successfully uploaded to the bucket.

picture javer on 20 Mar 2017
👍2

All 15 comments

@quantizer Thanks for reporting the issue. Can you please share the stacktrace of the error. Also which version of php, OS & curl you are using. This would help us in reproducing the issue.

Can you please share the code for upload that you have used and option that are being passed.

Thanks.

imshashank picture imshashank on 8 Mar 2017

@imshashank

PHP version 7.1.0 (also tested with 7.1.2)
curl 7.35.0 (also tested with 7.51.0)
OS Ubuntu 14.04

original exception:

[Aws\Exception\MultipartUploadException]                                                                                   
  An exception occurred while completing a multipart upload: Error executing "CompleteMultipartUpload" on "http://s3.ca-central-1.amazonaws.com/filepath?uploadId=UPLOADID"; AWS HTTP error: Client error: `POST http://s3.ca-central-1.amazonaws.com/filepath?uploadId=UPLOADID` resulted in a `400 Bad Request` response:                                                                                                  
  <Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our publis (truncated...)                                                                                                                                             
   MalformedXML (client): The XML you provided was not well-formed or did not validate against our published schema - <Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>         
  Request_ID</RequestId><HostId>HOST_ID</HostId>

Exception trace:

 () at workdir/vendor/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php:135
 Aws\Multipart\AbstractUploadManager->Aws\Multipart\{closure}() at workdir/vendor/guzzlehttp/promises/src/Promise.php:203
 GuzzleHttp\Promise\Promise::callHandler() at workdir/vendor/guzzlehttp/promises/src/Promise.php:156
 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}() at workdir/vendor/guzzlehttp/promises/src/TaskQueue.php:47
 GuzzleHttp\Promise\TaskQueue->run() at workdir/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:96
 GuzzleHttp\Handler\CurlMultiHandler->tick() at workdir/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:123
 GuzzleHttp\Handler\CurlMultiHandler->execute() at workdir/vendor/guzzlehttp/promises/src/Promise.php:246
 GuzzleHttp\Promise\Promise->invokeWaitFn() at workdir/vendor/guzzlehttp/promises/src/Promise.php:223
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:266
 GuzzleHttp\Promise\Promise->invokeWaitList() at workdir/vendor/guzzlehttp/promises/src/Promise.php:225
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:269
 GuzzleHttp\Promise\Promise->invokeWaitList() at workdir/vendor/guzzlehttp/promises/src/Promise.php:225
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:62
 GuzzleHttp\Promise\Promise->wait() at workdir/vendor/guzzlehttp/promises/src/Coroutine.php:65
 GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}() at workdir/vendor/guzzlehttp/promises/src/Promise.php:246
 GuzzleHttp\Promise\Promise->invokeWaitFn() at workdir/vendor/guzzlehttp/promises/src/Promise.php:223
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:266
 GuzzleHttp\Promise\Promise->invokeWaitList() at workdir/vendor/guzzlehttp/promises/src/Promise.php:225
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:62
 GuzzleHttp\Promise\Promise->wait() at workdir/vendor/guzzlehttp/promises/src/Coroutine.php:65
 GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}() at workdir/vendor/guzzlehttp/promises/src/Promise.php:246
 GuzzleHttp\Promise\Promise->invokeWaitFn() at workdir/vendor/guzzlehttp/promises/src/Promise.php:223
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:266
 GuzzleHttp\Promise\Promise->invokeWaitList() at workdir/vendor/guzzlehttp/promises/src/Promise.php:225
 GuzzleHttp\Promise\Promise->waitIfPending() at workdir/vendor/guzzlehttp/promises/src/Promise.php:62
 GuzzleHttp\Promise\Promise->wait() at workdir/vendor/aws/aws-sdk-php/src/S3/S3ClientTrait.php:27
 Aws\S3\S3MultiRegionClient->upload() at workdir/src/somephpfile.php

and completeMultipartUpload with debug true option:

command was set to workdir/vendor/aws/aws-sdk-php/src/TraceMiddleware.php:255:
 array(3) {
   'instance' =>
   string(32) "instance_ID"
   'name' =>
   string(23) "CompleteMultipartUpload"
   'params' =>
   array(5) {
     'MultipartUpload' =>
     array(1) {
       'Parts' =>
       array(0) {
       }
     }
     'UploadId' =>
     string(128) "UPLOAD_ID"
     'Bucket' =>
     string(24) "bucket_name"
     'Key' =>
     string(57) "filepath"
     '@http' =>
     array(3) {
       'timeout' =>
       int(3600)
       'connect_timeout' =>
       int(60)
       'debug' =>
       resource(4633) of type (stream)
     }
   }
 }

As working code for multi-region uploading we are using:

S3MultiRegionClient::getClientFromPool($region)->upload();
quantizer picture quantizer on 9 Mar 2017

Simple PHP snippet to reproduce this issue:

$filename = tempnam(sys_get_temp_dir(), 'aws-test-');

file_put_contents($filename, str_repeat('a', 20 * 1048576));

$client = new \Aws\S3\S3MultiRegionClient([
    'credentials' => [
        'key' => 'KEY',
        'secret' => 'SECRET',
    ],
    'region' => 'us-east-1',
    'version' => 'latest',
]);

$client->upload('bucket-in-ap-southeast-2', 'test/20mb', fopen($filename, 'r'));

Results in:

Fatal error: Uncaught GuzzleHttp\Exception\ClientException: Client error: `POST https://s3-ap-southeast-2.amazonaws.com/bucket-in-ap-southeast-2/test/20mb?uploadId=5NZL0AK90PlIA61j4qQ7MwVKj5L_e9nIdVsf6qFQ_.J9Dqbr9NV0bqzdVOH_sSoxnBKwZ5L6Sk_kJ1OkSLIovbxfB26ddjLa3otfEKfjYIOKgIbNmHGQmco43AxN0Xlqek88C5YZ0VxzWGf1mC5.yw--` resulted in a `400 Bad Request` response:
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our publis (truncated...)
 in vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:111
Stack trace:
#0 vendor/guzzlehttp/guzzle/src/Middleware.php(65): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response))
#1 vendor/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 vendor/guzzlehttp/prom in vendor/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php on line 135

If you decrease file size to 10 MB (i.e. less than ObjectUploader::DEFAULT_MULTIPART_THRESHOLD = 16777216 which triggers using MultipartUploader) - everything is ok.

javer picture javer on 9 Mar 2017
👍1

@quantizer @javer

So far I have not been able to reproduce the issue. I have tested on php 5.6, php 7.1 and using the latest SDK and also 3.20.3. For me, the file gets uploaded and I receive no exceptions or errors.

Do you guys receive this error every time?

imshashank picture imshashank on 10 Mar 2017

@imshashank Have you tested with the bucket located not in us-east-1 region? It's important moment, bucket should be not from the default region, this issue is 100% reproducible for me with buckets located in regions: ap-southeast-2, ca-central-1, eu-west-2, i.e. any non-default region.

When I changed bucket to the bucket in us-east-1 region - everything works well.

javer picture javer on 11 Mar 2017
👍1

@imshashank
Also after turning debug option to ON I saw that with bucket in non-default region UploadPart command has never entered "mup" sign handler, but if I use bucket in the default region - this handler was called:
https://github.com/aws/aws-sdk-php/blob/master/src/Multipart/AbstractUploader.php#L60

I suppose it's due to the late determining that we should call regional client instead of the default (only after PermanentRedirectException): https://github.com/aws/aws-sdk-php/blob/master/src/S3/S3MultiRegionClient.php#L208
After that getRegionalizedCommand creates a new Command just from the name and arguments:
https://github.com/aws/aws-sdk-php/blob/master/src/S3/S3MultiRegionClient.php#L259
I guess it's the place where "mup" handler from the original command is lost, because creating a new command clones HandlerList from the regional client and doesn't take into account handlers from the original command.

javer picture javer on 11 Mar 2017
👍1

@imshashank yes, @javer right. I forgot to mention about non-default region for uploading.

quantizer picture quantizer on 13 Mar 2017

@quantizer @javer Thanks for clarifying. I am investigating the bug right now and will keep this thread updated.

imshashank picture imshashank on 13 Mar 2017

@quantizer @javer Hey, I have been working on reproducing this bug and finding a fix. The bug was reproducible if the bucket is in a different region and the client was initiated with a different region.

Can you please confirm that the region passed to the client is same as the region for your bucket.

Also, the developer guide explains more on this:

http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multiregion-client.html

// If you would like to specify the region to which to send a command, do so
// by providing an @region parameter
$objects = $s3Client->listObjects([
    'Bucket' => $bucketName,
    '@region' => 'eu-west-1',
]);
imshashank picture imshashank on 16 Mar 2017

@imshashank Yes, I can confirm that MultiRegionClient was initiated with default us-east-1 region, but for uploading a bucket from another region (ap-southeast-2) was used. I believe it's for what MultiRegionClient is intended for - when I don't care which region should be used. If I would know the region of the bucket - there is no sense to use MultiRegionClient.

Also I have checked your recommendation to use @region for upload command and can confirm that it doesn't work as described, this option doesn't have any influence on the executing flow, you can check it yourself by enabling the debug mode. More over, this option doesn't change anything regardless multipart uploading flow or simple is used.

So I can confirm that option @region doesn't work at all at least for S3MultiRegionClient::upload().

But I can confirm that option @region works at least for S3MultiRegionClient::headObject().

javer picture javer on 16 Mar 2017
👍2

@javer Hi, S3MultiRegionClient is used to avoid creating a new client for buckets in different regions and hence the same client can be used for different regions by just passing a variable.

Also, I can confirm that I created a client in us-eat-1 and was able to upload an object to a bucket in ap-southeast-2 etc.

Are u getting the same error when you have created a client in us-east-1 and trying to upload an object to a bucket in region like ap-southeast-2?

imshashank picture imshashank on 17 Mar 2017

@imshashank
As I have already written in my first post in this issue, you should try to upload file with size bigger than 16 MB (value of constant ObjectUploader::DEFAULT_MULTIPART_THRESHOLD).

For about 2 weeks you have been trying to reproduce a bug for which I have posted the script which reproduces this bug with 100% probability. Have you ever run it?

So I need to repeat reproducing steps again.

How to reproduce

  1. Create a bucket bucket-in-ap-southeast-2 in the ap-southeast-2 region.
  2. Create AWS IAM key/secret pair and grant full access to this bucket.
  3. Create empty directory aws-test and install AWS SDK:
composer require 'aws/aws-sdk-php':'3.20.3'
  1. Create script test.php with the following content:
<?php
require __DIR__ . '/vendor/autoload.php';

$filename = tempnam(sys_get_temp_dir(), 'aws-test-');

file_put_contents($filename, str_repeat('a', 20 * 1048576));

$client = new \Aws\S3\S3MultiRegionClient([
    'credentials' => [
        'key' => 'AWS_KEY',
        'secret' => 'AWS_SECRET',
    ],
    'region' => 'us-east-1',
    'version' => 'latest',
]);

$client->upload('BUCKET', 'test/20mb', fopen($filename, 'r'));
  1. Replace in this script:
  2. AWS_KEY - put actual AWS key
  3. AWS_SECRET - put actual AWS secret
  4. BUCKET - put actual bucket name in ap-southeast-2 region
  5. Do not change anything else in the script.
  6. Run script:
php test.php

Actual result

Fatal error: Uncaught GuzzleHttp\Exception\ClientException: Client error: `POST https://s3-ap-southeast-2.amazonaws.com/BUCKET/test/20mb?uploadId=...` resulted in a `400 Bad Request` response:
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our publis (truncated...)
 in vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:111
Stack trace:
#0 vendor/guzzlehttp/guzzle/src/Middleware.php(65): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response))
#1 vendor/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 vendor/guzzlehttp/promises/src/Promise.php(156): Guzzl in vendor/aws/aws-sdk-php/src/Multipart/AbstractUploadManager.php on line 135

Expected result
There shouldn't be any errors and file should be successfully uploaded to the bucket.

javer picture javer on 20 Mar 2017
👍2

@quantizer @javer I am sorry that it took long for me to fully understand the bug. I appreciate your patience.

There are two approaches that the SDK uses in multiregionclient. For some errors, the SDK will look at the header and fix the problem for the client regarding bucket regions. For errors the SDK is not able to resolve, it will just show the error.

We are working on more explicit error messages and figuring out a better approach to working with multiple region clients in S3.

I have marked this as a bug. Will keep this thread updated.

Thanks a lot for your patience.

imshashank picture imshashank on 20 Mar 2017

@quantizer @javer The fix for the bug has been merged. Sorry that it took a bit longer to fix this one. Hope this will resolve the bug for you.

imshashank picture imshashank on 30 Mar 2017

@imshashank this fix will be in next release?

quantizer picture quantizer on 30 Mar 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

liquorvicar picture liquorvicar  路  5Comments
Intrepidity picture Intrepidity  路  4Comments
chrisguitarguy picture chrisguitarguy  路  3Comments
LCamel picture LCamel  路  4Comments
muratsplat picture muratsplat  路  4Comments