Aws-sdk-php: Using Async S3 Transfer With PHP SDK V3

[pihish]

I've been following the guide at https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-transfer.html to upload local directories onto S3 using the PHP V3 SDK.

I am able to successfully complete directory uploads synchronously using

$source = "c:/my-folder/90";
$dest = "s3://my-bucket/90";
$manager = new \Aws\S3\Transfer($client, $source, $dest);
$manager->transfer();

But when I try to upload my directory asynchronously using

$source = "c:/my-folder/90";
$dest = "s3://my-bucket/90";
$manager = new \Aws\S3\Transfer($client, $source, $dest);
$promise = $manager->promise();
echo 'one';
$promise->then(function(){
     echo 'hi';
 });
echo 'two';

The folder isn't uploaded. I am able to run through the script without errors, but the folder never shows up on S3. one and two are returned, but `hi`` isn't so it looks like the callback is never performed.

What am I missing here?

[jeskew]

You will need to wait on the promise at some point. The SDK isn't forking off background processes or farming work off to a NodeJS-style persistent global event loop, so scripts will need to include a directive to handle outstanding promises.

What the SDK _can_ do with promises is handle them concurrently, which can speed up scripts that involve a lot of I/O time. In fact, the transfer manager promise is composed of promises tracking the upload of individual files. The individual promises are handled five at a time, and when all uploads are complete, the transfer manager's meta promise is fulfilled. If you wanted to upload multiple directories, you could compose two transfer manager promises into another meta promise:

$promises = [];

$source = "c:/my-folder/90";
$dest = "s3://my-bucket/90";
$manager = new \Aws\S3\Transfer($client, $source, $dest);
$promises []= $manager->promise();

$source = "c:/my-other-folder/90";
$dest = "s3://my-other-bucket/90";
$manager = new \Aws\S3\Transfer($client, $source, $dest);
$promises []= $manager->promise();

$metaPromise = \GuzzleHttp\Promise\all($promises)->wait();

The SDK would then interleave those two directory uploads and spend less time simply waiting for HTTP responses.

Hope that helps!

[pihish]

I am still a little confused on how I can wait on the promises. I am thinking of promises in a JavaScript context - as callback functions I define when the event is completed. Can you show me example syntax on how this is done?

[jeskew]

The SDK uses Guzzle promises, which support synchronous waiting by calling ->wait() on a promise. Since a PHP process doesn't have a global event loop like node, scripts need to explicitly run through outstanding promises. Calling wait is the easiest way to do so.

[jeskew]

Closing, as it seems like all questions have been answered. Please feel free to reopen if you have any questions or concerns.

[neerajrathod05]

Hey, Will this work for uploading directory to S3
$s3->uploadDirectory($new_sourcefolderpath, $bucket, $new_destinationfolder)

[aayusharyan]

So, it's like 4 years after the initial question and we still haven't found a way to achieve True Async operations?

[nmeegama]

I have tried the code in
https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-transfer.html
still the sample code on that page does not work

[SamRemis]

Hi all, while there are ways to achieve true async in PHP, the most reliable libraries to do so require a minimum of PHP 7.0, which would break backward compatibility because we support 5.5+. We are looking to implement this in the next major version of the SDK.