Aws-sdk-php: uploadDirectory() always replaces all files, does not update them

[vfuentesedcorp]

Hi,

Whenever I execute the command it updates all the files, this I check it in S3 management in last modified. The files are unchanged, but still transfer again. If I modify a css file, the time it takes is too long, always the same time, considering that only one file has changed.

• I'm working on Symfony 3
• I have my assets folder in web/bundles/*
• I implemented a command to update the assets to an S3 bucket, which simply calls the function:

...
 $aws = array(
            'credentials' => array(
                'key'    => $key,
                'secret' => $secret,
            ),
            'region' => $region,
            'version'=> 'latest'
        );
        $this->s3 = new S3Client($aws);
    }
...
$this->s3->uploadDirectory($urlDirectory,$this->bucket,'', array(
            'params'      => array('ACL' => 'public-read'),
            'debug'       => true
        ));

Am I doing something wrong? Is it a known problem?

[cjyclaire]

@vfuentesedcorp the problem is with your params, for the option array, here are available options. Could you explain your params usage there? Are you trying to transfer only object qualified with public-read? If so, you should try before option instead

[vfuentesedcorp]

@cjyclaire I used it only to indicate that the files were public. Yes, I am transferring only public files. I'm going to try what you tell me.

Thanks, I'll tell you if it works.

[vfuentesedcorp]

@cjyclaire, Do you tell me to use the option before to check if a file exists?

Implement Transfer:

...
 $urlDirectory=$this->getContainer()->get('kernel')->getRootDir() . '/../web/bundles';
  $dest = 's3://'.$this->bucket."/".$urlToSaveDirectory;
        $manager = new Transfer($this->s3, $urlDirectory, $dest);
        $result=$manager->transfer();
...

I do not quite understand how to use the option before for uploads from multiple directories.
Could you guide me a little?

Thanks!

[cjyclaire]

@vfuentesedcorp Taking a second look at your code snippet, are you trying to "transferring" files from a S3 location to another S3 location? If so, S3 transfer manager/ uploadDirectory won't work because it works when :
If the $source argument is an Amazon S3 URI, then the $dest argument must be a local directory (and vice versa).
If my above assumption is correct, you would need to use copyObject instead.

[cjyclaire]

@vfuentesedcorp Adding to my last comment, if not, there is an example of how to customize command arguments inside Transfer Manager.
In this case, I assuming you are trying to download files with 'public-read' from S3 to local dir?
Then you might need to do something like:

$uploader = new Transfer($s3Client, $source, $dest, [
    'before' => function (\Aws\Command $command) {
        if (in_array($command->getName(), ['GetObject'])) {
            // Do something with 
           // GetObject (http://docs.aws.amazon.com/aws-sdk-php/v3/api/api-s3-2006-03-01.html#getobject)
        }
    },
]);

Since when GetObject requires Bucket and Key to identifier a object, in order to filter out objects with required permissions, you might need to use GetObjectAcl to help filter out qualified objects first then transfer those qualified objects.

[vfuentesedcorp]

@cjyclaire

• Thank you very much, but what I'm doing is transferring files from outside of S3 to S3
• Used PutObject
• Still taking the same time
• I do not understand what I should include in the function ( if (in_array($command->getName(), ['PutObject'])) { )

$manager = new Transfer($this->s3, $urlDirectory, $dest, [
                'before' => function (Command $command) {
                    // Set custom cache-control metadata
                   if (in_array($command->getName(), ['PutObject'])) {                        
                     $command['CacheControl'] = 'max-age=3600';
                      ....
                    }
                 },
                'debug'=>true
            ]);

[cjyclaire]

@vfuentesedcorp I see, so I'm a bit confused, from outside of S3, how do you know whether a file is public-read? If you are trying to set those files to public-read in S3, you would just need to do:

$uploader = new Transfer($s3Client, $source, $dest, [
    'before' => function (\Aws\Command $command) {
        // Commands can vary for multipart uploads, so check which command
        // is being processed
        if (in_array($command->getName(), ['PutObject', 'CreateMultipartUpload'])) {
            // Apply a canned ACL
            $command['ACL'] = strpos($command['Key'], 'CONFIDENTIAL') === false
                ? 'public-read'
                : 'private';
        }
    },
]);

You can configure the command with anything you would like to set with PutObject and CreateMultipartUpload

[vfuentesedcorp]

@cjyclaire I tried what indicas me. Yet still it is taking too long, no improvement.
I do not know what else to prove the truth, anyway the delay is not so long, but it would be nice to be able to optimize it and not always upload all the files.

[cjyclaire]

@vfuentesedcorp appreciate for following up, so to clear to my confusions, are you trying to filter out some files when upload to s3 from a directory? And all parameter options for PutObject doesn't work for you?

I totally understand that unsatisfied performance feels bad, I'd like to see how can I help or optimize it there. Yet condition based filtering is taken cared by S3 service operational APIs, it's not appropriate for SDK to make changes there.

[cjyclaire]

Closing, yet feel free to reopen with further details or questions :)

[vfuentesedcorp]

@cjyclaire
We had a problem and I had to leave this aside. I tell you, it turns out that I had not tried running the command from the server, I was doing it from my local to S3, I tried it from server to S3 and it takes less than 5 seconds. So perfect! It worked with the configurations that you indicated to me, with Transfer and with uploadDirectory ().

Very thankful.

[curtiscarlson]

@cjyclaire This should be re-opened. I am having the same issue. $client->uploadDirectory() is not working properly. It always detects a change on all files even when they have not changed. I suspect @vfuentesedcorp had a windows development machine and a linux server and that the problem occurs in windows environments. I know this was 3 years ago but I am having this exact issue. Thank you.