Aws-cli: aws s3 sync is not reliable

[jamestwebber]

As was described in #2297: aws s3 sync has been consistently failing to upload all of the files asked of it, without saying anything about failures. We're doing this from an EC2 instance (from an attached st1 volume), trying to uploading ~16000 files to s3. In some cases only a few thousand of those are uploaded. As these files were the product of hours of computation on EC2 this is quite aggravating.

aws-cli version:

~$ aws --version
aws-cli/1.11.157 Python/2.7.12 Linux/4.4.0-89-generic botocore/1.7.15

[jamestwebber]

For what it's worth I think we were having the same issues with aws s3 cp so this is probably not a sync-specific issue. Copies are failing silently and reporting success.

[nkadel-skyhook]

The "aws s3 sync" command is not the best integrated or supported of the awscli components. This is a situation where the lighter weight "s3cmd" tool may work better for you. Barring that, I've had good success with doing a "aws s3 sync --exact-timestamps --dryrun", running "aws s3 sync --exact-timestamps --quiet" in a loop to reduce the progress spew, and re-running "aws s3 sync --exact-timestamps --dryrun" to verify successful delivery afterwards.

It's a lot of extra work, but I've had too many deliveries fail partway through or be interrupted by local network issues to consider "aws s3 sync" reliable without a verification step.

[kyleknap]

Interesting, I wonder if it is related to the fact that you are reading from EBS volumes. In the past, I have seen scenarios where the read throughput is exceeded on the EBS volume and you get throttled in reading in the data to upload it to S3. Do you happen to have any debug logs that can help us dive into this? You can get debug logs by appending --debug to the command you ran.

[jamestwebber]

I started adding --debug to our commands and added a bunch of logging, which of course guarantees that we haven't seen these errors recently.

When the next big bolus of data processing comes down the pipe we should have more diagnostic information available.

[kyleknap]

Closing this issue for now. @jamestwebber if you are able to run into this issue again and have debug logs, please open a new issue.

[glarrain]

I just got bit by this, while trying to copy all the contents of a bucket to a new one. This a severe bug

[jrthorne]

When I installed aws tools, there was a warning about the wrong version of boto. Installing
pip install botocore==1.8.49
seemed to help. Just to note that I keep the aws tools and botocore in a separate environment to my app. Note also that the s3 bucket is not a traditional file system, and can often be counter intuitive and inconsistent when syncing.

[jqadev]

Same issue
awscli==1.16.74

@kyleknap please open this issue.

[benhannel]

I am also experience this issue @kyleknap. Please fix it as soon as possible.