Aws-cli: Uploading a stream failed

[nbrownus]

While uploading a stream to s3 the upload failed but the aws process did not die. The error occurred well after the upload started. I can only assume that aws tried to upload more than 9999 parts?

Output from other_process | aws s3 cp - s3://thebucket/thefile

upload failed: ./- to s3://thebucket/thefile
A client error (InvalidArgument) occurred when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

Version info

$ aws --version
aws-cli/1.7.12 Python/2.7.3 Linux/3.2.0-59-generic

[nbrownus]

Well I didn't read enough of the documentation.

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload. due to too many parts in upload.

[nbrownus]

Actually reopening because the upload failure did not cause aws to exit at all

[kyleknap]

@nbrownus
How long did you wait after the upload failed? Sometimes the threads need to finish uploading their respective part before they begin shutting down and cleaning up.

[nbrownus]

I noticed something was wrong after bytes transmitted out of the network interface dropped to 0. I manually stopped the process about 15 minutes after traffic stopped. I saw the above error in the logs but everything before aws in the pipeline was still chugging along.

[kyleknap]

Thanks! We will look into this.

[kyleknap]

With the release of 1.10.57, we replaced the underlying architecture for streaming and this should no longer be an issue as errors handling is much resilient and quicker in shutting down the entire transfer process. There should be no more hanging as well. Resolving issue.

[eladnava]

@kyleknap Unfortunately I just got this error when uploading a stream on aws-cli/1.15.76:

An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

[jpellman]

I've also received this error with aws-cli/1.15.76.

[combinatorist]

And I just got this error with aws-cli/1.16.102

[combinatorist]

... but setting --expected-size (as @nbrownus suggested) fixed it for me.