Aws-sdk-go: Is ListParts safe to use to CompleteMultipartUpload()?

[harshavardhana]

This issue is a more or less a clarification.

While developing a client side tool we found that we cannot use ListParts to complete an ongoing multipart upload on S3 recommended to be not used in the following documentation

https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html

Only use the returned listing for verification. You should not use the result of this listing when sending a complete multipart upload request. Instead, maintain your own list of the part numbers you specified when uploading parts and the corresponding ETag values that Amazon S3 returns.

Now two questions come to mind here

• Why is ListParts recommended to not be used when doing a CompleteMultipartUpload()?
• Why is ListParts never used in UploadManager? Is resuming never really supported? - I understand why true resuming cannot be implemented here, but wanted to know if you have more insights.

Version of AWS SDK for Go?

master

Version of Go (go version)?

1.8.3

[jasdel]

Thanks for reaching out to us @harshavardhana I think the best way to get answer to you question is via the Amazon S3 forums. There others who use S3 and may be able to help you with your question.

Why is ListParts never used in UploadManager? Is resuming never really supported? - I understand why true resuming cannot be implemented here, but wanted to know if you have more insights.

The S3 Upload Manager does not use ListParts because the uploader has full knowledge of the parts as they are uploaded. These parts numbers are only used in an upload failure condition to remove stale parts preventing them from taking space in your S3 bucket. Once a terminal failure occurs, and retries have been exausted the uploader will return the list of parts that were uploaded on error if LeavePartsOnError is true on Uploader.

[harshavardhana]

Thanks for reaching out to us @harshavardhana I think the best way to get answer to you question is via the Amazon S3 forums. There others who use S3 and may be able to help you with your question.

Okay I thought this is the right place, thanks for clarifying will ask in Amazon S3 forums.

[harshavardhana]

The S3 Upload Manager does not use ListParts because the uploader has full knowledge of the parts as they are uploaded.

So lets say uploadManager is not used then ListParts would be considered a valid way to do CompleteMultipartUpload?

[jasdel]

I would not recommend using ListParts when performing a CompleteMultipartUpload. While today it may work it is documented that the part numbers returned should not be used for completing a multipart.

The recommended way is for the application to maintain the list of parts that were uploaded and use that list for completing the multi-part upload.

[harshavardhana]

I would not recommend using ListParts when performing a CompleteMultipartUpload. While today it may work it is documented that the part numbers returned should not be used for completing a multipart.

The recommended way is for the application to maintain the list of parts that were uploaded and use that list for completing the multi-part upload.

@jasdel do you know what is the reason behind this ? is it due to the List() not being consistent with an in progress UploadPart() ?

[jasdel]

Sorry I do not know that off hand. This would be my assumption that ListParts is not guaranteed to be consistent. Whereas maintaining the list locally is always consistent because the application has knowledge of all the parts it uploaded.

[jasdel]

Thanks for reaching out to us @harshavardhana . Let us know if you have any additional questions or feedback with the SDK. Since it looks like this question is answered i'm going to close it. Please reach back out if you have further questions.