Aws-sdk-go: service/s3/s3manager: Should be able to track upload progress

[avitex]

I think a useful feature would be the ability to watch, for example, the progress of an upload to a s3 bucket.

[jasdel]

Hi @avitex What method of watching for S3 uploads were you thinking? Were you looking to monitor uploads via the s3manager.Uploader or just S3's PutObject?

[mischief]

you can already do this if you make your own io.Reader that tracks bytes read.

[avitex]

@jasdel Uploads via the s3manger.Uploader.
@mischief I had already tried this, but seemed to 'upload' instantly, perhaps my code was bugged?

[jasdel]

@avitex an io.Reader will only allow you to monitor when a part is prepared for upload to S3, not the upload progress it self. I think this feature request would require the s3manager.Upload to expose some functionality to track the when individual parts have completed their upload, or some kind of overall progress counter.

[benhinchley]

I have just been looking around about how to do this nicely, If I find anything I will post it here, but this would definitely be a nice feature to have

[ghost]

Is there any update on this, we have tried to implement such a feature but its impossible with the current upload manager.

[jasdel]

Hi @minervadata, thanks for your support of this issue. This issue is still in our backlog, and would like to implement this feature, but have not started work on it yet. We're also always more than glad to review PRs if anyone is looking to add this feature to the s3 manager. In addition we can discuss the design of the feature here too, to help the implementation.

[krishnakhandagale]

Is upload progress feature implemented yet?

[jasdel]

Hi @krishnakhandagale thanks for voting for support for this feature. This work is in our backlog and hand not be started yet. We're glad to review PRs that look at adding progress this issue though. It would be good to have a design chat prior to a PR to hash out how progress could be exposed.

[oleynikd]

No workaround for this?

[maccam912]

In my fork I ended up just adding in a progress bar (https://github.com/cheggaaa/pb) and as each part in a multipart upload is completed it adds that progess to the bar. It's not perfect, but it works for me. Maybe a good starting point?

It just adds another channel for each uploader to send back the number of bytes uploaded and each time something is put in that channel the progress bar updates.

https://github.com/maccam912/aws-sdk-go/blob/master/service/s3/s3manager/upload.go#L712

https://github.com/maccam912/aws-sdk-go/blob/master/service/s3/s3manager/upload.go#L673

[xinst]

I have just create a pull request, and add an example for upload with progress
https://github.com/aws/aws-sdk-go/pull/1868

[diehlaws]

Example added in https://github.com/aws/aws-sdk-go/commit/50ba1dfe47983b15b160b66c730a3b93d2961f8e - closing issue.

[dshearer]

This would be quite useful for me too.