Aws-cli: progress bar UI for s3 copying

[kaihendry]

So I can have an idea how my transfer is doing please

[brendandixon]

There is a progress meter (of sorts) once the file exceeds a 5MB and is uploaded via multi-part:

[406c8f070d69:~] $ aws s3 cp 512MB.zip s3://bmdixonb/512MB.zip
Completed 22 of 74 part(s) with 1 file(s) remaining

Since files smaller than 5MB tend to complete in a second (or two), a progress bar would not help a lot.

Is this what you are seeing? What size was your file? How long did the upload take?

[kaihendry]

I now see the progress UI as you explained and tbh it's sufficient.

[ciphertxt]

If you're in higher latency parts of the world where connectivity isn't great (e.g. Australia), not having the progress output appear until after the first 5 MB is insufficient. I regularly upload 100 MB files from Sydney to an S3 bucket in the East US and it's beyond frustrating to have no output from the CLI.

Expected behavior:

1. Upload a file from the CLI with _aws s3 cp s3://bucket/file.ext_
2. View file progress/output as soon as the command executes

Current behavior:

1. Upload a file from the CLI with _aws s3 s3://bucket/file.ext_
2. Sit at an empty terminal for extended periods of time

The current behavior leads to a poor interaction and negatively impacts the experience of the _aws s3_ sp command. Would it be possible to reopen and address this?

[pastafari]

:point_up: I agree. I work from India - uploads are really slow. Some indication of progress - not necessarily a bar, just the fact that an upload is taking place - would be great!

[ArashJalalian]

Can you reopen this please? From Australia and having the same problem.

[betesh]

+1

[mcasto]

I just found a solution for myself for this problem. I changed the AWS CLI S3 configuration. Details for doing so can be found here: http://docs.aws.amazon.com/cli/latest/topic/s3-config.html

I set:
max_concurrent_requests = 5 (this prevents timeouts due to hogging bandwidth, probably not necessary for most people)
multipart_threshold = 1MB
multipart_chunksize = 1MB

By setting the threshold & chunksize to 1 MB, it starts chunking pretty quickly. And even in areas with high latency, it should provide feedback pretty quickly. Of course, you could set it for something even smaller. Then, by piping the output to a file, you can easily determine the progress.

The chunked output looks like "Completed 473 of 477 part(s) with 1 file(s) remaining"

For my purposes, wanting a graphical progress bar, I wrote a script to read the output file, find the last line, and parse the (in my example above) 473 & 477. From there, it's a simple matter to find percent complete and send that to my graphical progress bar.

[mcasto]

Well ... nuts. As I wrote the previous post, I was running a test, and everything seemed to be humming. Shortly after I finished the previous post, I got an error (EntityTooSmall) because the final chunk was smaller than the minimum allowed size. Now I'm looking into either changing minimum allowed size or, if it can't be changed, finding out what it is and setting my multipart_chunksize accordingly.

[mcasto]

Ah. Well. It appears 5MB is the minimum allowable size for multipart upload. So this is a dead end for providing any sort of feedback in a timely manner over a slow connection. Sorry if I got anyone's hopes up.

[andybergon]

The total parts number displayed even on slow connections would be very useful!

+1

[enissay14]

+1

[fro]

Agree, It could be helpful.

[ilijaljubicic]

+1

[aussiDavid]

+1

[attaboy]

+1

[adamatan]

+1
Uploading from Israel. Waiting for the first 5Mb can take a long time.

[sebastianvera]

+1

[adamatan]

We can use the rsync progress bar format - both the CLI parameter --progress and the nicely-built percentage and size preview.

[daniel-ro]

+1

[LufoX11]

+1 This is really necessary. I'm uploading 5 GB to S3 and I don't know what's comming on since I'm stuck at package X of N with a poor internet connection... and don't blame the connection!

Please consider doint it like s3cmd does: http://s3tools.org/s3cmd

[jrmiller82]

+1 This is necessary. Don't know why the issue is closed.

[avinash]

+1 I agree. A progress bar is essential when syncing or copying data from different parts of the world. I routinely transfer data from Singapore to the East Coast and sometimes it's difficult to understand what is happening...

[adamatan]

This can easily be done using boto3 (I have written my own 'aws s3 cp' clone just for the sake of having a progress bar).

[pehcha]

+1 I agree this is absolutely necessary for such a common tool.

[shockey]

+1.

[jrmiller82]

+1 again. :)

[toshke]

+1

[davidrabinowitz]

+1, amazing a progress bar is missing from such action which may be lengthy

[venkatd]

+1 here too. Not sure why this isn't implemented yet.

[davidrabinowitz]

I have open a new issue (#2269) for raising this, as it seems this issue will remain close. I would appreciate it if everyone who had +1'ed this issue will do the same for the new one so we can get this going.

[andyfowler]

This appears to be mostly addressed in 1.11.0, (with improvements in subsequent versions).

[adamatan]

Yup. About time.

On Fri, Nov 4, 2016 at 11:08 PM Andy Fowler [email protected]
wrote:

This appears to be mostly addressed in 1.11.0
https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst#1110, (with
improvements in subsequent versions).
[image: image]
https://cloud.githubusercontent.com/assets/107073/20022485/3404db78-a2b1-11e6-8178-7177dae6b64c.png

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-cli/issues/312#issuecomment-258547853, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA95rVFZdUcZi9MO_EXSrhKlZd7VDwmLks5q657PgaJpZM4A9qzG
.