Aws-cli: Add ability to limit bandwidth for S3 uploads/downloads

[jamesls]

Original from #1078, this is a feature request to add the ability for the aws s3 commands to limit the amount of bandwidth used for uploads and downloads.

In the referenced issue, it was specifically mentioned that some ISPs charge fees if you go above a specific mbps, so users need the ability to limit bandwidth.

I imagine this is something we'd only need to add to the aws s3 commands.

[AustinSnow]

Hello jamesis,
Could you provide a timeframe when the bandwidth limit could become available?
Thanks
austinsnow

[kjohnston]

:+1:

[beauhoyt]

:+1:

[bhegazy]

:+1:

[seattledoug]

:+1:

[dsclassen]

[godefroi]

:+1:

[rayterrill]

:+1:

[kazeburo]

:+1:

[isaoshimizu]

:+1:

[quiver]

Under Unix-flavor systems, trickle comes in handy for ad-hoc throttling. trickle hooks socket-APIs using LD_PRELOAD and throttles bandwidth.

You can run commands something like

$ trickle -s -u {UPLOAD_LIMIT(KB/s)} command
$ trickle -s -u {UPLOAD_LIMIT(KB/s)} -d {DOWNLOAD_LIMIT(KB/s)} command

Built-in feature will be really useful, but given cross-platform nature of AWS-CLI, it can cost a lot to implement and maintain it.

[binaryorganic]

Trickle is specifically mentioned in issue #1078 which is linked to in the first comment here. The two (trickle and AWS-CLI) just don't play nice together in my experience.

[isp0000]

:+1:

[l3rady]

:+1:

[andrefelipe]

+1

[ddehghan]

+1

[joshpelz]

:+1:

[whiteadam]

+1

[mikeg0]

+1

[nhumphreys]

:+1:

[apeschar]

(Y)

[JulienChampseix]

:+1:

[ikoniaris]

:+1: this is much needed!

[aegixx]

:+1:

[fotag]

:+1:

[tlelson]

+1

[garygitton]

+1

[yayugu]

:+1:

[pavel-kazhavets]

👍

[jrmiller82]

Any update on this feature?

[mjaverto]

:+1:

[Necklaces]

[pakdel]

:+1:

[mtrcn]

👍

[walllam]

👍

[erikedlund]

[ikoniaris]

Any updates on this?

[lynchc]

👍

[erlinghaugstad]

👍

[fkei]

👍

[tristanbes]

👍

[zmalik]

👍

[aflugge]

:+1:

[moses-moore-spafax]

On the one hand: much faster than s3cmd.
On the other hand: my hosting company automatically halted a server for using "suspiciously high amounts of bandwidth".
Someone suggested aws configure set default.s3.max_concurrent_requests $n where $n is less than 10. Not sure if that is enough; will investigate the trickle tool mentioned above.

[nikitasius]

👍

[cobaltjacket]

Over two years in, and this request is still outstanding. Is there a timeframe by which this could be implemented?

[hilyin]

👍

[zouyixiong]

👍

[zouyixiong]

👍

[danielpfarmer]

👍🏿

[nullobject]

:+1:

[leonsmith]

Just nuked the internet in a shared office.
This would be a nice feature when you want to be kind to other people
👍

[pticyn]

👍

[markdavidburke]

you can use trickle -s -u 100 aws s3 sync . s3://examplebucket

[ikoniaris]

@sofuca does this work correctly though? There are many people that have tried trickle for this but the results were questionable. See #1078.

[markdavidburke]

@ikoniaris

Works perfectly for me.

The following command nukes the internet in the office, it's a 20Mb/s connection

aws s3 cp /foo s3://bar

And the following command uploads at a nice 8Mb/s

trickle -s -u 1000 aws s3 sync /foo s3://bar

Screen shot of the outside interface of the firewall I'm using

[mxins]

👍

[ctaperts]

Trickle and large s3 files will cause the trickle to crash

[wadejensen]

(y)

[ctaperts]

sorry, Trickle and large s3 files will cause the trickle to crash using boto3 with 10 concurrent(default settings) uploads, changing the concurrent uploads will resolve the issue. I need to add this in the boto3's github, thanks!

[bhicks-usa]

👍

So it's been over 2.5 years since this was opened. Is this request just being ignored?

[tantra35]

For us we use pv(https://linux.die.net/man/1/pv) in this maner:

/usr/bin/pv -q -L 20M $l_filepath | /usr/local/bin/aws s3 cp --region "us-east-1" - s3://<s3-bucket>/<path in s3 bucker>

This solution is not ideal(because it require additional support for filtering and recursion, we do it inside bash loop) but much better than trickle which in our case uses 100% of CPU, and behaves very unstable

Here our full usecase of pv(we limit upload speed to 20MB/s == 160Mbit/s)

for l_filepath in /logs/*.log-*; do
    l_filename=`basename $l_filepath`
    /usr/bin/pv -q -L 20M $l_filepath | /usr/local/bin/aws s3 cp --region "us-east-1" - s3://$S3BUCKET/${HOSTNAME}/$l_filename
    /bin/rm $l_filepath
done

[jonoaustin]

+1

Real life use case: Very large upload to S3 over DX, do not want to saturate the link and potentially impact production applications using the DX link.

[ischoonover]

throttle, trickle and pv all do not work for me on archlinux with the latest awscli from pip when uploading to a bucket. I have additionally set max_concurrent_connections for s3 in the awscli configuration to 1 with no difference made. This would be a much appreciated addition!

[tantra35]

@ischoonover seems that you don't pass --expected-size to aws cli when use it with pv, it very useful when you try to upload very big files

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload due to too many parts in upload.

[ischoonover]

@tantra35 Size was 1GB. I ended up using s3cmd, which has rate limiting built in with --limit-rate

[joguSD]

Implemented in #2997.