Borg: Add bandwidth limit support

[chebee7i]

This issue appears on the checkbox list in #5. It is attic issue: https://github.com/jborg/attic/issues/103. I wasn't sure if we should be adding comments there or here.

---

The original issue requests that support be added to limit bandwidth. trickle was mentioned as an option. People said it wouldn't/didn't work since borg uses a pipe through ssh, while trickle works with sockets.

I just wanted to report that if you are using borg with sshfs (i.e. backing up to rsync.net), then using trickle will work properly. You must call trickle when performing the ssh mount. So to limit bandwidth to 5 Mbps, you'd use:

trickle -u 5000 sshfs username@hostname: /path/to/mountpoint
borg create ...

[manwegit]

Limiting bandwidth can be accomplished with pipeviewer and a wrapper script.
My take on this is that documenting pv is preferable to adding bandwidth support to borg, because pv does what is needed and rate-limiting is not always easy to do in code (+ adds more complexity to already complex setup). With unix philosophy: do one thing and do it well (and pipes, I like pipes... ;)

Install pipeviewer (pv) on ubuntu and most likely debian it's simply: sudo apt-get install pv

Create a wrapper script: (I'll create it to /usr/local/bin/pv-wrapper)

#!/bin/bash
     ## -q, --quiet              do not output any transfer information at all
     ## -L, --rate-limit RATE    limit transfer to RATE bytes per second
export RATE=307200
pv -q -L $RATE  | "$@"

That done, add BORG_RSH env
export BORG_RSH='/usr/local/bin/pv-wrapper.sh ssh '

Now borg will be bandwidth limited. Nice thing about pv is that you can change rate-limit on the fly:
pv -R $(pidof pv) -L 102400

I've used same setup with rsync where I was moving multi terabyte set of data and had to moderate bandwidth during office hours.

# example cron setup:
00 09 * * 1-5 root ( pidof pv > /tmp/.pv-pid && pv -R $(cat /tmp/.pv-pid ) -L  307200)
00 18 * * * root ( pidof pv > /tmp/.pv-pid && pv -R $(cat /tmp/.pv-pid ) -L  819200)

Test:
Remote repo was removed before each test

source: 320M total
Without pipeviewer:
Duration: 5 minutes 8.50 seconds
Number of files: 60

## wrapper:
pv -q -L 307200 | "$@"
Duration: 18 minutes 7.00 seconds

pv -q -L 102400 | "$@"
Stopped exection after 40 mins, expected run time around 54 minutes and borg --progress output was inline with the estimate.

Original test with pipeviewer, but no rate-limit:
Duration: 6 minutes 29.38 seconds
please note that this test was not run on laboratory conditions so other users affect bandwidth availability.

[ThomasWaldmann]

could be added to faq.

[ThomasWaldmann]

merged pr #705.

[ddevault]

Bumping an old issue here, I know, but that's a rather convoluted workaround for something that probably ought to live in borg. Plus, it would be useful to have separate throttling options for network, I/O, and compute.

[ThomasWaldmann]

@ddevault if you have a specific problem related to this, please describe it in a new issue.