Borg: High memory usage on server after upgrading to 1.1.x

[robsmith11]

I've updated client and server to 1.1.4 from a 1.0.x version and recreated the repo using "borg init -e repokey-blake2".

The server is a VPS with only 512MB of RAM. This always worked fine when using 1.0.x, but within a minute of backing up a 5GB remote client, the borg server ate through all the memory and was killed.

Was there a change that is expected to require more memory? Are there any options for running the server in low-memory environment?

[ThomasWaldmann]

So you say you only did:

• borg init
• borg create (backing up 5GB)

And that's all you needed to run the server out of memory?

Usually, I'ld say the repository index (hashtable) is the biggest memory consumer on the server side (for big repos), but if you only backed up 5GB, that can't be the problem.

Can you give us some more data, like how memory consumption develops for borg 1.0.x and 1.1.x (both with fresh repos and same input data)?

[robsmith11]

The only commands I ran on the client were

borg init -e repokey-blake2 $repo
borg create --compression lz4 $repo::$prefix-`date +%Y%m%d` /my_dir

and the only change I made after the upgrade was the addition of the required -e option.

Memory usage increased linearly with time over about a minute while the borg create was running. I added a swap file and reran, and the entire backup completed in just over a minute, so 512MB was almost enough memory to run it. I'm not sure what memory usage was like before.. it's possible that there wasn't a big difference and I was just lucky to have just enough memory.

Unfortunately, I don't have time to do a deeper dive, so I'll just close this for now, unless there's something easy you'd like me to test.

[ThomasWaldmann]

@robsmith11 when you say "memory usage increased linearly", do you mean the borg process used more and more memory (like e.g. seen in "top")?

[robsmith11]

@ThomasWaldmann Yes, I was watching it in htop (on the 2nd attempt after it had been killed for eating all the memory the 1st time).

[mhaehnel]

I see a similar problem (albeit with 1.1.3). Archive is on a remote. I run create using:

borg create -v --filter AME --stats -p --exclude-from=exclude --remote-path=/Backups/borg/bin/bin/borg.sh ssh://user@host/Backups/borg.host::host-initial  /

When reaching a 28GB file that borg tries to backup the server side process (borg serve) starts eating all the RAM. RAM usage is roughly proportional to the dedouplicated size (D) progress of the file backed up (as shown when giving -p). Once it also fills the SWAP it fails with the following error output:

Remote: Exception ignored in: <bound method Repository.__del__ of <Repository /Backups/borg.host>>                                                                              
Remote: Traceback (most recent call last):
Remote:   File "/Backups/borg/bin/lib/python3.5/site-packages/borgbackup-1.1.3-py3.5-linux-x86_64.egg/borg/repository.py", line 168, in __del__
Remote:     assert False, "cleanup happened in Repository.__del__"
Remote: AssertionError: cleanup happened in Repository.__del__
Remote: Local Exception
Remote: Traceback (most recent call last):
  File "/Backups/borg/bin/lib/python3.5/site-packages/borgbackup-1.1.3-py3.5-linux-x86_64.egg/borg/archiver.py", line 4157, in main
    exit_code = archiver.run(args)
  File "/Backups/borg/bin/lib/python3.5/site-packages/borgbackup-1.1.3-py3.5-linux-x86_64.egg/borg/archiver.py", line 4089, in run
    return set_ec(func(args))
  File "/Backups/borg/bin/lib/python3.5/site-packages/borgbackup-1.1.3-py3.5-linux-x86_64.egg/borg/archiver.py", line 253, in do_serve
    storage_quota=args.storage_quota,
  File "/Backups/borg/bin/lib/python3.5/site-packages/borgbackup-1.1.3-py3.5-linux-x86_64.egg/borg/remote.py", line 223, in serve
    unpacker.feed(data)
  File "/Backups/borg/bin/lib/python3.5/site-packages/msgpack_python-0.5.4-py3.5-linux-x86_64.egg/msgpack/fallback.py", line 292, in feed
    self._buffer += view
MemoryError

Is there a way to backup files that exceed the ram size of the target host?

[mhaehnel]

This seems to be independent of individual file size. RAM usage on the 'borg serve' side seems to be roughly equal to the deduplicated size (D) from the progress on the client side.

[ThomasWaldmann]

Looks like this needs more investigation. I tried to reproduce once, but could not, but as multiple people seem to have issues, guess I'll try again.

[jtlawson]

Experiencing the same problem here with 1.1.4. Server on RHEL7.

./borg-run

Using a pure-python msgpack! This will result in lower performance.
Remote: Using a pure-python msgpack! This will result in lower performance.
Enter passphrase for key ssh://[email protected]/gpfsback1/xxx-borg1/borgtest1:
Remote: Exception ignored in: >
Remote: Traceback (most recent call last):
Remote: File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/borg/repository.py", line 171, in __del__
Remote: assert False, "cleanup happened in Repository.__del__"
Remote: AssertionError: cleanup happened in Repository.__del__
Remote: Local Exception
Remote: Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4175, in main
exit_code = archiver.run(args)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4107, in run
return set_ec(func(args))
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/borg/archiver.py", line 254, in do_serve
storage_quota=args.storage_quota,
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/borg/remote.py", line 223, in serve
unpacker.feed(data)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/msgpack/fallback.py", line 292, in feed
self._buffer += view
MemoryError

Platform: Linux xxxxxxxx-xxxx4.xxx.edu 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64
Linux: Red Hat Enterprise Linux Server 7.4 Maipo
Borg: 1.1.4 Python: CPython 3.6.3
PID: 4862 CWD: /users/j/t/jtl
sys.argv: ['/opt/rh/rh-python36/root/usr/bin/borg', 'serve', '--umask=077']
SSH_ORIGINAL_COMMAND: None

"borg serve" continued to allocate vram until there wasn't any left.

This repo was made ("init") starting with 1.1.4, no upgrade was done.

[ThomasWaldmann]

@jtlawson one obvious issue of your installation is that it used the pure python msgpack. This is slower, not sure about whether it is different in memory allocation behaviour.

[ThomasWaldmann]

@robsmith11 @mhaehnel do you also see the warning about the pure python msgpack in your borg output?

[mhaehnel]

@ThomasWaldmann yes. I also have this warning (and only on the remote/serv side). I intsalled msgpack and the dev package there and will run compile and run borg again now. I'll report back what I find. Thanks for the help!

[ThomasWaldmann]

can you all add infos about whether you use pure python msgpack (see warning above) AND the msgpack(-python) version you use?

[ThomasWaldmann]

Maybe a pure python msgpack (fallback implementation) bug, see link above.

[ThomasWaldmann]

So, if that is the case, this is one more reason to use the C implementation of msgpack (not the pure python fallback). We have the package requirements for this documented in our install docs (e.g. a C++ compiler).

Besides this potential bug, you want this anyway as it is much faster. This is why we emit that warning about slow pure python msgpack.

To fix, you need to uninstall msgpack(-python) package, install the build requirements (see our docs), then reinstall msgpack(-python) package and then check if the warning is gone.

[jtlawson]

Re-installed with C implementation of msgpack (took a bit to figure out the deps)

It's definitely not chewing up virtual memory like it was before ... Thanks!

[ThomasWaldmann]

\o/

Ok, I am closing this as it is not a bug in borg, but in msgpack pure python implementation, see bug there.

[mhaehnel]

Just to confirm (albeit a bit late) that the native msgpack did also solve the memory issue for me. Thanks for the quick help!

[ThomasWaldmann]

@mhaehnel yup, msgpack confirmed the bug in the pure python implementation. It is rarely used, only when installation is done in a sub-optimal way (e.g. no compiler). This also explains why I failed to reproduce the issue at first, I didn't use the pure python msgpack.

[m8nky]

I run into the same issue. Pure python came to my deployments, because I used PyPi to install borg.
pip install -U borgbackup
It depends on msgpack-python - which is deprecated since 0.5. The current version 0.5.4 of msgpack-python did not deploy native libs to my system (Debian 9 and Mint 18), which resulted in the fallback of the python implementation.

Recommendation:

• Update the package dependency from msgpack-python to msgpack, as stated in the official msgpack README (https://github.com/msgpack/msgpack-python/blob/master/README.rst).
• msgpack 0.5.4 on PyPi worked just fine, native libs are deployed to both systems.

[ThomasWaldmann]

If msgpack-python 0.5.4 is broken, please file a github issue there.

For the stable series (1.0, 1.1) I guess we rather should not change from msgpack-python to msgpack. The stable stuff is packaged in linux (and other distributions), so we rather should not switch names there.

What I did for stable is to require msgpack < 0.5.0 (it used to be very stable until some breakage began after 0.5.0 due to the renaming and other incompatible / troublesome changes).
https://github.com/borgbackup/borg/blob/1.1-maint/setup.py#L42