borgbackup master not working with python-msgpack 1.0

[cjk]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes. This issue was forked from #4558

Is this a BUG / ISSUE report or a QUESTION?

Issue, but not strictly a bug (yet)

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Can't check because borg crashes with python-msgpack 1.0
But I'm using the latest master branch on Arch-Linux.

Operating system (distribution) and version.

Linux cjk-tux 5.5.13-arch2-1 #1 SMP PREEMPT Mon, 30 Mar 2020 20:42:41 +0000 x86_64 GNU/Linux

Full borg commandline that lead to the problem (leave away excludes and passwords)

All commands lead to the problem.

Describe the problem you're observing.

After upgrading my Arch system (rolling release) to the current version, borg-backup no longer works, most likely due to Arch now using python-msgpack 1.0

Guess borg should eventually be compatible with the latest stable msgpack (e.g. 1.x)

Example error:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 1.0.0 (/usr/lib/python3.8/site-packages), Requirement.parse('msgpack<=0.6.2,>=0.5.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack<=0.6.2,>=0.5.6' distribution was not found and is required by borgbackup

[FelixSchwarz]

This is somewhat expected because {setup.py hard-codes the version requirement](https://github.com/borgbackup/borg/blob/master/setup.py#L74) - Arch must patch the setup.py when creating the package if they use the system-provided python-msgpack.

As a workaround please edit the requires.txt for borgbackup (somewhere in /usr/lib/python3.8/site-packages). That should get you a running borgbackup (though it is unknown if msgpack 1.0 works correctly unless the test suite passes).

[RonnyPfannschmidt]

did arch verify this wont corrupt backups or is arch just in yolo mode again

[oddfellow]

As far as I understand, Arch does not use the system-provided python-msgpack but instead uses the bundled msgpack, since this commit:

https://git.archlinux.org/svntogit/community.git/commit/trunk?h=packages/borg&id=cf797f293b0f1efec1261000f5b0f746cbdd5358

Works fine here on an up-to-date Arch... @cjk do you use Arch Linux or some derivative distribution?

[cjk]

@oddfellow I use standard Arch, but I should probably have emphasized that I use the borg-git package from AUR, not the stable release from community. Sorry for making that not clear enough.
As such, this is not a critical bug, but rather an issue that should be worked on before the next release, i guess.
I was just surprised that borg can break in such ways, because usually the upgrade-process in Arch is rather smooth, even for AUR-git-packages.
But I also understand that this behavior could prevent possible backup-corruption, so this should be handled with care.

[oddfellow]

I see... borg master branch does not seem to have a bundled msgpack, only 1.1-maint... but I could be wrong, I was just searching for "prefer_system_msgpack" in the repo and I did not find it in master branch.

[cjk]

As a workaround please edit the requires.txt for borgbackup (somewhere in /usr/lib/python3.8/site-packages). That should get you a running borgbackup (though it is unknown if msgpack 1.0 works correctly unless the test suite passes).

Thanks for the hint @FelixSchwarz !

Unfortunately, there is another check in helpers/msgpack.py so patching setup.py is probably not enough.

Guess I need to downgrade to v1.11 now and hope I can still access my archives.

[cjk]

Just ran the borg-testsuite against a build using msgpack 1.0.0 and everything looks normal. So I'll probably upgrade my fork and create a custom build for now.
That being said, I'm not a real python-dev and might be doing things the wrong way :wink:

Update: During built, I encountered a PackException, so the upgrade may not be that seamless after all :neutral_face:

ERROR borg/testsuite/archive.py - borg.helpers.msgpack.PackException: __init_...

[FelixSchwarz]

I just briefly checked msgpack 1.0 and borg will need to adapt to some API changes.

The deprecated encoding argument for Packer/Unpacker was removed:

• changelog entry
• commit e419cd8e

Currently borg uses a custom wrapper as these API changes were announced before. Now it seems we should not pass encoding at all. I'm not sure yet if we can do this also for msgpack 0.5.6/0.6.2.

[jdchristensen]

@cjk The fact that borg exits when the msgpack version isn't in the desired range is intentional. Changes in msgpack have very often led to data corruption errors, so you must use a tested msgpack version. Unless you are a borg developer, you shouldn't be running the development version of borg, and you especially shouldn't be running it with an untested msgpack in production. The easiest thing to do is to just download the self-contained binary for the latest stable version and use it, as it includes a known-good msgpack version.

[cjk]

@FelixSchwarz :+1: According to the changelog of msgpack this sounds correct.

In the meantime (+ because of what @jdchristensen said), I installed Borg-stable 1.11.1 and did a borg check on my archives, which resulted in no error, so I'm confident this route will work until the msgpack-stuff is figured out.

Thanks all for your help on this!

[FelixSchwarz]

I tried to remove the "encoding" argument from borg's msgpack calls. However it seems that is not enough (or I screwed up) to get the test suite passing. So probably there is more work to make borg compatible with msgpack 1.0.

[jjgalvez]

can confirm python-msgpack 1.0 breaks the current master. As a workaround I downgraded back to 0.6.2. Hopefully it won't too heavy of a lift to upgrade to 1.0

[ThomasWaldmann]

As there is no PR for this yet, I'll have a look, too...

[ThomasWaldmann]

@cjk guess I fixed the issues with msgpack 1.0.0 in master.

[cjk]

Great news, @ThomasWaldmann - thanks for that!

Have to say, this project has a extremely helpful community and borg-backup is a joy to use (just works, very efficient backups, top-security) :+1:

[FelixSchwarz]

yeah, thank you very much, Thomas. I stopped after it became clear to me that this was not just a few lines in one place and hoped you'd be able to fix this much quicker than me.

One (small?) reason why borg has a good community is that it really integrates well with linux distros. While we can bundle some important libraries it is pretty easy to unbundle. Also it has a good test suite. So I - as a Fedora/EPEL packager - can push new versions quickly without much effort.

[jjgalvez]

Thank you so much for fixing this so quickly!

[jjgalvez]

BTW on Arch it built without any issues. The only error that I ran into was:
Remote: ImportError: libxxhash.so.0: cannot open shared object file: No such file or directory
which required me to install the xxhash package. I don't know which packages missed this as a dependency the borg-git package or the python-msgpack, either way I suspect it's an Arch issue that I will report there just thought I would update this group incase its something that needs to be listed as a dependency because this is a new error.

Thanks again