Borg: DistributionNotFound error with msgpack-python 0.5.0 installed

[bradst]

Borg fails to launch with msgpack-python 0.5.0, because upstream has renamed the package to msgpack.

$ borg --help
Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3147, in <module>
    @_call_aside
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3131, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3160, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 666, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack-python>=0.4.6' distribution was not found and is required by borgbackup

[KopfKrieg]

I've got the same problem on archlinux. Temporary solution is to rename the requirement to msgpack instead of msgpack-python in requires.txt.

[ThomasWaldmann]

On pypi, there is a transition-package msgpack-python (in addition to the newly named msgpack package).

If there is no better way to deal with it, arch could just package that also.

[ThomasWaldmann]

A slightly dirty workaround is to pip install msgpack-python.

[The-Paavo]

I just had this error too with some of my Ubuntu (16.04, 17.10 and "18.04") and Rasbian machines. Upgrading borg via pip(3) is now installing msgpack 0.5.0 and this breaks borg.

My dirty little workaround was to downgrade msgpack: pip3 install msgpack-python==0.4.8 since pip3 install msgpack-python just gives me this:

Requirement already satisfied: msgpack-python in /usr/local/lib/python3.6/dist-packages
Requirement already satisfied: msgpack>=0.5 in /usr/local/lib/python3.6/dist-packages (from msgpack-python)

Not a perfect solution of course and this will break again if I try to upgrade borgbackup via pip: pip3 install -U borgbackup

[ThomasWaldmann]

@paavoris just some minutes ago, I upgraded borg with pip and it was no problem. There is that transition package msgpack-python on pypi and that pulls the msgpack package, so everything should be happy.

[The-Paavo]

Hmmm... is there something funky going on with my installs? All of them are failing if I upgrade. (Sorry if this is not related to original issue.)

root@helix:~# pip3 install -U borgbackup
Requirement already up-to-date: borgbackup in /usr/local/lib/python3.5/dist-packages
Collecting msgpack-python>=0.4.6 (from borgbackup)
  Using cached msgpack_python-0.5.0-py3-none-any.whl
Requirement already up-to-date: msgpack>=0.5 in /usr/local/lib/python3.5/dist-packages (from msgpack-python>=0.4.6->borgbackup)
Installing collected packages: msgpack-python
  Found existing installation: msgpack-python 0.4.8
    Uninstalling msgpack-python-0.4.8:
      Successfully uninstalled msgpack-python-0.4.8
Successfully installed msgpack-python-0.5.0

root@helix:~# pip3 install msgpack-python
Requirement already satisfied: msgpack-python in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied: msgpack>=0.5 in /usr/local/lib/python3.5/dist-packages (from msgpack-python)

root@helix:~# borg -V
Traceback (most recent call last):
  File "/usr/local/bin/borg", line 7, in <module>
    from borg.archiver import main
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 32, in <module>
    import msgpack
ImportError: No module named 'msgpack'

root@helix:~# pip3 install msgpack-python==0.4.8
Collecting msgpack-python==0.4.8
Installing collected packages: msgpack-python
  Found existing installation: msgpack-python 0.5.0
    Uninstalling msgpack-python-0.5.0:
      Successfully uninstalled msgpack-python-0.5.0
Successfully installed msgpack-python-0.4.8

root@helix:~# borg -V
borg 1.1.4

root@helix:~# python3 -V && pip3 -V
Python 3.5.2
pip 9.0.1 from /usr/local/lib/python3.5/dist-packages (python 3.5)

(I personally can't code even "hello world" with python, so I'm no python expert. I may have messed up something. 🤓 )

[eli-schwartz]

On pypi, there is a transition-package msgpack-python (in addition to the newly named msgpack package).

If there is no better way to deal with it, arch could just package that also.

We could, and probably should as borg is not the only package users are having issues with. That being said, transitional dummy packages are there for transitional purposes... so it's probably best to migrate to the new module name sooner rather than later.

Anyway, yeah, this issue should most likely only affect distro users.

[eli-schwartz]

This should be fixed for Arch Linux systems, as we now install the transitional dummy package alongside the main package: https://git.archlinux.org/svntogit/community.git/commit/trunk?h=packages/python-msgpack&id=d366653e36f3fbe10f689097de3ca2317118efd1

As new installations of newer versions of borg should be installed with then-current versions of msgpack, it makes sense to just fully move over to depending on msgpack now, instead of python-msgpack -- which will fix this for the future (when we potentially drop the dummy package) as well as for other distros that may not have (yet?) done this.

[0x022b]

This also happens on Ubuntu running on WSL.

$ sudo pip install --upgrade borgbackup
Collecting borgbackup
  Downloading borgbackup-1.1.4.tar.gz (3.4MB)
    100% |████████████████████████████████| 3.4MB 480kB/s
Collecting msgpack-python>=0.4.6 (from borgbackup)
  Downloading msgpack_python-0.5.0-py3-none-any.whl
Collecting msgpack>=0.5 (from msgpack-python>=0.4.6->borgbackup)
  Downloading msgpack-0.5.0.tar.gz (142kB)
    100% |████████████████████████████████| 143kB 5.0MB/s
Installing collected packages: msgpack, msgpack-python, borgbackup
  Running setup.py install for msgpack ... done
  Found existing installation: msgpack-python 0.4.8
    Uninstalling msgpack-python-0.4.8:
      Successfully uninstalled msgpack-python-0.4.8
  Found existing installation: borgbackup 1.1.3
    Uninstalling borgbackup-1.1.3:
      Successfully uninstalled borgbackup-1.1.3
  Running setup.py install for borgbackup ... done
Successfully installed borgbackup-1.1.4 msgpack-0.5.0 msgpack-python-0.5.0
ModuleNotFoundError: No module named 'msgpack'

$ borg --version
Traceback (most recent call last):
  File "/usr/local/bin/borg", line 9, in <module>
    load_entry_point('borgbackup==1.1.4', 'console_scripts', 'borg')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 32, in <module>
    import msgpack
ImportError: No module named 'msgpack'

Workaround is to downgrade msgpack-python.

sudo pip install msgpack-python==0.4.8

[ThomasWaldmann]

The issue was fixed in msgpack 0.5.1 (see link directly above this post).

As a side note: py34 testing (and thus official support for py34) was removed in msgpack 0.5.0.

As borg 1.0 and 1.1 still support py34 and only master branch requires py35+, we will adjust requirements:

• 1.0 and 1.1: msgpack-python>=0.4.6,<0.5.0
• master: msgpack-python>=0.4.6,!=0.5.0,<0.6.0

The <0.6.0 requirement for master is mostly a precaution against sudden incompatible api changes before we have tested them and will be increased after we have successfully tested new msgpack releases.

[quicktrick]

I have just the same problem as in the first message of this thread when trying to connect to a repo remotely via ssh. If I connect to the same repo locally from the server it's located on, everything is Ok. The thing is that the server containing the repo is Qnap where I installed Borg from the source code. The other server is openSUSE, and I installed Borg as a package there. At first everything worked fine, but after some attempts to install Fuse on Qnap I got this problem. I tried to install msgpack-python==0.4.8 and just msgpack, but it doesn't solve the problem. Borg works fine locally, but remotely it gives the errors just like in the first message of this thread.

[ThomasWaldmann]

@quicktrick maybe try the following:

• check differences between local environment and ssh environment (PATH, PYTHONPATH, ...)
• uninstall both msgpack and msgpack-python, then reinstall msgpack-python==0.4.8.

[quicktrick]

Thank you for your prompt reply, Thomas! I tried everything you suggested (to be honest I hadn't had PYTHONPATH variable before that, and I set it; environment is the same locally and via ssh), but the result is the same. Everything is Ok locally, and the same errors remotely.

[quicktrick]

Thomas, I found out that the problem is not in Qnap, but in openSUSE. I have openSUSE Tumbleweed, and yesterday I updated the system. And now Borg doesn't work there (it shows those errors). I tried to remove it with the dependencies and reinstall -- the problem didn't disappear.

(1/5) Installing: python3-llfuse-1.3.2-1.3.x86_64 .....................................................................................................................................[done]
(2/5) Installing: python3-msgpack-0.5.4-1.1.x86_64 ....................................................................................................................................[done]
(3/5) Installing: borgbackup-1.1.4-2.1.x86_64 .........................................................................................................................................[done]
(4/5) Installing: borgbackup-zsh-completion-1.1.4-2.1.noarch ..........................................................................................................................[done]
(5/5) Installing: borgbackup-bash-completion-1.1.4-2.1.noarch .........................................................................................................................[done]

Edit:
I've solved it.

(1/2) Removing python3-msgpack-0.5.4-1.1.x86_64 .......................................................................................................................................[done]
(2/2) Installing: python3-msgpack-python-0.4.8-2.4.x86_64 .............................................................................................................................[done]

And it works now. Locally and remotely on both servers. Sorry for bothering you. And thanks a lot for your help!

[ThomasWaldmann]

OK, it works now, but what was the root cause for the breakage?

Is python3-msgpack-0.5.4-1.1.x86_64 broken?

[quicktrick]

Thomas, as I can see here and especially here, last build for my OS (openSUSE Tumbleweed) failed, and not because of python3-msgpack-0.5.4-1.1.x86_64:

[   35s] + python3 setup.py build
[   35s] Traceback (most recent call last):
[   35s]   File "setup.py", line 172, in <module>
[   35s]     liblz4_prefix = setup_lz4.lz4_system_prefix(possible_liblz4_prefixes)
[   35s]   File "/home/abuild/rpmbuild/BUILD/borgbackup-1.1.4/setup_lz4.py", line 28, in lz4_system_prefix
[   35s]     if 'LZ4_compress_default' in fd.read():  # requires lz4 >= 1.7.0 (r129)
[   35s]   File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
[   35s]     return codecs.ascii_decode(input, self.errors)[0]
[   35s] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 13333: ordinal not in range(128)
[   35s] error: Bad exit status from /var/tmp/rpm-tmp.mWQehw (%build)
[   35s] 
[   35s] 
[   35s] RPM build errors:
[   35s]     Bad exit status from /var/tmp/rpm-tmp.mWQehw (%build)
[   35s] 
[   35s] build71 failed "build borgbackup.spec" at Tue Feb 27 10:32:08 UTC 2018.

Though the package is available in the openSUSE Tumbleweed Main Repository:

# zypper if --provides --requires borgbackup
Loading repository data...
Reading installed packages...

Information for package borgbackup:
-----------------------------------
Repository     : Main Repository (OSS)
Name           : borgbackup
Version        : 1.1.4-2.1
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 10.9 MiB
Installed      : Yes
Status         : up-to-date
Source package : borgbackup-1.1.4-2.1.src
Summary        : Deduplicating backup program with compression and authenticated encryption
Description    :
    BorgBackup is a deduplicating backup program which stores deltas. It
    supports compression and authenticated encryption as well,
    facilitating frequent backups and storing to not fully trusted
    targets.

    Run borg for a commands overview and check out the docs at
    /usr/share/doc/packages/borgbackup/html/index.html.
Provides       : [2]
    borgbackup = 1.1.4-2.1
    borgbackup(x86-64) = 1.1.4-2.1
Requires       : [22]
    libc.so.6()(64bit)
    libc.so.6(GLIBC_2.2.5)(64bit)
    libc.so.6(GLIBC_2.4)(64bit)
    libc.so.6(GLIBC_2.14)(64bit)
    libc.so.6(GLIBC_2.3.4)(64bit)
    libpthread.so.0()(64bit)
    libpthread.so.0(GLIBC_2.2.5)(64bit)
    /usr/bin/python3
    libcrypto.so.1.1()(64bit)
    libcrypto.so.1.1(OPENSSL_1_1_0)(64bit)
    libpython3.6m.so.1.0()(64bit)
    libc.so.6(GLIBC_2.6)(64bit)
    libacl.so.1()(64bit)
    libacl.so.1(ACL_1.0)(64bit)
    liblz4.so.1()(64bit)
    libzstd.so.1()(64bit)
    rpmlib(CompressedFileNames) <= 3.0.4-1
    rpmlib(PayloadFilesHavePrefix) <= 4.0-1
    rpmlib(PayloadIsXz) <= 5.2-1
    rpmlib(FileDigests) <= 4.6.0-1
    python(abi) = 3.6
    python3-msgpack-python >= 0.4.6

[ThomasWaldmann]

@quicktrick sounds like you want to discuss that with the opensuse borg maintainers.

Here's the fix for the non-ascii header file / ascii build env issue: https://github.com/borgbackup/borg/pull/3580/files (alternatively, one can just use a utf-8 capable locale, which is quite the norm nowadays).