Borg: msgpack(-python), document versions and issues

[ThomasWaldmann]

The 0.5+ releases of msgpack(-python) were a bit of a PITA:

• rename of the package and bad side effects of that (the fix from msgpack project was to make them also available under the old name)
• TypeErrors due to api changes in 0.5.3 (fixed in later versions)
• introducing that FutureWarning, which made borg output somehow unpretty / annoying (the fix from msgpack was to remove the FutureWarning again at some version)
• lack of py34 testing introducing some uncertainty for borg, which still supports py34 (no fix)

Likely there are some working 0.5+ versions if one has py35+ or does own testing on py34, but one needs to check if there are FutureWarnings in borg output.

So the msgpack-python<0.5 is basically a "we know that the given versions work, but when using newer versions you need to do own tests".

There are currently no plans to drop py34 compatibility in borg within the 1.1.x stable release series, so using any 0.5+ version is problematic due to the lack of testing within the msgpack project.

Also, it is not planned to change the name from msgpack-python to msgpack in our requirements within the 1.1.x stable release series, because the required versions are available under the old name and we do not want to break any distribution which uses the old name still.

If borg gets packaged for distribution based on py35 or py36, the lack of py34 testing is no issue, so newer msgpack-python versions can be used, but should be tested first. You can also change the package name to msgpack when making a distribution package, but you need to test if the result actually works.

[LocutusOfBorg]

   * Add debian/python{3,}-msgpack.pydist to help transition to the new upstream
     egg name.

In Debian we are providing the old and new namespace with the same source package, so we don't care anymore about the renaming.

[bpereto]

In recent fedora Version (rawhide, f28) there is no python3-msgpack lower than 0.5.0+.

So, basically I must use 0.5.0+ but don't know if its working?

[ThomasWaldmann]

@bpereto I'll do some tests using the vagrant-based borgbackup tests:

• platforms: linux (debian jessie 64bit), freebsd 64bit, darwin 64bit
• pythons: 3.4.0, 3.5.0, 3.6.0
• msgpacks: msgpack-python and msgpack package (via pip)

At the end, I'll have a look at the changelog also:

https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst

[ThomasWaldmann]

0.4.6...0.4.8: heavily tested with borg, works ok.

a msgpack pure python data corruption was fixed in 0.4.6, so don't use anything less than 0.4.6.

0.4.8 last version with official support for python 3.4.

[ThomasWaldmann]

pip install msgpack-python==0.5.0

broken: installs msgpack-python==0.5.0 AND msgpack==0.5.6 !?

---

pip install msgpack==0.5.0

broken: FutureWarning: use bin_type option is not specified (in msgpack), 20 borg tests broken.

[ThomasWaldmann]

pip install msgpack-python==0.5.1

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

---

pip install msgpack==0.5.1

tox tests linux64: OK

[ThomasWaldmann]

pip install msgpack-python==0.5.2

broken: "TypeError: expected bytes, str found" (in msgpack.Packer / packb)

[ThomasWaldmann]

pip install msgpack-python==0.5.3

broken: no package available on pypi.

---

pip install msgpack==0.5.3

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

[ThomasWaldmann]

pip install msgpack-python==0.5.4

tox tests linux64: OK

---

pip install msgpack==0.5.4

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

This version (at least, unclear whether previous versions are also affected) has a memory leak in the pure python fallback implementation, see #3573 and https://github.com/msgpack/msgpack-python/issues/283.

[ThomasWaldmann]

pip install msgpack-python==0.5.5

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

---

pip install msgpack==0.5.5

tox tests linux64: OK

[ThomasWaldmann]

pip install msgpack-python==0.5.6

tox tests linux64: OK
tox tests freebsd64: OK
tox tests darwin64: OK

---

pip install msgpack==0.5.6

tox tests linux64: OK

[ThomasWaldmann]

OK, looking at the test results from above AND the changelog at https://github.com/msgpack/msgpack-python/blob/master/ChangeLog.rst , I come to this conclusion:

From the 0.5.x series, 0.5.6 looks good, but everything else looks too problematic / broken.

TODO: update requirements: 0.4.6, 0.4.7, 0.4.8, 0.5.6.

[bpereto]

bad news, 0.5.1 is already in epel. no other version available.

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-3b0dd1f4e0

[ThomasWaldmann]

Fixed by #3767.

[ThomasWaldmann]

Just for completeness:

borg 1.1.x does not work with msgpack >= 0.6.0, there are multiple issues (search this issue tracker, if interested).

master branch has quite some changes to support recent msgpack versions (and drops support for old msgpack versions).