borg does not work on unsupported msgpack version

[quicktrick]

borg 1.1.9, Python 3.7.2
openSUSE Tumbleweed, fresh installation:

Linux mail 5.0.2-1-default #1 SMP Thu Mar 14 08:29:17 UTC 2019 (d1f1d19) x86_64 x86_64 x86_64 GNU/Linux

The backups are mountable, I can see the archives list at the mount point, I can enter an archive directory, but after that:

mail:/mnt/tmp # ls -la
total 0
drwxr-xr-x 1 root root 0 Mar 21 20:12 .
drwxr-xr-x 1 root root 6 Mar 21 19:23 ..
drwxr-xr-x 1 root root 0 Sep 30 02:10 mail-2018-09-30T02:10:03
drwxr-xr-x 1 root root 0 Oct 31 02:10 mail-2018-10-31T02:10:04
......
drwxr-xr-x 1 root root 0 Mar 13 02:10 mail-2019-03-13T02:10:08
drwxr-xr-x 1 root root 0 Mar 14 02:10 mail-2019-03-14T02:10:05

mail:/mnt/tmp # cd mail-2019-03-14T02:10:05
mail:/mnt/tmp/mail-2019-03-14T02:10:05 # ls -la
ls: cannot open directory '.': Input/output error

mail:/mnt/tmp/mail-2019-03-14T02:10:05 # cd ..
mail:/mnt/tmp # ls -la
ls: cannot open directory '.': Transport endpoint is not connected

mail:/mnt/tmp # borg umount /mnt/tmp
fusermount: failed to unmount /mnt/tmp: Device or resource busy
mail:/mnt/tmp # cd ..
mail:/mnt # borg umount /mnt/tmp
mail:/mnt #

So, I can unmount the repo and then I can mount it again, but the problem persists.
And, apart of fuse, I can list and extract the archives, so only fuse isn't working.

The log does not show anything about that error, only:

Mar 21 20:12:20 mail kernel: fuse init (API version 7.28)
Mar 21 20:12:20 mail systemd[1]: Mounting FUSE Control File System...
Mar 21 20:12:20 mail systemd[1]: Mounted FUSE Control File System.
Mar 21 20:15:53 mail systemd[1]: mnt-tmp.mount: Succeeded.
Mar 21 20:15:53 mail systemd[1989]: mnt-tmp.mount: Succeeded.

I have another Tumbleweed server with some older kernel:

Linux voyo 4.20.6-1-default #1 SMP PREEMPT Thu Jan 31 07:37:50 UTC 2019 (463cfd2) x86_64 x86_64 x86_64 GNU/Linux

Everything other is the same: borg 1.1.9, Python 3.7.2
And everything is working fine there.

So, I don't think it's a Borg bug, rather it's the kernel bug or, perhaps, some incompatibility of fuse with the new kernel.

[ThomasWaldmann]

Try borg mount -f ... to have the mount process stay in the foreground.

Then, in another terminal, repeat your experiment and see later what happened with the mount process - maybe it crashed and emitted some traceback? if so, post it here.

[quicktrick]

Hi, Thomas! Thank you for your reply!

When I got the I/O error in another terminal

mail:/mnt/tmp/mail-2019-03-14T02:10:05 # ls -la
ls: cannot open directory '.': Input/output error

in the first terminal I got

mail:~ # borg mount -f ssh://... /mnt/tmp
Enter passphrase for key /root/.config/borg/keys/...:
Local Exception
Traceback (most recent call last):
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4455, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 4387, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 1355, in do_mount
    return self._do_mount(args)
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 154, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.7/site-packages/borg/archiver.py", line 1365, in _do_mount
    operations.mount(args.mountpoint, args.options, args.foreground)
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 351, in mount
    signal = fuse_main()
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 34, in fuse_main
    return llfuse.main(workers=1)
  File "src/fuse_api.pxi", line 330, in llfuse.main
  File "src/handlers.pxi", line 436, in llfuse.fuse_opendir
  File "src/handlers.pxi", line 437, in llfuse.fuse_opendir
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 592, in opendir
    self._load_pending_archive(inode)
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 565, in _load_pending_archive
    self.process_archive(archive_name, [os.fsencode(archive_name)])
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 388, in process_archive
    consider_part_files=self.args.consider_part_files):
  File "/usr/lib64/python3.7/site-packages/borg/fuse.py", line 160, in iter_archive_items
    item = unpacker.unpack(write_bytes)
TypeError: unpack() takes no arguments (1 given)

Platform: Linux mail 5.0.2-1-default #1 SMP Thu Mar 14 08:29:17 UTC 2019 (d1f1d19) x86_64
Linux:
Borg: 1.1.9  Python: CPython 3.7.2
PID: 3926  CWD: /root
sys.argv: ['/usr/bin/borg', 'mount', '-f', 'ssh://...', '/mnt/tmp']
SSH_ORIGINAL_COMMAND: None

mail:~ # 

though the mount still exists

mail:~ # mount
.....
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
borgfs on /mnt/tmp type fuse (ro,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions)

And after borg umount ...

mail:~ # mount
.....
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

FYI, I tried everything described here on two fresh Tumbleweed servers, installed from different snapshots with a week time difference.

[infectormp]

https://github.com/borgbackup/borg/issues/4245 ?

[quicktrick]

@infectormp , you are right, python3-msgpack-0.6.1-1.1. I'll try to find 0.5.6. Thanks a lot!

[infectormp]

@ThomasWaldmann can you add here -> Borg: 1.1.9 Python: CPython 3.7.2 info aubout msgpack version?

[quicktrick]

I've found 0.5.6, but

# zypper in --oldpackage python3-msgpack-0.5.6-16.1.x86_64.rpm
Problem: nothing provides libpython3.6m.so.1.0()(64bit) needed by python3-msgpack-0.5.6-16.1.x86_64
 Solution 1: do not install python3-msgpack-0.5.6-16.1.x86_64
 Solution 2: break python3-msgpack-0.5.6-16.1.x86_64 by ignoring some of its dependencies

 # zypper se libpython3
i | libpython3_7m1_0       | Python Interpreter shared library | package

I guess, there is no working solution. Though I can try to make a softlink libpython3.6m.so.1.0 to libpython3.7.

Edit:
That "solution" with the softlink does not work at all.

Edit 2:
I looked into my other server with borg 1.1.9 and python3-msgpack-0.5.6 and found that there is libpython3.7m.so.1.0 too (not libpython3.6m.so.1.0) and it's working there. I don't know where to find a working version of python3-msgpack-0.5.6.

I would not like to install Borgbackup from source or using pip. I hope, the problem with the newer python3-msgpack will be solved in the future.

[infectormp]

@quicktrick you can use compiled borg binary from https://github.com/borgbackup/borg/releases/tag/1.1.9

[ThomasWaldmann]

Please check if you have these code lines (and if you don't have them, check why you don't):

• https://github.com/borgbackup/borg/blob/1.1.9/src/borg/archiver.py#L4351
• https://github.com/borgbackup/borg/blob/1.1.9/src/borg/helpers.py#L1293
• https://github.com/borgbackup/borg/blob/1.1.9/setup.py#L42

[quicktrick]

@infectormp , thank you very much, borg-linux64 works fine with fuse!

@ThomasWaldmann , I checked those files in the package, they don't have those lines indeed. And the code differs much from that in your links. Perhaps, I see in some wrong place (/usr/lib64/python3.7/site-packages/borg/).
borgbackup-1.1.9-2.1.x86_64.rpm.zip

[ThomasWaldmann]

@infectormp added msgpack version info to sysinfo, thanks for the suggestion.

@quicktrick you have to talk to package maintainer about this, see INFO/CHANGELOG in the rpm.

[ThomasWaldmann]

That guy f*cked with the version check code line even though there is a big fat "DO NOT CHANGE OR REMOVE!" right above it.

[frispete]

That guy was me. Sorry guys.

We had the choice of reverting msgpack 0.6.1 to 0.5.6, but then I checked with this project to see how future borgbackup versions will handle this, found, it supports 0.6.1, and f*cked with the version check code than. Since I'm using borg every day, I felt living in a safe place, but haven't used borg mount for a while.

I plead guilty as charged, and most subservient request for a light penalty.

The problem here is, that we try to provide the most current versions, while keep that package working with older distributions as well. The msgpack mess causes biggest headaches already...

@ThomasWaldmann when do you plan to release the next version, and will backups from that version be compatible with former versions?

@quicktrick thanks for bringing this up. Will look into this issue in the next couple of days and revise our options again!

[ThomasWaldmann]

@frispete maybe you checked against wrong branch:

• master branch does support newer msgpack (using quite a lot of changes, adapting to api and even wrapping the whole msgpack code into a wrapper module),
• but 1.1-maint branch does not (== all 1.1.x releases).

1.2.x releases (when master has gone through alpha, beta, rc testing and there is a first release) will support newer msgpacks (but not older). Currently we are at 1.2.0 alpha 5, so a final release will still take some months.

1.2 will be compatible to repos / archives made with 1.1, but not necessarily vice versa.

[frispete]

@ThomasWaldmann thanks for clarification. Much appreciated.

Looks, like I need to build a msgpack5 module with version 0.5.6 specifically for borg then... no big thing, but Tumbleweed as well as the top level python project maintainers hate this, since this will conflict with the 0.6.1 version, but I don't see any other good options without a big patch orgy, which will not pay off, as we can clearly see here now.