Borg: test fails with pure-Python msgpack (borg 1.1.x)

[FelixSchwarz]

ArchiverTestCase.test_compression_auto_compressible fails for me when using a pure Python msgpack.

borg 1.1.9 (also git version from "1.1-maint" branch)
Fedora 29 x86_64
msgpack 0.5.6
python3-pytest-3.6.4-1.fc29.noarch

python3 setup.py build
export MSGPACK_PUREPYTHON=1
export PYTHONPATH=$(pwd)/build/$(ls build/ | grep 'lib.')
py.test-3 -vx -k "ArchiverTestCase and test_compression"  $PYTHONPATH/borg/testsuite/*.py

    _________ ArchiverTestCase.test_compression_auto_compressible _________

    self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_compression_auto_compressible>

        def test_compression_auto_compressible(self):
            size, csize = self._get_sizes('auto,zlib', compressible=True)
    >       assert csize < size * 0.1
    E       AssertionError: assert 0 < (0 * 0.1)

    build/lib.linux-x86_64-3.7/borg/testsuite/archiver.py:2077: AssertionError

unset MSGPACK_PUREPYTHON
py.test-3 -vx -k "ArchiverTestCase and test_compression"  $PYTHONPATH/borg/testsuite/*.py
(test passes)

I suspect this might be caused by the slow message warning.

[ThomasWaldmann]

The output of the borg command had warnings about the slow msgpack and that test couldn't cope with that.

I've fixed it, PR coming.

[ThomasWaldmann]

1.1-maint fixed by #4558.

[katodev]

An other issue is with msgpack 0.6.1.
I can mount the repository and see all backups, but I can't enter into an archive. msgpack 0.5.4 works well.

[infectormp]

@katodev msgpack 0.6.1 is unsupported by borg

[0xSheepdog]

@katodev msgpack 0.6.1 is unsupported by borg

I confirm, Fedora 30 install includes python-msgpack-0.6.1 and borg commands fail

> 
> #  borg                                                                                      
> Traceback (most recent call last):                                                                    
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master       
>     ws.require(__requires__)                                                                          
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require             
>     needed = self.resolve(parse_requirements(requirements))                                           
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve             
>     raise VersionConflict(dist, req).with_context(dependent_req)                                      
> pkg_resources.ContextualVersionConflict: (msgpack 0.6.1 (/usr/lib64/python3.7/site-packages), Requirem
> ent.parse('msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.6,>=0.4.6'), {'borgbackup'})  
>                                                                                                       
> During handling of the above exception, another exception occurred:                                   
>                                                                                                       
> Traceback (most recent call last):                                                                    
>   File "/usr/bin/borg", line 6, in <module>                                                           
>     from pkg_resources import load_entry_point                                                        
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>           
>     @_call_aside                                                                                      
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside        
>     f(*args, **kwargs)                                                                                
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_
> working_set                                                                                           
>     working_set = WorkingSet._build_master()                                                          
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master       
>     return cls._build_from_requirements(__requires__)                                                 
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirem
> ents                                                                                                  
>     dists = ws.resolve(reqs, Environment())                                                           
>   File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve             
>     raise DistributionNotFound(req, requirers)                                                        
> pkg_resources.DistributionNotFound: The 'msgpack!=0.5.0,!=0.5.1,!=0.5.2,!=0.5.3,!=0.5.4,!=0.5.5,<=0.5.
> 6,>=0.4.6' distribution was not found and is required by borgbackup                                   

[FelixSchwarz]

This is a Fedora problem. I hope I can fix this soon (but my time is limited). Please read https://bugzilla.redhat.com/show_bug.cgi?id=1669083 for a more complete status report.

[ThomasWaldmann]

Please note that the issues of @0xSheepdog and @katodev have nothing to do with the topic (see top post) of this issue.

[cjk]

I'm hitting the same issue as @0xSheepdog and @FelixSchwarz discuss after upgrading Arch-Linux (Linux cjk-tux 5.5.13-arch2-1 #1 SMP PREEMPT Mon, 30 Mar 2020 20:42:41 +0000 x86_64 GNU/Linux) today.
Shall I open a separate issue here or report it on the Bugzilla-thread mentioned above?

Here is the error:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (msgpack 1.0.0 (/usr/lib/python3.8/site-packages), Requirement.parse('msgpack<=0.6.2,>=0.5.6'), {'borgbackup'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 586, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'msgpack<=0.6.2,>=0.5.6' distribution was not found and is required by borgbackup

[cjk]

More info - it seems like msgpack was updated in a recent update on Arch:

[...]
[2020-03-31T00:04:26+0200] [ALPM] upgraded python-msgpack (0.6.2-3 -> 1.0.0-1)
[...]

[ThomasWaldmann]

We have a builtin known-good version of msgpack.

https://github.com/borgbackup/borg/blob/1.1.11/src/borg/helpers.py#L34

Note: you should be using borgbackup 1.1.11.

[FelixSchwarz]

@cjk The bugzilla issue was about borgbackup 1.1 and msgpack 0.6 in Fedora. It has been fixed for many months now so adding your comment is not helping much.

However I'd be interested in a separate (borgbackup) issue if python-msgpack 1.0 does not work with borgbackup master. python-msgpack 1.0 will be added to Fedora 33. Currently Fedora uses the bundled msgpack but I hope to remove that after updating to borgbackup 1.2.

[cjk]

@ThomasWaldmann
Thanks for the pointer. Perhaps it would make sense to adjust the Arch-PKGBUILD, but it looks OK to me by the Arch-standards (no python-export though).

Note: you should be using borgbackup 1.1.11.

Yes I know, consider me an alpha-tester :) The latest version just has too many advantages for me and I didn't have any issues so far.
I even had issues with the current stable version so I upgraded to latest and everything was good. Not patient enough to wait months for a new release I guess.

@FelixSchwarz Thanks for your comment, I did as you suggested and opened #5065 for this.