Borg: blake2 / libb2 crashes with illegal instruction (AVX related)

[thgutmann]

Possible duplicate of Bug#884958: illegal instruction crash

https://www.mail-archive.com/[email protected]/msg1571740.html

libb2 bug: https://github.com/BLAKE2/libb2/issues/13

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.6-1~ubuntu14.04.1~ppa2

Operating system (distribution) and version.

Ubuntu 14.04.5 - Borg and python packages via http://ppa.launchpad.net/costamagnagianfranco/borgbackup/ubuntu

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg init -e keyfile-blake2 user@host://directory/reponame

Describe the problem you're observing.

The above command or even borg -v lead to a crash

~# borg -v
Fatal Python error: Illegal instruction

Current thread 0x00007f11d78ee740 (most recent call first):
  File "/usr/lib/python3/dist-packages/borg/testsuite/crypto.py", line 91 in test_blake2b_256
  File "/usr/lib/python3.4/unittest/case.py", line 577 in run
  File "/usr/lib/python3.4/unittest/case.py", line 625 in __call__
  File "/usr/lib/python3.4/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.4/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.4/unittest/suite.py", line 122 in run
  File "/usr/lib/python3/dist-packages/borg/selftest.py", line 64 in selftest
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4175 in prerun_checks
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4212 in run
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4312 in main
  File "/usr/bin/borg", line 9 in <module>
Illegal instruction

The server is a cloud server on Rackspace, I'll paste /proc/cpuinfo here:

processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 21
model       : 2
model name  : AMD Opteron(tm) Processor 4332 HE
stepping    : 0
microcode   : 0x600081f
cpu MHz     : 3000.120
cache size  : 2048 KB
physical id : 0
siblings    : 1
core id     : 0
cpu cores   : 1
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm rep_good nopl extd_apicid eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes f16c hypervisor lahf_lm cr8_legacy abm sse4a misalignsse 3dnowprefetch xop tbm arat ssbd ls_cfg_ssbd retpoline retpoline_amd vmmcall
bogomips    : 6000.24
TLB size    : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:

processor   : 1
...

On a similar machine (same OS, same version of borg and python libs) it runs just fine. The processor is different:

AMD Opteron(tm) Processor 4170 HE

Any ideas?

Thanks, Theo

[ThomasWaldmann]

testsuite.crypto:91:

        assert blake2b_256(b'', b'abc') == unhexlify('bddd813c634239723171ef3fee98579b94964e3bb1cb3e427262c8c068d52319')

So, this is just some python code, calling some cython wrapper in borg.crypto.low_level, calling some blake2 bundled code or libb2 library function.

Whether it uses the bundled code or the libb2 library depends on the prefer_system_libb2 = True switch in setup.py (showing the default).

[ThomasWaldmann]

Possible duplicate of Bug#884958: illegal instruction crash

It is unclear what you are referring to, please add the respective URL.

[thgutmann]

I am referring to this page:
https://www.mail-archive.com/[email protected]/msg1571740.html

You mentioned setup.py, where exactly is that file? If I get your answer right, I could set prefer_system_libb2 = False to get rid of these errors, right?

[ThomasWaldmann]

Usually python code (*.py) does not crash with that kind of error.
The python interpreter is written in C though, so if it had some strange bug, it could.
Also there could be some weird defect in the Cython / C functions called.

You're saying it works ok on another machine with same OS and libraries and borg. That sounds like you have a hardware issue here. Defect RAM maybe?

[ThomasWaldmann]

As you are using a pre-built .deb package, it is not easy to modify setup.py yourself.

Did you try already?:

apt install --reinstall borgbackup   # just to make sure the borg on-disk files are ok

You could try the borg 1.1.6 fat binary and check if that crashes also on that machine, see there:
https://github.com/borgbackup/borg/releases/tag/1.1.6

[ThomasWaldmann]

The mailing list post points to a problem with an AVX instruction in libb2 library.

[ThomasWaldmann]

Looks like this has been fixed in blake2 (libb2) code in December 2017 and a release v0.98 with that fix was released on Mar 27, 2018.

[ThomasWaldmann]

From the bionic libb2 package log:

libb2 (0.97+git20171226-2) unstable; urgency=medium

  * Team upload (collab-maint)
  * Upload to unstable

 -- Gianfranco Costamagna <[email protected]>  Fri, 29 Dec 2017 21:35:03 +0100

libb2 (0.97+git20171226-1) experimental; urgency=medium

  * Team upload (collab-maint)
  * New upstream snapshot
    - closes: #884958
  * Drop patches, upstream

 -- Gianfranco Costamagna <[email protected]>  Tue, 26 Dec 2017 18:11:32 +0100

[thgutmann]

OK, thanks for the report. Since the fat binary seems to work, I use that until v0.98 is available for my distro.

[ThomasWaldmann]

Hmm, I think the issue is that the libb2 packages for xenial and trusty in @LocutusOfBorg's borgbackup ppa do not have that avx patch yet (compared to the bionic package).

[ThomasWaldmann]

Note: borg packages which use the bundled blake2 (reference implementation) code do not have the avx issue as the reference implementation does not use specific cpu features. Thus, the code in borg does not need to get fixed / updated.

Leaving this issue open for better visibility until the libb2 in the ppa has been fixed.

[ThomasWaldmann]

debian stretch has the broken libb2 version, but also has borgbackup 1.0.9 (which does not support blake2 anyway), so no problem for borg right now, but soon due to #4046.

debian stretch-backports, buster and sid have the fixed libb2 and borg 1.1.x, so no problem there either.

[ThomasWaldmann]

@LocutusOfBorg so, to summarize: ppa needs fixing and also a fix for libb2 in stretch would be very welcome.

[LocutusOfBorg]

ack, opened https://bugs.debian.org/906239

[LocutusOfBorg]

Ppa is ok now

[ThomasWaldmann]

thanks!

seems the main issue (ppa) is fixed, closing.

[ThomasWaldmann]

Update: in debian, it is now in stretch-proposed-updates.