Openblas: Calculating eigenvalues crashes R and Julia

[danmrc]

Hello,

I tried to calculate eigenvalues in both Julia and R, and it throws an error. Even small matrices, like 10 x 10 and symmetric, throws an error. Here is an example in R:

aa <- matrix(rnorm(50*10),ncol=10)
 B <- t(aa)%*%aa
eigen(B)

And the core dump:

PID: 3209 (R)
           UID: 1000 (danielc)
           GID: 1000 (danielc)
        Signal: 4 (ILL)
     Timestamp: Thu 2020-12-17 17:48:18 -03 (7min ago)
  Command Line: /usr/lib64/R/bin/exec/R
    Executable: /usr/lib/R/bin/exec/R
 Control Group: /user.slice/user-1000.slice/session-2.scope
          Unit: session-2.scope
         Slice: user-1000.slice
       Session: 2
     Owner UID: 1000 (danielc)
      Hostname: daniel-vostro3560
       Storage: /var/lib/systemd/coredump/core.R.1000.5a08490df9c74153aa26c65ab6554092.3209.1608238>
       Message: Process 3209 (R) of user 1000 dumped core.

                Stack trace of thread 3209:
                #0  0x00007f0f5395cf76 dsymv_thread_L (libblas.so.3 + 0xccf76)
                #1  0x00007f0f5392f8e7 dsymv_ (libblas.so.3 + 0x9f8e7)
                #2  0x00007f0f2fd05b7f dsytd2_ (liblapack.so.3 + 0x237b7f)
                #3  0x00007f0f2fd0720c dsytrd_ (liblapack.so.3 + 0x23920c)
                #4  0x00007f0f2fcffcd1 dsyevr_ (liblapack.so.3 + 0x231cd1)
                #5  0x00007f0f54d85b50 n/a (lapack.so + 0x3b50)
                #6  0x00007f0f54ecfa26 n/a (libR.so + 0x136a26)
                #7  0x00007f0f54ee9a80 Rf_eval (libR.so + 0x150a80)
                #8  0x00007f0f54eeb7b7 n/a (libR.so + 0x1527b7)
                #9  0x00007f0f54eec614 Rf_applyClosure (libR.so + 0x153614)
                #10 0x00007f0f54ee9cb1 Rf_eval (libR.so + 0x150cb1)
                #11 0x00007f0f54f1e046 Rf_ReplIteration (libR.so + 0x185046)
                #12 0x00007f0f54f1e3e1 n/a (libR.so + 0x1853e1)
                #13 0x00007f0f54f1e499 run_Rmainloop (libR.so + 0x185499)
                #14 0x000055c0bb8ea03d main (R + 0x103d)
                #15 0x00007f0f54bc2152 __libc_start_main (libc.so.6 + 0x28152)
                #16 0x000055c0bb8ea07e _start (R + 0x107e)

I moved from OpenBLAS to BLAS and this solved the error, so I am guessing the issue is with OpenBLAS? I was using OpenBLAS 0.3.13 and the LAPACK version is 3.9.0-3, in EndeavourOS (arch based).

I will be glad to provide any other details if you need them.

Thanks in advance

[BPMinaker]

Can confirm on Arch Linux using OpenBLAS 0.3.13 crashes Julia on SVD calculation, revert to 0.3.12 fixes the issue. Error below:

Invalid instruction at 0x7f01e97399da: 0xc4, 0xe2, 0xe9, 0x9b, 0xc1, 0xc5, 0xf9, 0x2f, 0xc4, 0x77, 0x43, 0x48, 0x83, 0xf8, 0x10

signal (4): Illegal instruction
in expression starting at /home/....
dtrmv_thread_NUN at /usr/bin/../lib/libblas.so (unknown line)

[Diazonium]

Crashed with SIGILL? Thats strange. Is your OpenBLAS from an Arch repo, or have you built it yourself?
My first guess is that its trying to use an instruction your CPU does not support.

[BPMinaker]

Using the standard Arch repo, on an Intel i7. Maybe something in the Arch build process?

[Diazonium]

I assume the Arch build is using dynamic (i.e. runtime) CPU detection. A regression in the dynamic CPU detection is a possible explanation, although I would not expect anything but the AVX512 instructions to fail on an i7 CPU.
Does your CPU support AVX512, by any chance?

[BPMinaker]

My i7 is a 2600, which is circa 2011, so I think too old to support AVX (I think implemented in 2017).

[danmrc]

Crashed with SIGILL? Thats strange. Is your OpenBLAS from an Arch repo, or have you built it yourself?
My first guess is that its trying to use an instruction your CPU does not support.

My processor is an i5 from 2012, and I also downloaded the version from the arch repo.

I looked around and here it says it supports AVX. However I am not sure about the quality of the information

EDIT:

Can confirm on Arch Linux using OpenBLAS 0.3.13 crashes Julia on SVD calculation, revert to 0.3.12 fixes the issue. Error below:

That is funny, I can run SVD() without trouble but not eigen()

[Diazonium]

Ok, both of those are nowhere near AVX512 support. I am not that familiar with the codebase, so unfortunately I cannot provide much more insight regarding your issue.
If you feel like it, you could build OpenBLAS 0.3.13 from source for SANDYBRIDGE target, and replace the build from the repo with your own. If this works, its probably an issue with the runtime CPU detection. (and you have a viable workaround until it gets fixed)

[martin-frbg]

Does it work when you export OPENBLAS_CORETYPE=SANDYBRIDGE before running R ? I do not think the DYNAMIC_ARCH code itself changed, but the increased use of compiler intrinsics instead of assembler instructions has made it necessary to add compiler options like -mavx -mfma, and it could be that the wrong selection of those got applied to common code outside the cpu-specific BLAS kernels.
In particular, if https://github.com/archlinux/svntogit-community/blob/packages/openblas/trunk/PKGBUILD is the current build file then Arch does not set a TARGET for the common code in conjunction with DYNAMIC_ARCH, which will result in a library that may not run on any older generation cpu than the one used for its compilation.

[maximtrp]

@martin-frbg Setting OPENBLAS_CORETYPE variable does not work for me (Celeron 1007U, IvyBridge). I have tried to rebuild the package with TARGET set to SANDYBRIDGE. It does not help.

[martin-frbg]

Thanks. According to the information on ark.intel.com, the Celeron1007U does not have AVX (unlike regular Sandybridge) so perhaps TARGET=NEHALEM would work.

[danmrc]

Does it work when you export OPENBLAS_CORETYPE=SANDYBRIDGE before running R ?

No, it throws the same error (in both Julia and R). I tried the NEHALEM option too, with no success

[narpfel]

I’m also affected by this (openblas 0.3.13 from the Arch repos, Intel Core i5-760).

For me it crashes on a vmovdqa instruction in openblas_get_config():

(gdb) disassemble
Dump of assembler code for function openblas_get_config:
   0x00007ffff6f3b4a0 <+0>:     push   %rbp
   0x00007ffff6f3b4a1 <+1>:     sub    $0x20,%rsp
=> 0x00007ffff6f3b4a5 <+5>:     vmovdqa 0xf91c53(%rip),%xmm0        # 0x7ffff7ecd100
   0x00007ffff6f3b4ad <+13>:    mov    %fs:0x28,%rax
   0x00007ffff6f3b4b6 <+22>:    mov    %rax,0x18(%rsp)
   0x00007ffff6f3b4bb <+27>:    xor    %eax,%eax
   0x00007ffff6f3b4bd <+29>:    movabs $0x504d4e45504f5f45,%rax
[snip]

Minimal reproducer:

#include <stdio.h>

char* openblas_get_config(void);

int main(void) {
    puts(openblas_get_config());
}

Compile with

$ gcc -std=c18 -Wall -Wextra -pedantic -lopenblas -o openblas_sigill openblas_sigill.c

and

$ ./openblas_sigill
[1]    4774 illegal hardware instruction (core dumped)  ./openblas_sigill

Interestingly, rebuilding the package locally fixes it for me. (@martin-frbg The PKGBUILD you linked is indeed the current build file.) The R example also works for me with my local build.

[brada4]

When ARCH package site recovers we might be able to see the build log (last 10 lines tells minimal CPU for dynamic build)
R debuginfo would be handy to resolve the internal fortran function it calls (and maybe lapack too)

[brada4]

https://github.com/archlinux/svntogit-community/blob/cda9ed2a05fe5f150b82c13b8a44d8b284b140f2/trunk/PKGBUILD#L19
You need to add TARGET=CORE2 to DYNAMIC_ARCH build so that common code does not need AVX or any ISA inherited from build machine.

[danmrc]

Actually rebuilding the package locally fixes the issue, as @narpfel suggested. I didn't change anything and it was simpler than I thought.

I will leave it opened until someone else that commented on the topic and has the same issue confirm, since I am not using pure arch

If you have no idea of how to start (I didn't) start here: https://wiki.archlinux.org/index.php/Arch_Build_System

[brada4]

Please report that to Arch. It is their oversight.

[maximtrp]

@brada4 Thank you. I have added a task here: https://bugs.archlinux.org/task/69032

[brada4]

You miss serious detail in "task" - it is about PACKAGING, not the upstream product. It is quite surprising it ever worked as OpenBLAS compiles whole fortran LAPACK part tailored to build farm CPU.

[BPMinaker]

Just following up, I can confirm a local rebuild, no edits, just download PKGBUILD and makepkg resolved the issue for me as well.

[Diazonium]

That is probably because this is likely the result of the compile-time CPU detection "leaking into" the runtime CPU detection. The problem is probably only happening, if the CPU on the build machine supports instruction set extensions (e.g. later versions of AVX), but then someone tries to use that binary package on a CPU that does not support that instruction.

If you do a local rebuild, the compile-time CPU detection will ensure that no isns will be used that are unsupported by your CPU. But you might as well do a non DYNAMIC_ARCH build, if you are rebuilding locally.

[brada4]

They have to force target to smething that would be able to run everywhere.

[narpfel]

I can confirm that building with TARGET=GENERIC on a Haswell CPU makes it work on a Nehalem CPU.

This is also how Debian builds their openblas package, see https://sources.debian.org/src/openblas/0.3.13+ds-1/debian/rules/#L23.

[brada4]

That should work with recent versions. Would be nice to get some build log to see if it propagated right to all places needed.

[narpfel]

@brada4: I ran makepkg with the default PKGBUILD for the first log; for the second log I added TARGET=GENERIC to the make call just before libs shared:

--- PKGBUILD    2020-12-20 20:18:48.782374501 +0100
+++ PKGBUILD_target     2020-12-20 19:59:32.820185252 +0100
@@ -21,7 +21,7 @@

   make NO_STATIC=1 NO_LAPACK=1 NO_LAPACKE=1 NO_CBLAS=1 NO_AFFINITY=1 USE_OPENMP=1 \
        CFLAGS="$CPPFLAGS $CFLAGS" DYNAMIC_ARCH=1 \
-       NUM_THREADS=64 MAJOR_VERSION=3 libs shared
+       NUM_THREADS=64 MAJOR_VERSION=3 TARGET=GENERIC libs shared
 }

 package() {

openblas_build_no_target.txt
openblas_build_target_generic.txt

Build CPU was an Intel Core i5-8265U (supports AVX and AVX2, but not AVX512).

[brada4]

It is in every build line in "no_target" build
-msse3 -mssse3 -msse4.1 -mavx -mavx2 -mfma -mavx2
If you look in other build - sse3 and avx just appear on microarchitecture-specific kernel code.

I think it is good to go after build flag change.

I would speculate that some subtle development in gcc finally emitted AVX instruction in casual code that so far went through on those old computers.

sse2 is genetic to amd64/x86_64 (CORE2, or compiler defaults w GENERIC) , earliest AMD 64bit CPUs have 3dnow, which will not be good for common code as even AMD dropped that in later CPUs.

[brada4]

For diagnostics you could drop libs shared part so that build runs tests and emits detected parameters, like "common code is valid on haswell and up" etc.
It is not your mistake, it is just to better protect from mistakes made here.

[BPMinaker]

Can confirm that the update posted on Arch today has resolved the issue on my apparently outdated i7.

[danmrc]

Just updated to openblas 0.3.13-2 and had no problems at all

Thanks for the help for everyone that commented here