Openblas: Error: Detection of CPU type fails

[1kastner]

The Problem

The np.dot operation leads to a crash of my python interpreter:

import numpy as np
A = np.matrix([[1.], [3.]])
B = np.matrix([[2., 3.]])
np.dot(A, B)

The above code results into Illegal instruction (core dumped).

Once I set the environment variable, it works though:

import os
os.environ["OPENBLAS_CORETYPE"] = "nehalem"
import numpy as np
A = np.matrix([[1.], [3.]])
B = np.matrix([[2., 3.]])
np.dot(A, B)

That seems like a misidentification of my CPU type. I am not that much of a hardware person, so please bare with improper wording, I will try my best.

My system:

jovyan@742a0164afd5:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
stepping        : 4
microcode       : 0x2000050
cpu MHz         : 2294.609
cache size      : 16896 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 sssyscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smapxsaveopt arat flush_l1d arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4589.21
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
stepping        : 4
microcode       : 0x2000050
cpu MHz         : 2294.609
cache size      : 16896 KB
physical id     : 2
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 sssyscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smapxsaveopt arat flush_l1d arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4589.21
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
stepping        : 4
microcode       : 0x2000050
cpu MHz         : 2294.609
cache size      : 16896 KB
physical id     : 4
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 sssyscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smapxsaveopt arat flush_l1d arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4589.21
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
stepping        : 4
microcode       : 0x2000050
cpu MHz         : 2294.609
cache size      : 16896 KB
physical id     : 6
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 sssyscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smapxsaveopt arat flush_l1d arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4589.21
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:


jovyan@742a0164afd5:~$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

jovyan@742a0164afd5:~$ uname -a
Linux 742a0164afd5 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Furthermore, what might be of interest:
The place where the crash happens is a docker container (current docker version) running on a virtual machine (I have access to that) running on a cluster (I don't have access to).

For further references see e.g.:

• https://github.com/numpy/numpy/issues/11517
• https://github.com/jupyter/docker-stacks/issues/811
• https://github.com/conda/conda/issues/8373

[martin-frbg]

This is supposed to be fixed in the current develop version - the problem is that cpu type detection relied on the cpu model numbers without taking into account that the operating system (or in your case a virtualizer) may remove some of the original capability of the cpu. In your case, what looks like SKYLAKEX by cpuid alone is apparently unable to handle AVX2 instructions, let alone AVX512. You will probably get some speedup by specifying SANDYBRIDGE instead of NEHALEM.

[1kastner]

Thanks for that hint!
I'll be looking forward to when it is finalized and then used by the packages I rely on!

[1kastner]

As the issue seems to have been addressed already (even though not yet released) and I have found an acceptable solution, this ticket might as well be closed. It depends on how you like to organize issues.

[brada4]

Actually there is AVX2 in CPUID, so the build should be NO_AVX512 (resulting in HASWELL) only to work around bug addressed in next release , NO_AVX2 NO_AVX - add them one by one until SIGILL goes away, avx2 is in cpuid bits but it generates sigill? then there is a problem with hypervisor, probably worth making them aware of culprit.

[martin-frbg]

Right, missed the avx2 in the flags - so HASWELL should be best.

[1kastner]

For all people who find this article and think about setting the environment variable like I did, see
https://github.com/xianyi/OpenBLAS/blob/master/driver/others/dynamic.c#L421

[brada4]

You mean cpuid_x86.c probably...

[1kastner]

Well, I meant the selection through the environment variable as described in https://github.com/xianyi/OpenBLAS/blob/master/USAGE.md#how-to-choose-target-manually-at-runtime-when-compiled-with-dynamic_arch -- the implementation of reading the environment variable is in https://github.com/xianyi/OpenBLAS/blob/76a66eaac8aaa795dddc26af4e43acb455654a18/driver/others/dynamic.c#L758 and https://github.com/xianyi/OpenBLAS/blob/76a66eaac8aaa795dddc26af4e43acb455654a18/driver/others/dynamic_arm64.c#L167 as far as I can tell. At least there the environment variable name occurs when I search the repository.

I guess at https://github.com/xianyi/OpenBLAS/blob/master/cpuid_x86.c we rather see how the selection process works. As long as old versions of this library are out there (and not everybody can update their systems to their liking) the environment variable solution needs to be used, I fear.

[brada4]

Probably better wording is needed in that document (like "e.g. to filter out CPU features omitted by virtualisation software")

[1kastner]

I guess it could be a great help to have an explanation of how to appropriately select an architecture once a virtualization layer is in the middle.

[brada4]

Some virtualisation engines lie about CPUIDs (way before this AVX-512 bug was conceived actually ignoring CPUID capabilities, but going ahead with brand labels only)
The thing with documentation is how everyone sees the problem approaching - you cannot explain to everybody in same words. You can try at https://github.com/xianyi/OpenBLAS/wiki/faq#qemu

[ll-portes]

Hi! I'm having the same problem, but not with a virtual environment. My machine runs on Ubuntu 18.04.2 LTS, with Intel i9-7980XE cpu, and Anaconda3-2019.03-Linux-x86_64.sh installed. The same Conda/Ubuntu environment with an i7-7700 cpu has no problems... However, my "partial" solution for the i9-7980XE machine was a little bit different:

conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

And only then use this code in my Jupyter notebooks:

import os
os.environ['OPENBLAS_CORETYPE']='Haswell'

It is a "partial" solution because I just discovered that if I use the Python Igraph package, the computer starts again to freeze/reboot.

Just setting the environment variable didn't work. For one week, I tried without success several combinations of (re)installing different Anaconda versions, update and downgrade mkl, blas etc. I know nothing about computer science/programming, and so I don't know where I should report this issue. Since I found the solution here (thank you!), I thought it could be useful to report here as well (sorry if I did it wrong).

[1kastner]

Did you try to use the debugger and always stepped in until you found the one Python command which kills it?

[brada4]

freeze-reboot is typically cooling problem, far out of reach of usermode programs (kernel has memory protection, you know)

[ll-portes]

Thank you for the replies!
@1kastner, I'll do this and report here. Thank you.

@brada4, I agree with you, and I thought it was the case when everything happenned. But the connection between using os.environ['OPENBLAS_CORETYPE']='Haswell' and not using has been completely deterministic. For example, just after a freeze/reboot, I can run the same code (but now running this command as the first one) and the computer works fine. It will work even with joblib and all the 18 cores at 100%. So, the computer can be cold or hot, and the only requirement to freeze/reboot is running or not the code above. (Note: yesterday, by using IGraph, it will free/reboot anyway...).

I'll try to find a similar computer to verify if the same is happening. Thank you all again.

[brada4]

The quick workaround to the problem is to build NO_AVX512=1
i.e OpenBLAS wrongly uses AVX-512 ISA in a VM that masks it out from otherwise capable CPU
It is not specific to OpenBLAS, but it is quite a problem getting physical host crashing from a VM userland process that uses what should be just SIGILL.

[martin-frbg]

Closing as 0.3.6 with the fix for AVX512 (mis)detection has been released

[1kastner]

I can't wait to see it on https://anaconda.org/conda-forge/openblas !