Incubator-mxnet: Illegal instruction (core dumped)

[Hsintao]

After ipip installed the mxnet-cu80==1.2.1 or 1.3.0,import mxnet will report Illegal instruction (core dumped),the 1.1.0 is OK,but i want to use the gluoncv.
SystemOS: ubuntu16.04
GPU: NVIDIA GeForce GTX Titan X
CUDA:8.0
CuDNN:7

However, i install it in another server with GTX1080TI.
Is it because TITANX is the Maxwell architecture ？

[haojin2]

Can you also post the whole error message here?

[Hsintao]

just Illegal instruction (core dumped). i google this and cant find any solution. @haojin2

[szha]

@Hsintao what's the CPU that you're using? For pre-built package we need at least avx2 support.

[Hsintao]

Intel(R) Xeon(R) CPU E5620 @ 2.40GHz @szha

[szha]

Just double checking, if you do pip install mxnet-cu80==1.3.0b20180703, does it throw the error?

[Hsintao]

I just try this version，same error。。。。 我试试从源码编译 @szha

[Hsintao]

i build mxnet from source and it still doesn't work. Thanks.
please closes this issue. @szha

[szha]

~Glad it works, @Hsintao~. Also, looks like this specific CPU is limited to SSE4.2.

[Hsintao]

Sorry, it doesn't work。 i will continue to use the 1.1.0 @szha
Thank u anyway.

[szha]

Sorry I misread your message. In this case I think it would be better for us to figure out why it doesn't work, as building from source should cover almost all platforms.

Would you mind pasting your complete build log here when building from scratch?

[Hsintao]

build.log.
Maybe it's my computer's problem. @szha
I will rebuilt it and i will tell u if it works.
Sorry to trouble you so much.

[Hsintao]

it works. thank u again. @szha

[mirekphd]

The issue is more general (affecting mxnet-cu90) and the explanation given must be different (the specified CPU instruction set does not prevent it from occuring, but downgrading version to 1.2.0 does).

No need to reopen the issue, as it seems to have been solved in one of the pre-releases of version 1.3.1 of the mxnet-cu90 library - I confirm it is gone in the latest such pre-release: 1.3.1b20181004.

@Hsintao : does version 1.3.1 pre-release of mxnet-cu80 library solve the bug for you too? Try e.g.:
pip install mxnet-cu80==1.3.1b20181004

More info
I keep getting this error ("Illegal instruction (core dumped)") in python, or a nondescript kernel restart (with a core dump) in Jupyter Notebook for a different CUDA build (mxnet-cu90 running on nvidia/cuda:9.0-cudnn7-devel image), while I'm not getting it for the originally reported build (mxnet-cu80). Downgrading library version from 1.3.0.post0 to 1.3.0 or 1.2.1 did not help, but 1.2.0 worked (can be loaded).

There are no NVIDIA drivers installed in my build server's host system (which indeed prevents some other GPU-enabled libs loading, notably CNTK and Tensorflow), but CPU does support SSE 4.2 instructions:

[root@mx01 ml-gpu-py36-cuda90]# cat /proc/cpuinfo | grep flags
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 **sse4_2** x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm tsc_adjust xsaveopt

[Vermeille]

Same shit with all the gpu docker images I've tested so far.

[szha]

This is likely related to the f16c-related optimization on CPU. It looks like the target CPU doesn't support f16c instruction set.

[Vermeille]

I'm talking about GPU. Just to make sure.

[szha]

@Vermeille yes, you mentioned so. Still, f16c optimization is a CPU instruction set, and is only enabled in GPU versions. It results in import error if the CPU on the target hardware doesn't support f16c. @apeforest is looking into it

[GSanchis]

Any update on this? I believe I'm stumbling into the same issue. pip-installing mxnet with versions 1.2.1 or 1.3.0 leads in my case also to the same problem upon importing mxnet in python (Illegal instruction (core dumped)).

In my case and in this machine I'm not using the cuda version, so it's just the plain mxnet package from pip. My CPU is a Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz (according to /proc/cpuinfo).

[szha]

@GSanchis According to Intel(R) Core(TM) i5-2520M spec, it doesn't support AVX2 instruction set, which is currently the minimum requirement for pre-built binary.

[GSanchis]

Thanks @szha. Looks like I'll have to compile it from source then. In any case, thanks for the info!

[antoniosap]

1.3.1 core dumped

pip install mxnet

ipython
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import mxnet as mx                                                                                                                                      
Istruzione non consentita (core dump creato)

[szha]

@antoniosap what are the supported instruction sets of the CPU you're using?

[antoniosap]

Intel(R) Celeron(R) CPU B820 @ 1.70GHz
...
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer xsave lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm arat pln pts flush_l1d

[chanajianyu]

pip install mxnet-cu90==1.3.1b20181004
and it works

[RustyRaptor]

So is this related to what version of cuda I have installed? I have cuda 10.0 installed and installing mxnet-cu100 didn't seem to do anything

[szha]

@antoniosap it's again due to the lack of support for avx2 instruction set, which is a minimum requirement of the pre-built binary. Building from source will fix the problem.

[szha]

@RustyRaptor whether it causes illegal instruction error upon import shouldn't depend on the cuda version per se. There was a problem in the build logic that causes illegal instruction on CPUs without f16c instruction, though the problem should have been fixed.

If this is not the case, could you share what version of mxnet you are using, and the supported CPU instruction set on your platform?

[RustyRaptor]

could you share what version of mxnet you are using, and the supported CPU instruction set on your platform?

I am using 1.3.1

here are my flags
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti tpr_shadow vnmi flexpriority ept vpid dtherm ida arat

[RustyRaptor]

I'll go ahead and try building from source then.

[RustyRaptor]

Ok After compiling the program and going into the python directory and running sudo pip install -e .
It works. Thanks.