Spacy: -march=native option makes gcc compiler to fail on Power8 system with Ubuntu

[sdmonov]

My Environment:

• Operating System: Ubuntu 16.04.3 LTS on Power8
• Python Version Used: Python 3.6.2 |Anaconda, Inc.| (default, Sep 15 2017, 20:38:23) [GCC 4.8.4] on linux
• spaCy Version Used: Master from 11/16/2017 (commit a3d4dd1a5d1ea827761bcf04302762936d0e143c)
• gcc version: 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.5)
• System Information: PowerNV 8335-GTB with POWER8NVL CPU

Recent commit 79fcf8576aa077749c91315d102c0d70d888ca60 added -march=native to the compile options for gcc/clang in setup.py. The problem is that -march=native is not supported by all the gcc compilers and as far as I know some older versions of clang too. I've tested it with gcc on ppc64le. Also tested gcc-arm-linux-gnueabi ARM cross compiler on x86 with Ubuntu 14.04.

This is the output of the ppc64le:

............
copying spacy/tokens/doc.pyx -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tokens/token.pyx -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tokens/span.pxd -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tokens/token.pxd -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tokens/__init__.pxd -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tokens/doc.pxd -> build/lib.linux-ppc64le-3.6/spacy/tokens
copying spacy/tests/tokenizer/sun.txt -> build/lib.linux-ppc64le-3.6/spacy/tests/tokenizer
running build_ext
building 'spacy.parts_of_speech' extension
creating build/temp.linux-ppc64le-3.6
creating build/temp.linux-ppc64le-3.6/spacy
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/user/miniconda3/envs/spacy/include/python3.6m -I/home/user/AI/spaCy/include -I/home/user/miniconda3/envs/spacy/include/python3.6m -c spacy/parts_of_speech.cpp -o build/temp.linux-ppc64le-3.6/spacy/parts_of_speech.o -O3 -Wno-strict-prototypes -Wno-unused-function -march=native
gcc: error: unrecognized command line option ‘-march=native’
error: command 'gcc' failed with exit status 1

After removing the -march=native option, spaCy builds and runs just fine.

Here is the output from the arm cross compiler.

................
writing top-level names to spacy.egg-info/top_level.txt
reading manifest file 'spacy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'spacy.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
building 'spacy.parts_of_speech' extension
arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/moni/miniconda3/envs/pytorch/include/python3.6m -I/home/moni/src/ibm/AI/spaCy/include -I/home/moni/miniconda3/envs/pytorch/include/python3.6m -c spacy/parts_of_speech.cpp -o build/temp.linux-x86_64-3.6/spacy/parts_of_speech.o -O3 -Wno-strict-prototypes -Wno-unused-function -march=native
cc1plus: error: unrecognized argument in option ‘-march=native’
cc1plus: note: valid arguments to ‘-march=’ are: armv2 armv2a armv3 armv3m armv4 armv4t armv5 armv5e armv5t armv5te armv6 armv6-m armv6j armv6k armv6s-m armv6t2 armv6z armv6zk armv7 armv7-a armv7-m armv7-r armv7e-m armv8-a armv8-a+crc iwmmxt iwmmxt2 native
cc1plus: warning: command line option ‘-Wno-strict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

I guess the best solution would be to add code to detect the best CPU flags and add them as -march parameters or simply detect if -march=native is supported by the compiler and only use it in such case.

[honnibal]

Thanks; very valuable report and analysis.

I think spaCy should not set -march=native. There are a few dot products implemented as basic loops. This is fine so long as the compiler vectorises the loops reasonably well, but I see that cross-platform support is going to be a real problem. Instead of trying to get this right in spaCy, it'll be better to call into a library.

If we get rid of these dot products we can also compile with -O2, to hopefully reduce compile times, which have been creeping upwards. The long compilation is starting to get quite annoying, and there was a report of an out-of-memory error on a 1gb VM.

[rmax]

Somewhat related, the default -march=native seems to cause problems in the conda packages: https://github.com/conda-forge/spacy-feedstock/issues/24

[sdmonov]

Looks like the best would be the if compiler CPU optimization flags can be provided through env variable for example in case of binaries generation or detected using a library if not provided for local installs optimization.

[benhjames]

I think this might cause https://github.com/explosion/spaCy/issues/1589

I was building a Docker image with spaCy with a different set of supported CPU instructions to the target set, which results in Illegal instruction. It would seem -march=native causes this to be the case.

I think the built library should be largely independent of target platform. For now, the workaround is simply to build the Docker image on the target platform (instead of "anywhere"!)

[NicolasEhrhardt]

We weren't able to use spaCy 2.0.5 on EMR because of this issue. We had to publish an internal version that reverses commit https://github.com/explosion/spaCy/commit/a3d4dd1a5d1ea827761bcf04302762936d0e143c in order to avoid the Illegal instruction error at import time. I second folks suggesting that the -march=native option should not be the default bdist option.

[honnibal]

Thanks all for the report. I've now removed the march=native flag.

[megatesla]

The closest equivalent option for Power systems is currently -mcpu=native. See these gcc docs for details.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.