Spacy: Illegal instruction in lex_attrs

[danielhers]

I'm not sure why, but on some machines I get this and on some machines I don't. After loading spaCy, I get Illegal instruction. Using strace I saw this happens after opening the file lib/python3.5/site-packages/spacy/lang/__pycache__/lex_attrs.cpython-35.pyc, so I suppose it's due to some code in there.
The following code already causes this:

import spacy
spacy.load("en")

Info about spaCy

• Platform: Linux-4.8.4-aufs-1-x86_64-with-debian-stretch-sid
• Models: fr_core_news_md, fr, en, de, en_core_web_md, en_core_web_sm, de_core_news_sm, en_core_web_lg
• Python version: 3.5.2+
• spaCy version: 2.0.3

[danielhers]

Seems to be fixed by

pip uninstall spacy
pip --no-cache-dir install spacy

On a virtualenv specific to the machine I'm running on. Looks like build outputs were cached on network filesystem from another machine.

[honnibal]

This problem happens to me constantly during development. I wish I had a better solution than specifying --no-cache-dir.

[danielhers]

Nice to know I'm not the only one this happens to :)

[msta]

We have the same problem. It's especially difficult because we're shipping a Docker image with spacy preinstalled, and SIGILL crashes the Python process entirely, so we are basically forced to reinstall spacy unless this gets fixed

[astronomeralex]

Having the exact same issue at @msta :(

[benhjames]

Yes, same issue here! It was running fine on 1.9.0, but upgrading to 2.0.0 started causing Illegal instruction (core dumped).

[benhjames]

For reference, here's the strace output I'm getting for Spacy 2.0.5:

stat("/usr/local/lib/python2.7/site-packages/spacy/lang/unicodedata", 0x7fff0af0d2f0) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.7/site-packages/spacy/lang/unicodedata.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.7/site-packages/spacy/lang/unicodedatamodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.7/site-packages/spacy/lang/unicodedata.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/python2.7/site-packages/spacy/lang/unicodedata.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
close(5)                                = 0
close(4)                                = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b6c6f6000
close(3)                                = 0
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b6c6b6000
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x7f5b5aadbcea} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

[benhjames]

Further investigation with gdb led me to this:

Program received signal SIGILL, Illegal instruction.
0x00007fffd9eb6cea in __pyx_f_5spacy_6lexeme_6Lexeme_c_set_flag (__pyx_v_lex=<optimized out>, __pyx_v_value=0, __pyx_v_flag_id=<optimized out>)
    at spacy/vocab.cpp:20292
20292   spacy/vocab.cpp: No such file or directory.

Don't know if that means anything to anyone!

[philgooch]

I also get this crash when trying to load spaCy within a Dockerised container on Heroku, even when installing, uninstalling and reinstalling via --no-cache-dir. I've raised this with Heroku support to see if there is something they can do at their end.

[byrro]

@philgooch Did you have any luck with Heroku support? I'm using Spacy in an AWS Lambda container and having the same issue... Their support was unable to present a solution. It seems to be an issue when Spacy runs "containerized". Crazy part is that we deployed the same code on multiple containers with the exact same configuration. Some work, others don't...

[philgooch]

@byrro No, unfortunately they were unable to help either. I've had to revert to spaCy 1.10.0 in order to deploy it reliably with Docker.

[byrro]

@philgooch Version 1.10 also worked well on Lambda.

I received more details from AWS team and the problem is that some machines in the Lambda server fleet do not support a compiled code from Spacy. Here's what they said:

A lot of "invalid opcode" errors come from vocab.cpython-36m-x86_64-linux-gnu.so. The issue occurs when the Lambda function runs on a specific instance type that does not support one or more of its compile time optimisation options.

$ objdump --dwarf vocab.cpython-36m-x86_64-linux-gnu.so | grep 'avx' shows that the library is compiled with -march=core-avx2. These options are hardware specific and works correctly when the Lambda function just happens to run on the right hardware.

Older versions of Spacy do not have code compiled with this option. That's why version 1.10 works and 2.0.X doesn't.

[byrro]

Just got a response from the Spacy team (@honnibal). The compiler flag that's causing the issue will be removed in the next version (to be released in approx. 2 weeks). One could also compile from source without the flag and have the issue fixed before that.

[philgooch]

@byrro great detective work, thanks!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.