Spacy: Illegal hardware instruction in spacy/lang/en/syntax_iterators

[sr-rolando]

Running a fresh install of spacy on a FreeBSD system results in »illegal hardware instruction« and a core dump.

How to reproduce the problem

% python3.7 -m venv spacytest
% source spacytest/bin/activate
% pip --no-cache-dir install spacy
% python -m spacy download en_core_web_sm
% python -m spacy download de_core_news_sm
% python -v -c "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')"
[...]
# code object from '/home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/morph_rules.cpython-37.pyc'
import 'spacy.lang.en.morph_rules' # <_frozen_importlib_external.SourceFileLoader object at 0x8059aae90>
# /home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc matches /home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/syntax_iterators.py
# code object from '/home/r/temp/spacytest/lib/python3.7/site-packages/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc'
import 'spacy.lang.en.syntax_iterators' # <_frozen_importlib_external.SourceFileLoader object at 0x8059ad150>
import 'spacy.lang.en' # <_frozen_importlib_external.SourceFileLoader object at 0x805831890>
zsh: illegal hardware instruction (core dumped)  python -v -c

Your Environment

• spaCy version: 2.2.4
• Platform: FreeBSD-12.1-RELEASE-p3-amd64-64bit-ELF
• Python version: 3.7.7

Model compatibility

The loaded model looks good:

% python -m spacy validate
✔ Loaded compatibility table

====================== Installed models (spaCy v2.2.4) ======================
ℹ spaCy installation:
/home/r/temp/spacytest/lib/python3.7/site-packages/spacy

TYPE      NAME              MODEL             VERSION
package   en-core-web-sm    en_core_web_sm    2.2.5   ✔
package   de-core-news-sm   de_core_news_sm   2.2.5   ✔

Pointers on what I might be missing or how to track down the root cause are very much appreciated.

[adrianeboyd]

Hmm, to try to narrow things down: does it work if you compile from source as described here? https://github.com/explosion/spaCy/#compile-from-source. After compiling, install with pip install ..

[sr-rolando]

Thanks, good idea. I happily tried that as follows:

git clone https://github.com/explosion/spaCy.git
cd spaCy
python3.7 -m venv .env
source .env/bin/activate
pip install -U pip
pip install wheel
pip install cython
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py build_ext --inplace
pip install .
python -m spacy download en_core_web_sm
python -v -c "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')"

Unfortunately, the result is just about the same as before:

[...]
import 'spacy.lang.en.lex_attrs' # <_frozen_importlib_external.SourceFileLoader object at 0x805b78490>
# /home/r/temp/spaCy/spacy/lang/en/__pycache__/morph_rules.cpython-37.pyc matches /home/r/temp/spaCy/spacy/lang/en/morph_rules.py
# code object from '/home/r/temp/spaCy/spacy/lang/en/__pycache__/morph_rules.cpython-37.pyc'
import 'spacy.lang.en.morph_rules' # <_frozen_importlib_external.SourceFileLoader object at 0x805b78d50>
# /home/r/temp/spaCy/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc matches /home/r/temp/spaCy/spacy/lang/en/syntax_iterators.py
# code object from '/home/r/temp/spaCy/spacy/lang/en/__pycache__/syntax_iterators.cpython-37.pyc'
import 'spacy.lang.en.syntax_iterators' # <_frozen_importlib_external.SourceFileLoader object at 0x805b7bfd0>
import 'spacy.lang.en' # <_frozen_importlib_external.SourceFileLoader object at 0x805a07410>
zsh: illegal hardware instruction (core dumped)  python -v -c

Thus, hints on how else to dig into this would still be helpfull.

[sr-rolando]

In case that's of any help, running spacy's tests results in the same:

% python -m pytest .
=================================================================================== test session starts ===================================================================================
platform freebsd12 -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /home/r/temp/spaCy, inifile: setup.cfg
plugins: timeout-1.3.4
collected 2638 items

spacy/tests/test_architectures.py .                                                                                                                                                 [  0%]
spacy/tests/test_cli.py ....                                                                                                                                                        [  0%]
spacy/tests/test_displacy.py .......                                                                                                                                                [  0%]
spacy/tests/test_gold.py ..................                                                                                                                                         [  1%]
spacy/tests/test_json_schemas.py ...........                                                                                                                                        [  1%]
spacy/tests/test_language.py Fatal Python error: Illegal instruction

Current thread 0x0000000800a39000 (most recent call first):
  File "/home/r/temp/spaCy/.env/lib/python3.7/site-packages/thinc/neural/_classes/maxout.py", line 76 in begin_update
  File "/home/r/temp/spaCy/.env/lib/python3.7/site-packages/thinc/neural/_classes/layernorm.py", line 62 in begin_update
  File "/home/r/temp/spaCy/.env/lib/python3.7/site-packages/thinc/neural/_classes/feed_forward.py", line 46 in begin_update
  File "/home/r/temp/spaCy/.env/lib/python3.7/site-packages/thinc/api.py", line 379 in uniqued_fwd

Line 76 of thinc's maxout.py is:

         output__boc = self.ops.gemm(X__bi, W, trans2=True)

Which, honestly, doesn't tell me much.

[adrianeboyd]

Hmm, I don't really know at this point and it's not one the platforms we support, so it's hard for us to do any testing.

I would try pip install -U -r requirements.txt and recompiling first to see if that helps (especially updating cython), but otherwise I'm afraid I don't know. I think you might need to recompile a number of dependencies from source, too, probably thinc, srsly, and blis. Given the gemm error above, see the notes for blis: https://github.com/explosion/cython-blis/#building-blis-for-alternative-architectures

[sr-rolando]

Thanks for the pointer that the gemm method belongs to blis.

After building blis, srsly, thinc and spaCy from source (in this order) according to their respective README's and following up with a pip install . in each case, I still get the illegal hardware instruction.

I'll see to run some test suites of the packages that spaCy depends on. But if that doesn't reveal anything helpful, I might just have to setup the first Linux machines in the house in order to get the app using spaCy up and running.

(Thanks for your help and pointers so far!)

[adrianeboyd]

I'm afraid I really don't know much more about the details here. Does numpy work as expected?

If you can switch to linux, that might be the simplest way to get spacy up and running.

[svlandeg]

Wondering whether maybe @outpaddling has any idea ?

[outpaddling]

The error seems to indicate that zsh is crashing, not spaCy. What's the name of the core file? I would suspect zsh.core.

Illegal instruction generally means it was compiled with optimizations to utilize instructions that are not present on the system running it. So how/where was zsh built/installed and on what processor are you running it?

(grep CPU /var/run/dmesg.boot)

[sr-rolando]

@outpaddling Thanks, that sounds like a valid assumption. But it's a python3.7.core dump. This python has been installed by compiling the lang/python37 port. /etc/make.conf is:

WITH_PKGNG=yes
# Python 3.7 (not 3.8 as that's not running well with poetry)
DEFAULT_VERSIONS= python3=3.7 python=3.7
# curl doesn't like OpenSSL from ports
DEFAULT_VERSIONS+=ssl=base

OTOH, I'm pretty sure that you are right in some compiler setting being off. Question is: which one? Didn't find it so far. But after running some more tests, it already seems to be blis that's failing. (Sadly, I'm just totally not familiar with blis in order to debug that.)

As for the processor:

CPU: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz (2095.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x50657  Family=0x6  Model=0x55  Stepping=7
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xf7fa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xd19f4fbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL>
  Structured Extended Features2=0x80c<UMIP,PKU>
  Structured Extended Features3=0x84000400<MD_CLEAR,IBPB,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Hypervisor: Origin = "KVMKVMKVM"

Does that clarify anything?

[svlandeg]

Hi @sr-rolando, have you gotten any further with this?

[sr-rolando]

Well, I could boil it down to seeing that it fails in blis, not spaCy. But even recompiling the whole stack down to Python doesn't resolve the core being dumped.

The current "solution" over here is to move the spaCy application from FreeBSD to some Linux machine. (Not nice but it gets the job done.)

[sr-rolando]

(Thus, close this issue here?)

[svlandeg]

Ah, that's unfortunate. Yea I think none of us on the team have experience with FreeBSD? And it's not a supported platform. It's good to know the issue is with Blis, but I'm afraid we don't really have a solution right now. Sorry!
Still, good to hear you can work around it through a Linux machine ;-)

I guess I'll close, yea. Sorry we couldn't be of more help!

[outpaddling]

Unfortunately, I didn't see the response back in May.

FWIW, I upgraded the spaCy WIP port at https://github.com/outpaddling/freebsd-ports-wip to 2.3.0.

Instructions for installing from the WIP collection are in the README.

I don't know how to test spaCy myself - I originally created the port for a researcher I work with - but it builds and installs cleanly.

Good luck,

JB

[svlandeg]

Oh wow, that's awesome @outpaddling !

@sr-rolando: would be cool if you'd have the time to test this!

[sr-rolando]

Oh, you built your own kind of a ports collection? Crazy interesting.

At a first shot, the build fails here with: »Failed building wheel for srsly«. Tracking this down would require more time than I have at hand right now. (My notes on srsly only state a simple build via python setup.py build_ext --inplace.)

BTW: A brief test for spaCy is:

shell% python -m spacy download en_core_web_sm
shell% python -v -c "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')"

[outpaddling]

Yes, I house my FreeBSD ports here until they're ready to commit to the main repo. It's modeled on the pkgsrc WIP collection (https://pkgsrc.org/wip/) which I use on CentOS, NetBSD, and OS X.

So you installed my WIP collection using the freebsd-wip-checkout script and tried the following?

cd /usr/ports/wip/py-spacy
make install

[outpaddling]

So I added pip and wheel as dependencies for spacy and srsly. I'll have to check setup.cfg in all the ports to make sure they're clean. This allowed the ports spaCy build to download the models, though I have to specify a target to keep it from trying to install to /usr/local/lib/python3.7/site-packages.

Running the script below spewed a lot of what looks like debug output, but I didn't see any errors. I'm not a spaCy user and don't know anything about NLP, so if you want to test this further I'll need feedback from a real user.

#!/bin/sh

for lib in en_core_web_sm de_core_news_sm; do
    if [ ! -e $HOME/spaCy/$lib ]; then
    python3.7 -m spacy download $lib --target=$HOME/spaCy
    fi
done

export PYTHONPATH=/usr/local/lib/python3.7:$HOME/spaCy
python3.7 -v -c \
    "import spacy; spacy_nlp = spacy.load('en_core_web_sm'); spacy_nlp('This is some fine text.')" \
    2>&1 | tee out

[sr-rolando]

Thank you much for adding the dependencies. Your wip-port compiles now fine over here as well. However, even using our most simplistic test case here is still ending up in the "illegal hardware instruction" core dump (and all the log/debug output is due to the -v switch, sorry about silently throwing it in).

As you don't seem to get the core dump, it now looks to me that there is something wrong with my machine. I'll see to get my hands on another one and run some tests there. This will take some time but I'll post an update here once I have it.

Little side note: you might consider adding numpy as a dependency as well:

shell% python3.7 -m spacy info
[...]
ModuleNotFoundError: No module named 'numpy'

Thanks again.

[outpaddling]

${NUMPY} is already the first thing in RUN_DEPENDS. Maybe you're actually running your venv installation by accident? What's the Location shown by spacy info?

FreeBSD coral.acadix  bacon ~ 1003: python3.7 -m spacy info
============================== Info about spaCy ==============================
spaCy version    2.3.0                         
Location         /usr/local/lib/python3.7/site-packages/spacy
Platform         FreeBSD-12.1-RELEASE-p6-amd64-64bit-ELF
Python version   3.7.7                         
Models                                         

In any case, I'd suggest the following to make sure all the latest WIP ports are installed:

cd /usr/ports/wip/py-spacy
pkg remove py37-spacy
pkg autoremove  # Remove everything installed as a dependency
If you installed any deps such as srsly manually, autoremove won't touch them, so remove manually, e.g.
pkg remove py37-srsly
Run 'grep wip Makefile' to see the possibilities.
make clean clean-depends install

Cheers,
Jason

[outpaddling]

I added a script to quickly and cleanly reinstall wip/py-spacy & dependencies:

/usr/ports/wip/Misc/Scripts/reinstall-spacy

For testing in a pristine environment, you might try a fresh FreeBSD install under VirtualBox. It takes me about 3 minutes to do a basic OS install this way using mostly default responses. Then run the following after the first boot:

> pkg install -y auto-admin
> auto-pkg-latest   # If you want bleeding-edge packages instead of quarterly and the ability to use portsnap instead of svn to manage the ports tree.  portsnap is much faster, but does not currently support quarterly branches.
> auto-freebsd-wip-checkout
> /usr/ports/wip/Misc/Scripts/reinstall-spacy

I'd note that there were ZERO patches required to make the FreeBSD ports for spaCy, srsly, etc. work. It was only a matter of installing the necessary prereqs.

For curiosity, I ran the install under pip on FreeBSD 12.1 as well. There seems to be a missing cython dependency, but after pip installing cython, everything went smoothly and I did not get an illegal instruction error when running your test command from the original post. My output from both the ports and pip installations is here:

http://acadix.biz/Misc/

So it doesn't look to me like there are any problems with the basic spaCy install on FreeBSD. Must be something wrong with your OS installation or your hardware.

There doesn't seem to be much advantage to having spaCy & deps in the ports collection given that the pip install is nearly trouble-free. I may commit them anyway, since they're pretty trivial and the ability to run "pkg install py37-spacy" might save someone some time and trouble.

Cheers,

JB

[outpaddling]

Actually, there was a typo in my numpy dep in wip/py-spacy. I just committed a fix.

[sr-rolando]

Yes, you are absolutely right: The build of spaCy and its dependencies is as straight forward on FreeBSD as in most other environments.

I now got my hands on a different machine, rebuild everything from scratch and the first simple tests all run fine. Thus, this was indeed an issue with either the hardware or the setup of the initial machine. It's not an issue with running spaCy on FreeBSD. Sorry about the noise.

BTW: From my point of view there isn't much need to include spaCy in the ports collection. Most applications will install their Python packages in some virtual environment – and thus local for the application – anyways. But other people's need might differ. So, if you have some capacity available, please go ahead and finalize the ports. You are doing some fine work there.

[svlandeg]

Great to hear it runs fine now, @sr-rolando !
And thanks for all your help, @outpaddling !

[outpaddling]

Happy to help when I can.

One other thing you might want to check for is something in your environment, /etc/make.conf, etc. causing builds to use specific optimizations (e.g. AVX) that your CPU does not support. I assume you're running on the same machine that builds the binaries.

[outpaddling]

One more thing you might try is setting BLIS_ARCH=generic before a fresh pip install. I noticed that the blis build system defaults to linux-x86_64 (even on FreeBSD-{i386,powerpc64}) and this build environment has some specific optimizations like avx2 hard-coded. If you're running on old hardware, that could be tripping it up. I haven't looked closely at the other packages, but I wonder if they may be doing the same.
With BLIS_ARCH=generic, the basic test passed on my PowerMac G5 under FreeBSD 12.1 and an i386 build succeeded under poudriere (FreeBSD's containerized port test / package building system).

[outpaddling]

Also noticed that spaCy 2.3.0 rejects the latest releases of catalogue and srsly. Is that an error in the requirements, or must we really use older releases?

[svlandeg]

That's probably correct. The reason is that we're working on spaCy v.3 on the develop branch, and that branch uses updated versions of our other libraries thinc and srsly (not catalogue though I think?). But these new versions are probably not compatible with spaCy 2.x.