Spacy: pip install fails on spacy v0.100.0

[bdewilde]

Hi, I've never had any problems installing spacy in the past, but for this latest version (on OS X El Capitan, with both Python 3.5 and 2.7.10), pip install failed. The full log output is attached below. After failing to build a spacy wheel, here's what happened:

Installing collected packages: spacy
  Running setup.py install for spacy
    Complete output from command /Users/burtondewilde/.pyenv/versions/3.5.0/bin/python3.5 -c "import setuptools, tokenize;__file__='/private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-nl2o3k3q-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    package init file 'spacy/tests/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/matcher/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/morphology/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/munge/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/parser/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/serialize/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/spans/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/tagger/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/tokenizer/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/tokens/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/vectors/__init__.py' not found (or not a regular file)
    package init file 'spacy/tests/vocab/__init__.py' not found (or not a regular file)
    running build_ext
    building 'spacy.lexeme' extension
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/burtondewilde/.pyenv/versions/3.5.0/include/python3.5m -I/private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/include -I/Users/burtondewilde/.pyenv/versions/3.5.0/include/python3.5m -c spacy/lexeme.cpp -o build/temp.macosx-10.10-x86_64-3.5/spacy/lexeme.o -O3 -Wno-strict-prototypes -Wno-unused-function -mmacosx-version-min=10.8 -stdlib=libc++
    In file included from spacy/lexeme.cpp:259:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/include/numpy/arrayobject.h:4:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/include/numpy/ndarrayobject.h:18:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/include/numpy/ndarraytypes.h:5:
    /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y56zthfr/spacy/include/numpy/npy_endian.h:11:14: fatal error: 'endian.h' file not found
        #include <endian.h>
                 ^
    1 error generated.
    error: command 'clang' failed with exit status 1

Any ideas what's gone wrong? Thanks very much!

pip_install_spacy.txt

[honnibal]

Hmm! This is pretty vexing. We really thought we had the install problems licked with this release.

What happens if you do:

    pip install numpy
    pip install spacy

[bdewilde]

I know! I saw the changes you and Henning have made to make installation easier, sorry for the continued hassle. I already had numpy installed, so I uninstalled then reinstalled:

~$ pip install numpy
Collecting numpy
  Using cached numpy-1.10.4-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.10.4

Then pip install spacy resulted in what looks the same error as before:

...
    running build_ext
    building 'spacy.lexeme' extension
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/burtondewilde/.pyenv/versions/3.5.0/include/python3.5m -I/private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y3gj9cnu/spacy/include -I/Users/burtondewilde/.pyenv/versions/3.5.0/include/python3.5m -c spacy/lexeme.cpp -o build/temp.macosx-10.10-x86_64-3.5/spacy/lexeme.o -O3 -Wno-strict-prototypes -Wno-unused-function -mmacosx-version-min=10.8 -stdlib=libc++
    In file included from spacy/lexeme.cpp:259:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y3gj9cnu/spacy/include/numpy/arrayobject.h:4:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y3gj9cnu/spacy/include/numpy/ndarrayobject.h:18:
    In file included from /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y3gj9cnu/spacy/include/numpy/ndarraytypes.h:5:
    /private/var/folders/w7/dfn6swb12cv8sfbshhqfwwb00000gn/T/pip-build-y3gj9cnu/spacy/include/numpy/npy_endian.h:11:14: fatal error: 'endian.h' file not found
        #include <endian.h>
                 ^
    1 error generated.
    error: command 'clang' failed with exit status 1

[honnibal]

Hey, it's not your fault for the hassle :). Thanks for reporting.

What version of pip and setuptools do you have? Is there any possibility of interaction from some sort of system numpy?

[bdewilde]

I'm using pyenv, so (I think) I should be shielded from system versions. I have the latest version of pip, and was using v19.2 of setuptools; after upgrading to the latest (19.4), I got the same error for installing spacy.

[honnibal]

What do you get for

pip list

I use pyenv too but sometimes I get tricked, especially by the need to "pyenv rehash".

[bdewilde]

Here's my pip list:

~$ pip list
affinegap (1.10)
alabaster (0.7.7)
appnope (0.1.0)
arrow (0.7.0)
astroid (1.4.1)
Babel (2.2.0)
backports.ssl-match-hostname (3.4.0.2)
beautifulsoup4 (4.4.1)
boto (2.39.0)
boto3 (1.2.2)
botocore (1.3.11)
BTrees (4.2.0)
bz2file (0.98)
cachetools (1.1.5)
canonicalize (1.3)
categorical-distance (1.8)
certifi (2015.11.20.1)
cffi (0.9.2)
chardet (2.3.0)
cld2-cffi (0.1.1)
click (6.2)
cloudpickle (0.1.1)
colorama (0.3.3)
cssselect (0.9.1)
cycler (0.9.0)
cymem (1.30)
Cython (0.23.4)
cytoolz (0.7.4)
decorator (4.0.6)
dedupe (1.2.2)
dedupe-hcluster (0.3.2)
Django (1.9)
docutils (0.12)
fake-factory (0.5.3)
fastcluster (1.1.20)
first (2.0.1)
Flask (0.10.1)
ftfy (4.0.0)
funcsigs (0.4)
future (0.15.2)
futures (3.0.3)
fuzzywuzzy (0.8.0)
gensim (0.12.3)
gnureadline (6.3.3)
haversine (0.4.4)
highered (0.1.6)
html5lib (0.9999999)
httpretty (0.8.10)
idna (2.0)
ipykernel (4.2.2)
ipython (4.0.1)
ipython-genutils (0.1.0)
ipywidgets (4.1.1)
itsdangerous (0.24)
Jinja2 (2.8)
jmespath (0.9.0)
jsonschema (2.5.1)
jupyter (1.0.0)
jupyter-client (4.1.1)
jupyter-console (4.0.3)
jupyter-core (4.0.6)
Keras (0.3.0)
langdetect (1.0.5)
lazy-object-proxy (1.2.1)
line-profiler (1.0)
logilab-common (1.1.0)
lxml (3.5.0)
MarkupSafe (0.23)
matplotlib (1.5.1)
memory-profiler (0.39)
Metafone (0.5)
mistune (0.7.1)
mock (1.3.0)
murmurhash (0.26.0)
mwclient (0.8.0.dev1)
mwparserfromhell (0.4.3)
nbconvert (4.1.0)
nbformat (4.0.1)
networkx (1.10)
nltk (3.1)
nose (1.3.7)
nose2 (0.5.0)
notebook (4.1.0)
numpy (1.10.4)
pandas (0.17.1)
path.py (8.1.2)
patsy (0.4.1)
pbr (1.8.1)
persistent (4.1.1)
pexpect (4.0.1)
pickleshare (0.5)
Pillow (3.0.0)
pip (8.0.0)
pip-tools (1.2.0)
plac (0.9.1)
preshed (0.46.1)
psycopg2 (2.6.1)
ptyprocess (0.5)
pycld2 (0.31)
pycparser (2.14)
Pygments (2.1)
pyhacrf (0.1.2)
PyLBFGS (0.2.0.3)
pylint (1.4.4)
pyparsing (2.0.7)
Pyphen (0.9.2)
pyquery (1.2.9)
python-dateutil (2.4.2)
python-Levenshtein (0.12.0)
pytz (2015.7)
PyYAML (3.11)
pyzmq (15.1.0)
q (2.6)
qtconsole (4.1.1)
readability-lxml (0.6.1)
requests (2.8.1)
rlr (2.1)
scikit-learn (0.17)
scipy (0.16.1)
seaborn (0.6.0)
semver (2.2.1)
setuptools (19.4)
SexMachine (0.1.1)
simplecosine (1.0)
simplegeneric (0.8.1)
simplejson (3.8.1)
six (1.10.0)
smart-open (1.3.2)
snakeviz (0.4.0)
snowballstemmer (1.2.1)
Sphinx (1.3.4)
sphinx-rtd-theme (0.1.9)
sputnik (0.7.2)
statsmodels (0.6.1)
sympy (0.7.6.1)
terminado (0.6)
text-unidecode (1.0)
textblob (0.11.0)
Theano (0.7.0)
thinc (4.2.0)
tldextract (1.7.4)
toolz (0.7.4)
tornado (4.3)
traitlets (4.0.0)
ujson (1.35)
Unidecode (0.4.18)
virtualenv (13.1.2)
Werkzeug (0.11.2)
wget (3.2)
wheel (0.26.0)
wikipedia (1.4.0)
wrapt (1.10.5)
xlrd (0.9.4)
yapf (0.6.2)
zmq (0.0.0)
zope.index (4.1.0)
zope.interface (4.1.3)

Please let me know what else I can do to help debug. And thanks a ton for the quick responses!

[honnibal]

Just reproduced this on my machine. The problem seems to have been introduced with pip 8! I think it's to do with wheel install. @henningpeters , we've been pip'ed =/.

Working on a workaround.

[honnibal]

Yeah...pip 8 was released _yesterday_, and has backwards incompatible changes that screw us. We'll come up with a solution.

[bdewilde]

Well, _shoot_. Hope it's an easy fix. Thanks again for the quick turnaround, @honnibal! Much appreciated.

[honnibal]

pip was innocent after all.

The issue here is that we somehow have to get access to numpy and murmurhash header files at compile time. The stated mechanism to do this in setuptools, setup_requires, is quite broken. You can't specify the same package as both a setup requirement and an install requirement. Naturally we want the numpy headers because we want to use numpy. My previous hack was to use an auxiliary package, headers_workaround, which could be listed in setup_requires, and which would supply the headers. However, this was brittle, as some environments would have slightly different file structures.

The new solution by @henningpeters builds a directory include/ during the source distribution, so that we can access the files during installation, but not install them. This works very well, and solves all the niggly problems introduced by headers_workaround. It also fixes some remaining problems, to help us finally support Windows.

The catch is that we didn't realise that the numpy install process rewrites the headers so that they're platform specific. This bug only surfaced during release, because that's the first time that we tried to install a package built on our Linux server on OSX.

Our CI server tests 32 and 64 bit builds on Ubuntu, OSX and Windows, for Python2.7 and Python3.4. We test pip installation and source compilation, and we're about to add conda installation. But we don't have any tests for cross-platform builds yet.

Long story short: we'll have the fixed package out soon. And our build tensor will grow some new dimensions :)

[bdewilde]

Well, I can confirm that v0.100.1 does indeed install. Thanks a ton for the detective work and very quick resolution! One _last_ nit: I tried to update the data, but got a new error. Looks like sputnik doesn't know about the latest version?

~$ python -m spacy.en.download
Traceback (most recent call last):
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/spacy/en/download.py", line 58, in <module>
    plac.call(main)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/plac_core.py", line 309, in call
    cmd, result = parser_from(obj).consume(arglist)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/plac_core.py", line 195, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/spacy/en/download.py", line 42, in main
    package = sputnik.install(about.__name__, about.__version__, about.__default_model__)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/sputnik/__init__.py", line 44, in install
    archive = cache.fetch(package_name)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/sputnik/cache.py", line 57, in fetch
    package = self.get(package_string)
  File "/Users/burtondewilde/.pyenv/versions/3.5.0/lib/python3.5/site-packages/sputnik/package_list.py", line 68, in get
    (self.app_name, self.app_version, package.compatibility))
sputnik.package_list.CompatiblePackageNotFoundException: running spacy 0.100.1 but requires {'spacy': '==0.100.0'}

Sorry again...

[honnibal]

Yeah, discovered this just after I pushed. The fix is easy but we can't push it out until tomorrow. The penalty for pushing out a hasty fix at 10pm, with a new process.

The only thing that's failing here is the assertion --- the pin is too tight, it's supposed to be ranged on minor version instead of exact. So, if you just want to get started, the best thing to do right now is just to make a quick local edit to either the code that does the consistency check or the package.json file that declares the compatibility.

[henningpeters]

It should now be fixed. Please upgrade spaCy to >=0.100.2. Sorry for the trouble.

[bdewilde]

@henningpeters I saw — and successfully updated package and data. Thanks a ton!

[notnami]

I am trying to install spacy 0.100.3 on a venv with pip 8.0.2 (OS X 10.11.3), and still getting this installation error, where 'endian.h' is not found. I have spent a while rm-ing and reinstalling the virtual environments, trying various combinations of venv and virtualenv and even conda (which doesn't seem to have the latest version of spacy), but nothing has helped. Any ideas?

[henningpeters]

Sorry, this was a packaging error on our side that slipped through again. Can you please try again with 0.100.5?

[phdowling]

I'm seeing this error on anaconda now! I can import spacy, but when I try downloading model data:

Philipps-Air:expertise_classification_keras dowling$ python -m spacy.en.download
Traceback (most recent call last):
  File "/Users/dowling/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Users/dowling/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/dowling/anaconda/lib/python2.7/site-packages/spacy/en/download.py", line 13, in <module>
    plac.call(main)
  File "/usr/local/lib/python2.7/site-packages/plac_core.py", line 309, in call
    cmd, result = parser_from(obj).consume(arglist)
  File "/usr/local/lib/python2.7/site-packages/plac_core.py", line 195, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/dowling/anaconda/lib/python2.7/site-packages/spacy/en/download.py", line 9, in main
    download('en', force)
  File "/Users/dowling/anaconda/lib/python2.7/site-packages/spacy/download.py", line 24, in download
    package = sputnik.install(about.__title__, about.__version__, about.__models__[lang])
  File "/usr/local/lib/python2.7/site-packages/sputnik/__init__.py", line 44, in install
    archive = cache.fetch(package_name)
  File "/usr/local/lib/python2.7/site-packages/sputnik/cache.py", line 57, in fetch
    package = self.get(package_string)
  File "/usr/local/lib/python2.7/site-packages/sputnik/package_list.py", line 68, in get
    (self.app_name, self.app_version, package.compatibility))
sputnik.package_list.CompatiblePackageNotFoundException: running spacy 0.100.7 but requires {u'spacy': None}

[ghost]

I am getting the same error now. I haven't updated my version of spacy (still at 0.100.6), but starting today I get the following error when installing via pip:

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/spacy/en/download.py", line 58, in <module>
    plac.call(main)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/plac_core.py", line 309, in call
    cmd, result = parser_from(obj).consume(arglist)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/plac_core.py", line 195, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/spacy/en/download.py", line 42, in main
    package = sputnik.install(about.__title__, about.__version__, about.__default_model__)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/sputnik/__init__.py", line 44, in install
    archive = cache.fetch(package_name)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/sputnik/cache.py", line 57, in fetch
    package = self.get(package_string)
  File "/home/ryan/Envs/extractor/lib/python3.4/site-packages/sputnik/package_list.py", line 61, in get
    raise CompatiblePackageNotFoundException(package_string)
sputnik.package_list.CompatiblePackageNotFoundException: en>=1.0.0,<1.1.0

[evaljy]

Hi I still have the same trouble to install my Spacy on both conda and pip. Are there any solutions for that? Thanks!

[supertramp01]

Hi, I've bumped into similar issues. Will there be any fixes soon ?

Thanks

[TimothyMwiti]

I am facing the same problem while trying to install using pip for Python 2.7. Everytime I run the command it mt.exe (from the microsoft tool) keeps failing and thereafter the whole process fails too with a 'Failed building wheel for spacy' error.

[tripathysa]

Same issue getting for spacy version : 0.100.6

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.