Spacy: setup.py has several usability issues

[erg]

In fumbling around with python's build commands, I ran into some issues with setup.py and dev_setup.py.

I'm on MacOSX 10.10.5 and everything should be the latest version.

1) setup.py assumes at least one argument

# I should give it a command, but I forgot:
ergmac:spaCy erg$ [master*] python --version
Python 2.7.10
ergmac:spaCy erg$ [master*] python setup.py
Traceback (most recent call last):
  File "setup.py", line 172, in <module>
    if sys.argv[1] == 'clean':
IndexError: list index out of range

1.a) Same issue for dev_setup.py

ergmac:spaCy erg$ [master*] python dev_setup.py
Traceback (most recent call last):
  File "dev_setup.py", line 116, in <module>
    if sys.argv[1] == 'clean':
IndexError: list index out of range

2) python setup.py install doesn't work. Is that because the command is supposed to be build_ext?

ergmac:spaCy erg$ [master*] python --version
Python 2.7.10
ergmac:spaCy erg$ [master*] python setup.py install
zip_safe flag not set; analyzing archive contents...
headers_workaround.__init__: module references __file__

Installed /Users/erg/spaCy/.eggs/headers_workaround-0.17-py2.7.egg
running install
...
creating build/temp.macosx-10.10-x86_64-2.7/spacy
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c spacy/parts_of_speech.cpp -o build/temp.macosx-10.10-x86_64-2.7/spacy/parts_of_speech.o -O3 -Wno-strict-prototypes
clang: error: no such file or directory: 'spacy/parts_of_speech.cpp'
clang: error: no input files
error: command 'clang' failed with exit status 1

3) My python3 can't find Cython. Not sure who is to "blame" for this one.

ergmac:spaCy erg$ [master*] python3 --version
Python 3.4.3
ergmac:spaCy erg$ [master*] cython --version
Cython version 0.23.1
ergmac:spaCy erg$ [master*] python3 setup.py build_ext
Traceback (most recent call last):
  File "setup.py", line 176, in <module>
    main(MOD_NAMES, use_cython)
  File "setup.py", line 147, in main
    cython_setup(modules, language, includes, compile_args, link_args)
  File "setup.py", line 71, in cython_setup
    import Cython.Distutils
ImportError: No module named 'Cython'

4) dev_setup.py needs murmurhash3 but doesn't install it?

ergmac:spaCy erg$ [master*] python dev_setup.py build_ext
running build_ext
skipping 'spacy/typedefs.cpp' Cython extension (up-to-date)
skipping 'spacy/strings.cpp' Cython extension (up-to-date)
building 'spacy.strings' extension
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c spacy/strings.cpp -o build/temp.macosx-10.10-x86_64-2.7/spacy/strings.o
spacy/strings.cpp:249:10: fatal error: 'murmurhash/MurmurHash3.h' file not found
#include "murmurhash/MurmurHash3.h"
         ^
1 error generated.
error: command 'clang' failed with exit status 1

The command that seems to work for me is:

ergmac:spaCy erg$ [master*] python --version
Python 2.7.10
ergmac:spaCy erg$ [master*] python setup.py build_ext
...
success.

I can't find a working command for dev_setup.py at all.

I'm open to helping fix these issues. Cheers!

[honnibal]

How much experience do you have with Python packaging? All I can say is, if you haven't had to do this before, it's harder than you'd assume =/. If you've fought these demons and won, then okay, I'd be glad of the help. If not, let me explain why the current script seems so crazy.

The thing is, the setup.py file needs to do many different things. It needs to build the project from source (the .pyx files), but it also needs to install the package from a distribution, i.e. be run by setuptools. That process can't depend on Cython. The distribution ships the .cpp files (which are not in the repository).

The two tasks are almost completely different. For a start, when run by setuptools, the install_requires and setup_requires arguments will be respected, and those dependencies will be downloaded. This isn't done if you just write "python setup.py".

The setup.py also includes lots of work-arounds for various bugs. For instance, install_requires doesn't actually work. If you specify the same package in install_requires and setup_requires, it won't actually be installed. I need the murmurhash and numpy header files during installation, so I made this thing, headers_workaround, that fixes pip installation. But, to just compile the project, you should run "pip install -r requirements.txt" (note that this list of requirements are specified separately --- there's no way to tell pip to just read the requirements from setup.py. Obviously.)

The priority for setup.py is that this:

pip install spacy

Must "just work", provided user has a C compiler installed. It took a lot of very frustrating work to get that to happen, but I'm pleased to say that installation works cleanly on the supported platforms, OSX and Linux. This is sort of rare. Most scientific-Python libraries don't install cleanly from pip.

When building the project myself, I use fabric. For instance, this is how I publish a new version:

fab clean make sdist install test publish

This does the following:

• Clean .so and .cpp files
• Generate .cpp and .so files
• Create a distribution tar file, e.g. in dist/spacy-0.90.tar.gz
• Install spaCy from the tar file, into the current virtualenv
• Run py.test tests/
• If all that succeeded, push to master, and upload the new distribution to PyPi.

[erg]

I just found the git install instructions on http://spacy.io/ and the workflow is completely different from what I've used before.

After trying it that way, there was an error with the C murmurhash3 headers, but I just found out that if I do pip install spacy first then murmurhash3 gets installed to the right place and the git build progresses.

Thank your for the detailed reply! I suppose it's user error on my side, except for possibly the murmurhash issue.

[docnick]

I saw error #2 listed above as well, but only when pip installing spacy with python3.5 (it worked without any issues when installing under python2.7).

clang: error: no such file or directory: 'spacy/parts_of_speech.cpp'
clang: error: no input files
error: command 'clang' failed with exit status 1

I ran pip install --upgrade wheel which successfully uninstalled wheel-0.24.0 and installed wheel-0.29.0. Afterwards, I re-ran pip install spacy and it installed without any problems.

Successfully installed spacy-0.101.0

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.