Spacy: npy_endian.h cannot detect the correct processor endianness/byte order for ppc64le

[sdmonov]

My Environment

• Operating System: Ubuntu 16.04.3 LTS on Power8
• Python Version Used: Python 3.6.2 |Anaconda, Inc.| (default, Sep 15 2017, 20:38:23) [GCC 4.8.4] on linux
• spaCy Version Used: Master from 11/16/2017 (commit a3d4dd1)
• System Information: PowerNV 8335-GTB with POWER8NVL CPU

Numpy headers included in spaCy come from older version of numpy and cannot detect the byte order for ppc64le CPU correctly. If I replace the spaCy/include/numpy/ with the latest headers, spaCy works fine on ppc64le.

[honnibal]

Pull request? Sounds good to me.

We're going to be flying blind on this one because I don't think we'll manage to get a test setup for that architecture. So if you could submit a patch that works for you and builds on the CI servers, that'd be great.

Also: I notice you're using Anaconda. Do you install via our conda wheels? If so, has that worked for you? I think I compiled thinc with --march=native for those wheels, which would likely be a problem.

Edit: Just saw your other thread...

[sdmonov]

I've been thinking of two approaches to fix this one:

1. Replace the numpy headers with the latest from numpy github
2. Detect the current numpy install headers and include them in the compiler include path in setup.py

I will be very happy to submit a pull request for this one but the issue though is that my company is still working on accepting the CLA and for this moment I'm not able to contribute code to the project. I would be very happy to test any new code on Power8 machine though.

Edit: We could not find any wheels from Anaconda available for ppc64le platform.

Thanks!

[honnibal]

Trying to fix this: https://github.com/explosion/spaCy/pull/1842

[jateeq]

@honnibal I'm experiencing the same issue on a machine with similar specs as @sdmonov, except I'm using the master version of SpaCy. I see there's already a pull request with suggested changes, would you have an estimate of when it would be merged? I thought I'd try the feature/update-numpy-headers branch and follow install from source instructions but the error persists.

$ python -m spacy download en

Traceback (most recent call last):
  File "__init__.pxd", line 1011, in numpy.import_array
RuntimeError: FATAL: module compiled as big endian, but detected different endianness at runtime

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jateeq/anaconda3_minsky/envs/allennlp/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/jateeq/anaconda3_minsky/envs/allennlp/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/jateeq/anaconda3_minsky/envs/allennlp/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/jateeq/spaCy/spacy/__init__.py", line 4, in <module>
    from .cli.info import info as cli_info
  File "/home/jateeq/spaCy/spacy/cli/__init__.py", line 6, in <module>
    from .train import train
  File "/home/jateeq/spaCy/spacy/cli/train.py", line 12, in <module>
    from ..gold import GoldCorpus, minibatch
  File "morphology.pxd", line 25, in init spacy.gold
  File "vocab.pxd", line 27, in init spacy.morphology
  File "tokens/doc.pxd", line 28, in init spacy.vocab
  File "/home/jateeq/spaCy/spacy/tokens/__init__.py", line 1, in <module>
    from .doc import Doc
  File "token.pxd", line 11, in init spacy.tokens.doc
  File "token.pyx", line 10, in init spacy.tokens.token
  File "__init__.pxd", line 1013, in numpy.import_array
ImportError: numpy.core.multiarray failed to import

Thank you!

[sdmonov]

@jateeq, to workaround this issue I just create a symlink to the installed numpy package include dir for now:

#from spaCy source root directory
cd include
mv numpy numpy_old

#replace the path of the python lib directory where numpy is installed with yours
ln -s  ~/miniconda3/envs/allennlp/lib/python3.6/site-packages/numpy/core/include/numpy ./

This works fine for me as a temporary solution.

[jateeq]

@sdmonov Awesome that worked, thank you!

[nehaljwani]

spaCy is available as a Conda package for ppc64le in defaults now. conda install spacy

[Sankalikhitha]

.env) C:UsersLotusDesktopChatBot_Rasa.envScripts> python -m spacy download en
Traceback (most recent call last):
File "C:Python36librunpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "C:Python36librunpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "C:Python36librunpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "C:UsersLotusDesktopChatBot_Rasa.envlibsite-packagesspacy__init__.py", line 10, in
from .cli.info import info as cli_info
File "C:UsersLotusDesktopChatBot_Rasa.envlibsite-packagesspacycli__init__.py", line 6, in
from .train import train
File "C:UsersLotusDesktopChatBot_Rasa.envlibsite-packagesspacyclitrain.py", line 12, in
from ..gold import GoldCorpus, minibatch
File "strings.pxd", line 23, in init spacy.gold
ImportError: DLL load failed: The specified module could not be found.

Getting above error.Could anyone please resolve the above error?

[honnibal]

@Sankalikhitha I believe that issue is probably unrelated.

@jateeq Sorry for not getting to this sooner. My PR had problems I never figured out, and I don't have access to a machine to test this. If someone could make a working PR, I'd happily merge it.

In the meantime, the workarounds and the conda-forge solution suggested both seem like good approaches.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.