Pomegranate: Can't pip install because NeuralNetworkWrapper isn't Cython

[adamnovak]

Trying to pip install --local pomegranate on my Ubuntu 18.04 gives me a failure and a log that ends like this:

    building 'pomegranate.distributions.NeuralNetworkWrapper' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -I/home/anovak/.local/lib/python2.7/site-packages/numpy/core/include -c pomegranate/distributions/NeuralNetworkWrapper.c -o build/temp.linux-x86_64-2.7/pomegranate/distributions/NeuralNetworkWrapper.o
    x86_64-linux-gnu-gcc: error: pomegranate/distributions/NeuralNetworkWrapper.c: No such file or directory
    x86_64-linux-gnu-gcc: fatal error: no input files
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-kAZIbJ/pomegranate/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-8zaroy-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-kAZIbJ/pomegranate/

The new NeuralNetworkWrapper is pure Python, but setup.py still tries to build it, and that doesn't work. Either it needs to be made a .pyx, or the build process needs to skip over it when building stuff.

[adamnovak]

This seems to be related to trying to install without cython installed, in which case it tries to use a fixed list of .c files instead of *.pyx.

[adamnovak]

Installing with cython installed doesn't work either, because the tarball PyPI is shipping (https://files.pythonhosted.org/packages/ea/aa/1ee99e690884a11d1acefd8075f389bd8f8e25193fec1593e0e9e6519dfa/pomegranate-0.11.0.tar.gz) doesn't ship any .pyx files, just the C files. So when setup.py tries to use Cython on the pyx files, it doesn't work. That problem seems to be #554.

[adamnovak]

Actually this whole issue is really a duplicate of #554. Sorry!

[jmschrei]

I just added new mac and linux wheels to the repo, which should circumvent this issue. Making a new python3.6 virtual environment, I'm able to pip install pomegranate with no issues. Can you confirm?

[adamnovak]

Yep, I can confirm that pip install pomegranate in a 3.6 venv works
now, both with and without cython installed.

On 2/20/19, Jacob Schreiber [e-mail] wrote:

I just added new mac and linux wheels to the repo, which should circumvent
this issue. Making a new python3.6 virtual environment, I'm able to pip
install pomegranate with no issues. Can you confirm?

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/jmschrei/pomegranate/issues/555#issuecomment-465794042

[jmschrei]

Great! Thanks!

[caaespin]

I get this error, however my venv is python 3.8. Is there expected support for python 3.8 in the near future?

[carueda]

@jmschrei first off, thanks a lot for such an amazing lib (and associated documentation)!

I'm also hitting this issue with python 3.8 (on a mac, where the error looks like enclosed below in case it's useful).

For now, as part of my exploration of the library, I'm just creating and using a dockerized environment (centos7-based) and mapping files to be processed/generated to my host; all good so far, but far from ideal. Anyways, as already mentioned above, really just wondering about any plans to support 3.8. Thanks again.

---

$ python3.8 -m venv virtenv
$ source virtenv/bin/activate
(virtenv) ➜  pomegranate_installation  pip install pomegranate
Collecting pomegranate
  Using cached pomegranate-0.13.5.tar.gz (4.5 MB)
Collecting numpy>=1.8.0
  Using cached numpy-1.19.4-cp38-cp38-macosx_10_9_x86_64.whl (15.3 MB)
Collecting joblib>=0.9.0b4
  Downloading joblib-0.17.0-py3-none-any.whl (301 kB)
     |████████████████████████████████| 301 kB 1.9 MB/s
Collecting networkx>=2.0
  Using cached networkx-2.5-py3-none-any.whl (1.6 MB)
Collecting scipy>=0.17.0
  Using cached scipy-1.5.3-cp38-cp38-macosx_10_9_x86_64.whl (29.0 MB)
Processing /Users/carueda/Library/Caches/pip/wheels/13/90/db/290ab3a34f2ef0b5a0f89235dc2d40fea83e77de84ed2dc05c/PyYAML-5.3.1-cp38-cp38-macosx_10_13_x86_64.whl
Collecting decorator>=4.3.0
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Using legacy 'setup.py install' for pomegranate, since package 'wheel' is not installed.
Installing collected packages: numpy, joblib, decorator, networkx, scipy, pyyaml, pomegranate
    Running setup.py install for pomegranate ... error

...

    16 warnings generated.
    clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -L/usr/local/opt/llvm/lib -I/usr/local/opt/llvm/include build/temp.macosx-10.14-x86_64-3.8/pomegranate/distributions/JointProbabilityTable.o -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -o build/lib.macosx-10.14-x86_64-3.8/pomegranate/distributions/JointProbabilityTable.cpython-38-darwin.so
    ld: warning: directory not found for option '-L/usr/local/opt/llvm/lib'
    building 'pomegranate.distributions.NeuralNetworkWrapper' extension
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/opt/llvm/include -I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/Users/carueda/github/pomegranate_installation/virtenv/include -I/usr/local/Cellar/[email protected]/3.8.6_1/Frameworks/Python.framework/Versions/3.8/include/python3.8 -I/Users/carueda/github/pomegranate_installation/virtenv/lib/python3.8/site-packages/numpy/core/include -c pomegranate/distributions/NeuralNetworkWrapper.c -o build/temp.macosx-10.14-x86_64-3.8/pomegranate/distributions/NeuralNetworkWrapper.o
    clang: error: no such file or directory: 'pomegranate/distributions/NeuralNetworkWrapper.c'
    clang: error: no input files
    error: command 'clang' failed with exit status 1
    ----------------------------------------

[carueda]

@adamnovak @jmschrei I just hit this issue starting from scratch on a CentOS 8 box, with Python 3.6.8. Any comments welcome!

$ python3 --version
Python 3.6.8

$ python3 -m venv venv
$ source venv/bin/activate

$ pip install pomegranate
Collecting pomegranate
  Downloading https://files.pythonhosted.org/packages/67/a1/17396f2bf7e00054ed00fa0667c2a28b3d71a97d4569bacc959abb458212/pomegranate-0.13.5.tar.gz (4.5MB)
    100% |████████████████████████████████| 4.5MB 338kB/s
Requirement already satisfied: numpy>=1.8.0 in ./venv/lib/python3.6/site-packages (from pomegranate)
Requirement already satisfied: joblib>=0.9.0b4 in ./venv/lib/python3.6/site-packages (from pomegranate)
Requirement already satisfied: networkx>=2.0 in ./venv/lib/python3.6/site-packages (from pomegranate)
Requirement already satisfied: scipy>=0.17.0 in ./venv/lib/python3.6/site-packages (from pomegranate)
Requirement already satisfied: pyyaml in ./venv/lib/python3.6/site-packages (from pomegranate)
Requirement already satisfied: decorator>=4.3.0 in ./venv/lib/python3.6/site-packages (from networkx>=2.0->pomegranate)
Installing collected packages: pomegranate
  Running setup.py install for pomegranate ... error

...

    gcc -pthread -shared -Wl,-z,relro -Wl,-z,now -g build/temp.linux-x86_64-3.6/pomegranate/distributions/JointProbabilityTable.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/pomegranate/distributions/JointProbabilityTable.cpython-36m-x86_64-linux-gnu.so
    building 'pomegranate.distributions.NeuralNetworkWrapper' extension
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/u/carueda/soundscape/venv/include -I/usr/include/python3.6m -I/u/carueda/soundscape/venv/lib64/python3.6/site-packages/numpy/core/include -c pomegranate/distributions/NeuralNetworkWrapper.c -o build/temp.linux-x86_64-3.6/pomegranate/distributions/NeuralNetworkWrapper.o
    gcc: error: pomegranate/distributions/NeuralNetworkWrapper.c: No such file or directory
    gcc: fatal error: no input files
    compilation terminated.
    error: command 'gcc' failed with exit status 1

[adamnovak]

Does pomegranate==0.10 or maybe pomegranate==0.11 work? It could be a packaging issue in the current version.

Another workaround might be to have Cython installed before trying to install Pomegranate.

[carueda]

Yes, pomegranate==0.11 worked.

Then I rechecked about cython, which I assumed was in place because of the gcc steps during the build in my previous attempt with pomegranate. Seems like that was not the case. So, I also just tried pip-installing cython as well as (after removing it) a regular pomegranate install (with no explicit version) and finally I see a Successfully installed pomegranate-0.13.5.

Thanks @adamnovak!

[adamnovak]

Cython is needed to generate the c files; if there's no Cython it uses
pre-generated c files. That's why it fails: we have one component that
doesn't need Cython or a c file, but for some reason at install time,
if Cython isn't there, we try to build a pre-generated c file for it
anyway, and because there isn't one, we fail to install.

On 11/9/20, Carlos A. Rueda [e-mail] wrote:

Yes, pomegranate==0.11 worked.

Then I rechecked about cython, which I assumed was in place because of the
gcc steps during the build in my previous attempt with pomegranate. Seems
like that was not the case. So, I also just tried pip-installing cython as
well as (after removing it) a regular pomegranate install (with no explicit
version) and finally I see a Successfully installed pomegranate-0.13.5.

Thanks @adamnovak!

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/jmschrei/pomegranate/issues/555#issuecomment-724314659

--
Adam Novak (He/Him)
Senior Software Engineer
Computational Genomics Lab
UC Santa Cruz Genomics Institute
"Revealing life’s code."

Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

[mk44214]

I was getting the same issue..
I did pip3 install cython followed by pip3 install pomegranate ...

It worked.

[mhrvth]

I had the same issue on a Mac (Python 3.8), installed Cython as per @adamnovak 's advice, that solved the problem for me.