Nuitka: ModuleNotFoundError for Pandas on standalone build

[jamesstidard]

I've tried compiling a standalone application with a pandas==0.23.4 dependency, which gives a ModuleNotFoundError: No module named 'pandas._libs.tslibs.np_datetime' exception on running.

This section of the pandas code does seem to also be cpython extensions / compiled *.so's. Also for reference, this is the line it raises on.

I'm unsure if nuitka should be auto descovering these kind of dependancies or not, I've seen similar things in pyinstaller. I did try adding variations of the argument --include-module=pandas._libs.tslibs.np_datetime to the build command, was unsuccessful. Though looking in the

This might just be user error, though building this as a compiled module did not seem to have this problem, as I mentioned in #164.

Here's the relevant code snippets:

0.6.0.3
Python: 3.7.0 (default, Sep  5 2018, 03:25:31) 
Executable: /usr/local/bin/python
OS: Linux
Arch: x86_64€

$ pip install nuitka==0.6.0.3 pandas==0.23.4`

# main.py
import pandas as pd
print(pd.Series([1, 2, 3]))

$ python -m nuitka main.py --standalone
$ main.dist/main
Traceback (most recent call last):
  File ".../main.dist/pandas/__init__.py", line 26, in <module pandas>
  File ".../main.dist/pandas/_libs/__init__.py", line 4, in <module pandas._libs>
  File "pandas/_libs/tslibs/conversion.pxd", line 11, in init pandas._libs.tslib
  File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
ModuleNotFoundError: No module named 'pandas._libs.tslibs.np_datetime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../main.dist/__main__.py", line 1, in <module>
  File ".../main.dist/playground/__init__.py", line 1, in <module playground>
  File ".../main.dist/playground/pandas_.py", line 1, in <module playground.pandas_>
  File ".../main.dist/pandas/__init__.py", line 32, in <module pandas>
ImportError: C extension: No module named 'pandas._libs.tslibs.np_datetime' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

[kayhayen]

Thanks for the detailed report, I agree this is about a hidden dependency. It's only newer Pandas that has this. I think I saw this one before, but was held back by #146 which fires for me even before I get to that point where I see the actual error. Let's see if this is better now.

Otherwise your report will be good enough to make a tentative fix that you can try out at least.

I will keep you updated.

[kayhayen]

So I managed to reproduce it and was not affected by the bug. I am adding hidden dependencies one by one. On factory in a cleanup, there is already one of them, but more will be needed. Due to the relatively large compile time, this will take a few cycles. I didn't want to add all the libraries, without seeing them used, but there are a lot, and potentially I will make a compilation for each one.

So as a side action, I am completing this, aiming at inclusion in 0.6.1 with some certainty.

[jamesstidard]

Thanks, @keyhayen.

Is this kind of library specific dependency handling going to look
something like pyinstaller's hook, if you're familiar with that, where the
community can keep a list of known hidden imports for a package. Here
is their pandas one.

Does that also mean that I could have potentially been able to solve this
if I was able to figure out the correct --include-module=*?

Thanks for the update,
James

On Sun, 14 Oct 2018 at 07:08, Kay Hayen [e-mail] wrote:

So I managed to reproduce it and was not affected by the bug. I am adding
hidden dependencies one by one. On factory in a cleanup, there is already
one of them, but more will be needed. Due to the relatively large compile
time, this will take a few cycles. I didn't want to add all the libraries,
without seeing them used, but there are a lot, and potentially I will make
a compilation for each one.

So as a side action, I am completing this, aiming at inclusion in 0.6.1
with some certainty.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Nuitka/Nuitka/issues/169#issuecomment-429599500, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABtvEgI-IWGszpNd9OF3sWPRCugtkrqEks5uktT4gaJpZM4XTy84
.

[holydrinker]

Hello guys, I use this thread because I'm copeing with problems that seems related to this issue.
I'm on ubuntu 16.04, I wrote my application in python 3.5 and I use lots of dependencies.

I'm trying to use nuitka to obtain my executable file. Dist and build directories are generated correctly, but when I run my main file, I faced with one error related to scikit-learn and scipy.

So I decided to write the most simple application to isolate my issue and reproduce the same error. I wrote this:

import sklearn
import scipy

print('hello world')

and I run nuitka with

python3 -m nuitka main.py --standalone

Build and dist directory are created correctly, but if I run dist/main, I obtain this error

Traceback (most recent call last):
  File ".../main.dist/main.py", line 1, in <module>
  File ".../main.dist/sklearn/__init__.py", line 134, in <module sklearn>
  File ".../main.dist/sklearn/base.py", line 11, in <module sklearn.base>
  File ".../main.dist/scipy/sparse/__init__.py", line 241, in <module scipy.sparse>
  File ".../main.dist/scipy/sparse/csgraph/__init__.py", line 165, in <module scipy.sparse.csgraph>
  File "_shortest_path.pyx", line 20, in init scipy.sparse.csgraph._shortest_path
ImportError: No module named 'scipy.sparse.csgraph._validation'

I tried to solve it including the scipy.sparse.csgraph._validation in nuitka manually with:

python3 -m nuitka --include-module scipy.sparse.csgraph._validation --standalone main.py

Then I run again dist/main and I faced with this error

Traceback (most recent call last):
  File ".../main.dist/main.py", line 1, in <module>
  File ".../main.dist/sklearn/__init__.py", line 134, in <module sklearn>
  File ".../main.dist/sklearn/base.py", line 13, in <module sklearn.base>
  File ".../main.dist/sklearn/utils/__init__.py", line 10, in <module sklearn.utils>
  File ".../main.dist/sklearn/utils/validation.py", line 18, in <module sklearn.utils.validation>
  File ".../main.dist/sklearn/utils/fixes.py", line 144, in <module sklearn.utils.fixes>
  File ".../main.dist/scipy/sparse/linalg/__init__.py", line 114, in <module scipy.sparse.linalg>
  File ".../main.dist/scipy/sparse/linalg/isolve/__init__.py", line 8, in <module scipy.sparse.linalg.isolve>
  File ".../main.dist/scipy/sparse/linalg/isolve/lgmres.py", line 10, in <module scipy.sparse.linalg.isolve.lgmres>
  File ".../main.dist/scipy/linalg/__init__.py", line 207, in <module scipy.linalg>
  File "_decomp_update.pyx", line 1, in init scipy.linalg._decomp_update
ImportError: No module named 'scipy.linalg.cython_blas'

Is this the right approach to solve my problems?
What can I do to make my project works with nuitka?

Thank you very much

[kayhayen]

So this is fixed on the factory branch right now, which is held back by non-critical issues. You might want to give it a try.

[kayhayen]

I need to add only a few of the libs as hidden dependencies to make that import work. Possibly the others are hidden dependencies of parts of pandas not used in the example, but I prefer to minimize them, so I wasn't adding all of them, in fact a lot less.

[kayhayen]

Part of the last pre-release, let me close this now, as it will soon be released.

[xennygrimmato]

@kayhayen When will this feature be released?

[kayhayen]

@xennygrimmato Nuitka 0.6.1 will be released when it's ready, largely driven by wishes for optimization features to be finished and included. The pre-release is usable however.

[oz123]

This is still an issue with 0.6.5. Maybe it's me being noob ... I did the following:

$ nuitka3 --version
0.6.5
Python: 3.6.5 (default, Jan 27 2019, 14:36:53) 
Executable: /usr/bin/python3.6
OS: Linux
Arch: x86_64
$ cat test.py 
import pandas as pd
print(pd.Series([1, 2, 3]))

Compile

nuitka3 --follow-imports --recurse-all  --show-progress test.py 
Nuitka:INFO:PASS 1:
Nuitka:INFO:Initial optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:Next global optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:PASS 2 ... :
Nuitka:INFO:Next global optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:Total memory usage before generating C code: 29.86 MB (31305728 bytes):
Nuitka:INFO:Total memory usage before running scons: 30.11 MB (31576064 bytes):
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc

Run the binary

$ ./test.bin 
Traceback (most recent call last):
  File "/home/oznn/Software/cloudpoint-vsphere-exporter-schema/test.py", line 1, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

It would be really great if this repository contained a few examples on how to compile non tivial python code with some popular libraries like opencv, PIL and numpy.

(The test suite seems awesome but it requires sometime to figure out ...)

Update

I found out that if install nuitka in my virtualenv and run it again
and run it in the following way:

python -m nuitka --follow-imports --recurse-all  --show-progress test.py

Now the situation is a bit better

$ ./test.bin
Traceback (most recent call last):
  File "/home/oznn/.cache/pypoetry/virtualenvs/poetry-demo-py3.6/lib64/python3.6/site-packages/pandas/__init__.py", line 30, in <module pandas>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
ImportError: cannot import name 'hashtable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/oznn/Software/cloudpoint-vsphere-exporter-schema/test.py", line 1, in <module>
    import pandas as pd
  File "/home/oznn/.cache/pypoetry/virtualenvs/poetry-demo-py3.6/lib64/python3.6/site-packages/pandas/__init__.py", line 34, in <module pandas>
    raise ImportError(
ImportError: C extension: 'hashtable' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

[mirhmousavi]

+1

[JorjMcKie]

Hi @oz123 - I know you to be a PyMuPDF stargazer since a long time ;-)

It would be really great if this repository contained a few examples on how to compile non tivial python code with some popular libraries like opencv, PIL and numpy.

I recommend you use "hinted compilation" for standalone builds. As pointed out there, any required standard plugins are automatically included, and you in addition should end up with a smaller dist folder after a shorter compilation time.
Let me repeat a few highlights here:

• Only suitable for standalone builds
• Consists of two separate steps:
  
  
  
  1. Run your script "under observation" of a logger script (get-hints.py), which records all imports actually executed.
  
  
  2. Run a modified compile script python nuitka-hints.py yourscript.py. As you see, this command line is very short. This is because the log file created in previous step is used to generated any additionally required parameters.