Nuitka: Numpy (from Conda) - Missing libmkl_*.so in Standalone

[ohlr]

OS: Ubuntu 16.04
python=3.7.3
nuitka==0.6.4
conda 4.6.14

conda install numpy

main.py

import numpy as np

use

python -m nuitka --follow-imports --standalone main.py

Things I tried:
--plugin-enable=numpy
--include-plugin-directory=/home/user/anaconda3/envs/env/lib

Error when running the standalone:
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.

Manually copying all libmkl_*.so files resolves the issue

Related issue: 190

[JorjMcKie]

I do not have that numpy version installed in my Linux.
Can you please tell me the folder in which those .so files live in your numpy version?
Just need to add this location to the numpy plugin.

[ohlr]

For example:
/home/user/anaconda3/envs/myenv/lib/libmkl_core.so

[JorjMcKie]

so this is not obviously related to the Pthon numpy installation?
Ugly!

[ohlr]

When installing numpy with conda I get mkl installed with it.
These are the dependencies that conda installs when installing numpy:

- blas=1.0=mkl
- intel-openmp=2019.4=243
- libgfortran-ng=7.3.0=hdf63c60_0
- mkl=2019.4=243
- mkl_fft=1.0.12=py37ha843d7b_0
- mkl_random=1.0.2=py37hd81dba3_0
- numpy=1.16.4=py37h7e9f1db_0
- numpy-base=1.16.4=py37hde5b4d6_0

[JorjMcKie]

Of course I'm looking for logic to algorithmically locate them. But obviously any folder name containing things like "myenv" cannot be detected (otherwise Nuitka would have done it already).

Or I would need to somehow dive into some numpy config data to spot those places. So far, the numpy plugin only looks for some standard sub-folders (.lib, _lib and so on) of numpy, numpy/core and even Python itself ..., but not arbitrary other places like yours.

If you have information at hand how your numpy would know where to look, that would be helpful.

[ohlr]

The directory structure looks like this:

myenv
-lib
--python3.7 #that is the active python
---/site-packages/numpy
--libmkl_core.so
-bin
-share
...

Maybe this helps
which python yields:~/anaconda3/envs/myenv/bin/python

[JorjMcKie]

Did you see a message NUITKA:Warning:Additional MKL libraries found. in your compile log?
That would mean that those files have already been detected. Only copying them to appropriate places in the .dist folder would need to be added ...

[kayhayen]

@JorjMcKie The Anaconda is a non-system Python installation, where everything is included. They are crazy enough to package gcc, and what not. It is very popular among scientist, because, well you get everything included, and they do test if packages work together, and try to provide a working set.

For Anaconda, but not unlike other Pythons, we need to use the standard methods, searching in sys.path, sys.prefix, etc. to locate things, so that is easy.

Installation of Anaconda is very easy, and a good way to get packages without having to resort to a virtualenv, although they do that too. Sometimes they do things slightly different. I have e.g. noticed that the PyQt plugins folders were different on Windows, but never really that much of a difference.

In terms of performance, Anaconda allows static linking (which I need to peruse more), and gives hope for much better performance therefore. Also, generally, in my testing, it was always faster than CPython or so I remember at least.

Yours,
Kay

[kayhayen]

So well yes, @JorjMcKie from a system admin standpoint, horrible, ugly,but worth it. :)

[ohlr]

@JorjMcKie Indeed it's there:

Nuitka:WARNING:Unresolved '__import__' call at '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/core/function_base.py:453' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/lib/utils.py:366' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/lib/utils.py:865' may require use of '--include-plugin-directory' or '--include-plugin-files'.
Nuitka:WARNING:Unresolved '__import__' call at '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/lib/utils.py:923' may require use of '--include-plugin-directory' or '--include-plugin-files'.

Nuitka:INFO:Total memory usage before running scons: 363.43 MB (381087744 bytes):
Nuitka:INFO: Additional MKL libraries found.
Nuitka:INFO: Not copying MKL binaries in '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/.libs' for numpy!

Nuitka:INFO:Colliding DLL names for libncursesw.so.6, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libncursesw.so.6' <-> '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libncursesw.so.6'.
Nuitka:INFO:Colliding DLL names for libtinfow.so.6, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libtinfow.so.6' <-> '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../././libtinfow.so.6'.
Nuitka:INFO:Colliding DLL names for libcrypto.so.1.1, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libcrypto.so.1.1' <-> '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libcrypto.so.1.1'.
Nuitka:INFO:Colliding DLL names for libz.so.1, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libz.so.1' <-> '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libz.so.1'.
Nuitka:INFO:Colliding DLL names for libmkl_rt.so, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/site-packages/mkl_fft/../../../libmkl_rt.so' <-> '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/../../../libmkl_rt.so'.
Nuitka:INFO:Colliding DLL names for libmkl_rt.so, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/site-packages/mkl_fft/../../../libmkl_rt.so' <-> '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/core/../../../../libmkl_rt.so'.
Nuitka:INFO:Colliding DLL names for libmkl_rt.so, checking identity of '~/anaconda3/envs/myenv/lib/python3.7/site-packages/mkl_fft/../../../libmkl_rt.so' <-> '~/anaconda3/envs/myenv/lib/python3.7/site-packages/numpy/linalg/../../../../libmkl_rt.so'.
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/bin/../lib/libpython3.7m.so.1.0' (used by main.dist/main).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libffi.so.6' (used by main.dist/_ctypes.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libncursesw.so.6' (used by main.dist/_curses.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libtinfow.so.6' (used by main.dist/_curses.so, main.dist/_curses_panel.so, main.dist/readline.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libpanelw.so.6' (used by main.dist/_curses_panel.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libcrypto.so.1.1' (used by main.dist/_hashlib.so, main.dist/_ssl.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../liblzma.so.5' (used by main.dist/_lzma.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libsqlite3.so.0' (used by main.dist/_sqlite3.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libssl.so.1.1' (used by main.dist/_ssl.so).
Nuitka:INFO:Included used shared library '/usr/lib/x86_64-linux-gnu/libXdmcp.so.6' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libtk8.6.so' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '/usr/lib/x86_64-linux-gnu/libXau.so.6' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../.././libz.so.1' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libtcl8.6.so' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '/usr/lib/x86_64-linux-gnu/libxcb.so.1' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '/usr/lib/x86_64-linux-gnu/libX11.so.6' (used by main.dist/_tkinter.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/site-packages/mkl_fft/../../../libmkl_rt.so' (used by main.dist/mkl_fft/_pydfti.so).
Nuitka:INFO:Included used shared library '~/anaconda3/envs/myenv/lib/python3.7/lib-dynload/../../libreadline.so.7' (used by main.dist/readline.so).

[JorjMcKie]

hinting.zip

Here is a version of components which I tested with the following results:

• vanilla PIP numpy Linux and Windows (only OpenBlas shared lib is used)
• MKL libraries contained in numpy.core (Christoph Gohlke's wheel for Windows)
• numpy+MKL Anaconda distro for Linux.
  Does not work in the Windows case, still trying to figure out a trick here. The shared libraries (Win DLLs) are not found, no matter where I put them ...

So you can give it a try with your Linux environment.
The hack you need to do is copying the contained plugin folder over to your nutika installation and then use the hinting scripts for compilation.

[JorjMcKie]

Update for my previous post:

* vanilla PIP numpy Linux and Windows (only OpenBlas shared lib is used)

* MKL libraries included in `numpy.core`

* numpy+MKL Anaconda distro for Linux and Windows

So the new numpy plugin should work in any of the known cases ...

[ohlr]

I only copied the plugin folder and it works - Good work

If you need any further information/testing from my side let me know

[JorjMcKie]

Thanks for the feedback and glad it does its job now.

[JorjMcKie]

So well yes, @JorjMcKie from a system admin standpoint, horrible, ugly,but worth it. :)

As I said: I did not grasp Anaconda's installation scheme at first sight. But of course there is one, and it is not more complicated than any other.

[JorjMcKie]

I think the recent Nuitka versions already contains this fix (0.6.4 or 0.6.5).