Reticulate: Problem using sklearn with reticulate using MKL and Python3.7: high CPU usage, hangs indefinitely

[scottgigante]

I'm having an issue with reticulate since I upgraded to python 3.7. When I run sklearn.utils.extmath.randomized_svd, it uses all 36 available cores and runs for >24h. Running the same thing from a python or IPython CLI completes in less than a second.

> reticulate::repl_python()
Python 3.7.0 (/usr/bin/python)
Reticulate 1.10.0.9001 REPL -- A Python interpreter in R.
>>> r['tree.data']['data'].shape
(3000, 100)
>>> from sklearn.utils.extmath import randomized_svd
>>> randomized_svd(r['tree.data']['data'], n_components=10)

Python package versions:

>>> sklearn.__version__
'0.19.2'
>>> numpy.__version__
'1.15.0'
>>> scipy.__version__
'1.1.0'

R and platform info:

> R.version
               _
platform       x86_64-pc-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          3
minor          5.1
year           2018
month          07
day            02
svn rev        74947
language       R
version.string R version 3.5.1 (2018-07-02)
nickname       Feather Spray

[scottgigante]

Further update to this issue: it's caused by the interaction between reticulate and MKL.

If I run this standard MKL environment variable setup before imports, it hangs on randomized_svd as above:

> Sys.setenv(MKL_NUM_THREADS="12")
> Sys.setenv(MKL_DYNAMIC="FALSE")
> Sys.setenv(OMP_NUM_THREADS=1)

If I set the number of MKL threads to 1, then randomized_svd completes in less than a second.

> Sys.setenv(MKL_NUM_THREADS="1")
> Sys.setenv(MKL_DYNAMIC="FALSE")
> Sys.setenv(OMP_NUM_THREADS=1)

[jan-glx]

Are you using conda? Are you using Rstudio?

[scottgigante]

Conda no, RStudio yes. I just checked and it doesn't happen outside of RStudio.

[jan-glx]

R studio server?

[scottgigante]

Ah, yes - didn't clarify, sorry.

[jan-glx]

Rstudioserver is somehow messing with LD_LIBRARY_PATH. It can be modified using rsession-ld-library-path in /etc/rstudio/rserver.conf You could inspect a running (your randomized_svd) normal R session and compare that to the RStudioserver one using lsof -p NNNN | awk '{print $9}' | grep '\.so' and replacing NNNN by the Sys.getpid() to see if different libraries are loaded. Good luck!

[scottgigante]

Bingo! The session inside RStudio-server doesn't have MKL libraries attached.

$ diff <(sort r_ldconf.txt) <(sort rstudio_ldconf.txt)
1,3c1
< /opt/intel/compilers_and_libraries_2018.3.222/linux/mkl/lib/intel64_lin/libmkl_core.so
< /opt/intel/compilers_and_libraries_2018.3.222/linux/mkl/lib/intel64_lin/libmkl_sequential.so
< /usr/lib/gconv/ISO8859-1.so
---
> /home/scottgigante/R/x86_64-pc-linux-gnu-library/3.5/yaml/libs/yaml.so
7a6
> /usr/lib/libcrypto.so.1.0.0
23a23
> /usr/lib/libnss_files-2.28.so
32a33,34
> /usr/lib/libutil-2.28.so
> /usr/lib/libuuid.so.1.3.0
38a41
> /usr/lib/R/library/tools/libs/tools.so

I added the following line to /etc/rstudio/rserver.conf

rsession-ld-library-path=/opt/intel/compilers_and_libraries_2018.3.222/linux/mkl/lib/intel64_lin/:/usr/lib/gconv/

and after a restart, it works! Thanks so much @jan-glx

[eddelbuettel]

Glad you have it working. For completeness, you _can_ install MKL on a Debian/Ubuntu box without extra hoops so that these become the automatically chose default BLAS/LAPACK libraries -- see this repo for a script and my blog for some more.

[scottgigante]

Not sure what's changed, but this issue has reappeared. That line is still in my /etc/rstudio/rserver.conf, and Sys.getenv("LD_LIBRARY_PATH") shows those folders are indeed on the path, but lsof shows the same results as prior to the fix.

[scottgigante]

I guess at this point this is not an issue with reticulate - maybe @jan-glx you have an idea of where best to find further help with this.

[jan-glx]

Not really. Maybe the libraries have been updated?
Replacing the ld library path modification with Dirk's solution would prevent this from causing problems.

If it still occurs only in Rstudio R sessions but not in other R sessions (using the same R library sites & after restarting the server), you could try their issue tracker/support.

An very alternative workaround for you might be to use irlba (from the same named r package) instead of randomized svd. In case you only need a few components, this is super fast.

[scottgigante]

Unfortunately I'm on Arch Linux, so no go there. sklearn is actually just a dependency of my python workflow, so no luck there either. I'll check the Rstudio issue tracker. Thanks for the ideas!