Cuml: [BUG] pip installation has ImportError: libcuml.so

[tlkh]

Describe the bug
A clear and concise description of what the bug is.

When following the instructions to install cuML using pip, unable to import cuML due to the below error.

>> import cuml
ImportError: libcuml.so: cannot open shared object file: No such file or directory

Steps/Code to reproduce bug
Follow this guide http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.

Steps used to install and import:

# apt install libopenblas-base libomp-dev
# pip install cuml-cuda100
# python3
>> import cuml
ImportError: libcuml.so: cannot open shared object file: No such file or directory

Tried running ldconfig, but it makes no difference.

Expected behavior
A clear and concise description of what you expected to happen.

Expect cuML to be able to import. cuDF, cuGraph, cuStrings all work fine.

Environment details (please complete the following information):

• Environment location: custom docker (able to replicate inside Kaggle kernel for testing)
• Linux Distro/Architecture: Ubuntu 16.04 amd64 and Ubuntu 18.04 amd64
• GPU Model/Driver: [V100 and driver 396.44] P100 (410), GTX1080Ti (410 and 418),
• CUDA: 10.0
• Method of cuDF & cuML install: pip

Additional context
Add any other context about the problem here.

[jeng1220]

Additional context
I encounter same issue.
The cuML version is cuml-cuda100 (0.6.0)
Python version is 3.6.7

[WDot]

I have the same issue, same information as the above posters.

[edwin-yan]

I have the same issue. The cuML version is cuml-cuda100 (0.6.1)

[AK-ayush]

In order to solve this issue, you can update the LD_LIBRARY_PATH with location of libcuml.so after installation using pip.

• find / -name libcuml.so <- it will print the absolute path of libcuml.so.
  /path/to/libcuml.so

• now update the LD_LIBRARY_PATH using:
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to

After that try import cuml, hopefully it should work...

[edwin-yan]

still the same issue. I have sourced bashrc and made sure $LD_LIBRARY_PATH reflects the changes in virtualenv before i import cuml.

PS: cuml is the only package I have issue. cudf works perfectly (also installed through pip)

!echo $LD_LIBRARY_PATH
:/usr/local/cuda-10.0/lib64:/root/.virtualenvs/RAPIDS/lib/python3.6/site-packages/libcuml.so

import cuml

ImportError Traceback (most recent call last)
in
----> 1 import cuml

~/.virtualenvs/RAPIDS/lib/python3.6/site-packages/cuml/__init__.py in
15 #
16
---> 17 from cuml.cluster.dbscan import DBSCAN
18 from cuml.cluster.kmeans import KMeans
19

~/.virtualenvs/RAPIDS/lib/python3.6/site-packages/cuml/cluster/__init__.py in
15 #
16
---> 17 from cuml.cluster.dbscan import DBSCAN
18 from cuml.cluster.kmeans import KMeans

ImportError: libcuml.so: cannot open shared object file: No such file or directory

[AK-ayush]

@Bluesn0w set the LD_LIBRARY_PATH as follows:

export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:/root/.virtualenvs/RAPIDS/lib/python3.6/site-packages/
do not include the libcuml.so in LD_LIBRARY_PATH

[edwin-yan]

Thanks @AK-ayush. I can confirm it works. In that case, will you suggest me to copy this file to somewhere so it won't just associate with 1 specific virtualenv?

I am also wondering why this needs to be set at LD_LIBRARY_PATH? cudf uses some libs, eg librmm.so, why it does not need to set in environment variable?

Thanks!

[AK-ayush]

Hi @Bluesn0w
Answer to your first question:
I wouldn't recommend to copy this file somewhere central location because there may be different versions in different virtualenvs, in that case it will cause issue.

and the answer to second question:
In @datametrician 's word- pip has been a nightmare for our build infrastructure and until pip sets new standards we would rather not fight with build and focus on library dev

[edwin-yan]

@AK-ayush , Thank you!

[tlkh]

I think we should place this under the installation instructions for pip - no doubt many people will go that route and face issues.

[anirbankonar123]

Has anybody installed this in Google colab pls ? I am facing same error in Google colab env. Tried setting LD_LIBRARY_PATH with path to libcuml.so, didnt work

[AK-ayush]

@anirbankonar123 please have a look at this blog, which contains detailed steps.

[anirbankonar123]

Excellent , thanks a lot

[AK-ayush]

Excellent , thanks a lot

glad it helped.