Cudf: Cannot even import cudf (librmm.so error)

[aguinaldoabbj]

• [ ] I am using the latest version of cuDF from conda, built from master, or
  built from the latest tagged release.
• [ ] I have included the following environment details:
  Linux Distro, Linux Kernel, GPU Model

Linux: Ubuntu 18.04
kernel 4.15.0
miniconda3
python 3.6.6
cudf 0.3.0
gpu GTX 1050 Ti

Problems on getting started to rapids.ai.

Error on loading cudf:

What have I missed?

`---------------------------------------------------------------------------
OSError Traceback (most recent call last)
in
----> 1 import cudf

~/miniconda3/lib/python3.6/site-packages/cudf/__init__.py in
1 # Copyright (c) 2018, NVIDIA CORPORATION.
----> 2 from cudf import dataframe # noqa: F401
3
4 from cudf.dataframe import DataFrame # noqa: F401
5 from cudf.dataframe import Index # noqa: F401

~/miniconda3/lib/python3.6/site-packages/cudf/dataframe/__init__.py in
----> 1 from cudf.dataframe import (buffer, dataframe, series, # noqa: F401
2 index, numerical, datetime) # noqa: F401
3
4 from cudf.dataframe.dataframe import DataFrame # noqa: F401
5 from cudf.dataframe.index import (Index, GenericIndex, # noqa: F401

~/miniconda3/lib/python3.6/site-packages/cudf/dataframe/buffer.py in
1 import numpy as np
2
----> 3 from librmm_cffi import librmm as rmm
4
5 from cudf.utils import cudautils, utils

~/miniconda3/lib/python3.6/site-packages/librmm_cffi/__init__.py in
43 return path
44
---> 45 librmm_api = ffi.dlopen(_get_lib_name())
46 librmm = _RMMWrapper(ffi, librmm_api)
47

OSError: cannot load library 'librmm.so': libcudart.so.9.2: cannot open shared object file: No such file or directory`

[harrism]

@aguinaldoabbj Are you building from source or using the container?

This looks like a duplicate of #431

[aguinaldoabbj]

I have just installed cudf from conda, according to the documentation, by runing
$ conda install -c numba -c conda-forge -c nvidia -c rapidsai -c defaults cudf=0.3.0

[Msabih]

Can you check in your anaconda/lib if librmm is there ? I had the same issue and I got over it by explicilitly adding thr anaconda/lib path to LD_LIBRARY_PATH

[aguinaldoabbj]

LD_LIBRARY_PATH

Thanks for the help. librmm.so is in /miniconda3/lib/ . I've exported conda's lib path as you advised, by running `export LD_LIBRARY_PATH=/path/to/miniconda3/lib/' but nothing changed (I've run 'env' to see if LD_LIBRARY_PATH was correctly set. Any other idea?

[AK-ayush]

@aguinaldoabbj I can see that you are using pretty old gpu card, while minimum recommended card is pascal arch. And what version of cuda have you installed?

[harrism]

If you conda install cffi does it fix the issue? We have a PR in progress that add's the cffi package to the Dockerfile and it fixes the "can't import librmm" issues we have seen.

[aguinaldoabbj]

@aguinaldoabbj I can see that you are using pretty old gpu card, while minimum recommended card is pascal arch. And what version of cuda have you installed?

Dear,
GTX 1050 Ti isn't that old. It's the current nvidia's pos entry-level card, still in production and sale today. It is also compliant with Pascal architecture as the whole GTX 10 family. I've installed Cuda 9.2 from conda.

[AK-ayush]

What is your LD_LIBRARY_PATH? Since same installation process is working for me. LD_LIBRARY_PATH for my setup is: LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64:/usr/local/cuda-9.2/extras/CUPTI/lib64:/usr/local/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

And the file which is giving error libcudart.so.9.2 should be there in /usr/local/cuda-9.2/lib64/libcudart.so.9.2. Can you verify this please?

[aguinaldoabbj]

If you conda install cffi does it fix the issue? We have a PR in progress that add's the cffi package to the Dockerfile and it fixes the "can't import librmm" issues we have seen.

I've noticed cffi is already installed in my conda.

[aguinaldoabbj]

What is your LD_LIBRARY_PATH? Since same installation process is working for me. LD_LIBRARY_PATH for my setup is: LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64:/usr/local/cuda-9.2/extras/CUPTI/lib64:/usr/local/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

And the file which is giving error libcudart.so.9.2 should be there in /usr/local/cuda-9.2/lib64/libcudart.so.9.2. Can you verify this please?

Is your installation system-wide? I've installed all cuda stuff through conda. Indeed, libcudart.so.9.2 isn't in my system.

[AK-ayush]

Yes, my installation is system-wide.

Can you find this file libcudart.so.9.2 in your conda env using
find / -name libcudart.so.9.2?
Location of this file should be there in LD_LIBRARY_PATH.
I guess this is the file causing the import issue in your case.

[aguinaldoabbj]

Yes, my installation is system-wide.

Can you find this file libcudart.so.9.2 in your conda env using
find / -name libcudart.so.9.2?
Location of this file should be there in LD_LIBRARY_PATH.
I guess this is the file causing the import issue in your case.

@AK-ayush Thanks for the help. I have just discovered I must install "cudatoolkit" package in conda. I had installed "cuda92" package only. It is not clear in the docs.

Now i'm able to start things properly.

Thank you all for the help.

[AK-ayush]

Glad It worked! I'll suggest changes in doc.

[MurrayData]

Installing cudatoolkit with conda worked for me too. Now running on latest version. Thanks @AK-ayush @aguinaldoabbj

[harrism]

@aguinaldoabbj Are you unable to install the native CUDA toolkit on your system rather than installing via conda? We recommend the native toolkit (https://developer.nvidia.com/cuda-toolkit) because the conda installation often lags behind (e.g. no CUDA 10 support).

[aguinaldoabbj]

@aguinaldoabbj Are you unable to install the native CUDA toolkit on your system rather than installing via conda? We recommend the native toolkit (https://developer.nvidia.com/cuda-toolkit) because the conda installation often lags behind (e.g. no CUDA 10 support).

No, I just believe it is better to maintain things with conda, envs and stuff like that.

[kkraus14]

@aguinaldoabbj Are you unable to install the native CUDA toolkit on your system rather than installing via conda? We recommend the native toolkit (https://developer.nvidia.com/cuda-toolkit) because the conda installation often lags behind (e.g. no CUDA 10 support).

No, I just believe it is better to maintain things with conda, envs and stuff like that.

You can still install it into a userspace and set the relevant PATHs to it. See using toolkitpath here: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-advanced.

[kkraus14]

Closing as no activity and issue was resolved.

[jangorecki]

Could we re-open this issue. It seems to be resolved by manually post-fixing instructions that are in documentation (using steps described in comments).
I run into same issue, but not in a container. Could we include required fixes in readme in installation instructions? so it is clear how to approach error:

OSError: cannot load library 'librmm.so': libcudart.so.9.2: cannot open shared object file: No such file or directory

Precisely speaking, whatever is necessary but haven't been mentioned in installation instructions, like extra software to install, source from where to install that software, what to put to LD_LIBRARY_PATH, etc.

[harrism]

@jangorecki can you try the 0.5 release and let us know if you are still having issues like this?

[manojkumardas7]

@harrism
I have cuda toolkit 10.

here are my environment entries
export PATH=/usr/local/cuda-10.0/bin${PATH:+:$PATH}}
export PATH=/usr/local/MATLAB/bin${PATH:+:$PATH}}
export CUDADIR=/usr/local/cuda-10.0
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export LD_LIBRARY_PATH=/usr/local/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export NUMBAPRO_NVVM=/usr/local/cuda-10.0/nvvm/lib64/libnvvm.so
export NUMBAPRO_LIBDEVICE=/usr/local/cuda-10.0/nvvm/libdevice

1. nvcc -V works
2. deviceQuery from cuda examples test has PASSED
3. Python version is 3.7
4. OS is Ubuntu 18.04.2 LTS

for cudf-cuda100==0.6, I get the following error on import
OSError: cannot load library 'librmm.so': librmm.so: cannot open shared object file: No such file or directory

for cudf-cuda100==0.5, import is succesful but on creating a dataframe:
cudf.DataFrame({'x': np.random.rand(100)})
below is returned
GDFError: CUDA ERROR. cudaSuccess: no error

[mazzma12]

Yes, my installation is system-wide.
Can you find this file libcudart.so.9.2 in your conda env using
find / -name libcudart.so.9.2?
Location of this file should be there in LD_LIBRARY_PATH.
I guess this is the file causing the import issue in your case.

@AK-ayush Thanks for the help. I have just discovered I must install "cudatoolkit" package in conda. I had installed "cuda92" package only. It is not clear in the docs.

Now i'm able to start things properly.

Thank you all for the help.

Faced this issue today with current install on python 3.7 CUDA 10.1. This solved the issue

[bhuiyanmobasshir94]

https://github.com/rapidsai/cudf/issues/441#issuecomment-491269101

I tried the same approach and the same python and Cuda version yet getting the same import error issue. Any help?

[AK-ayush]

@bhuiyanmobasshir94 How are you installing the rapids? I assume you are using conda.

[bhuiyanmobasshir94]

@AK-ayush yes I have installed rapids through conda. I have reinstalled it and its working now no need to setup the path though. It's working fine through python console but getting error while using jupyter notebook. What can be the issue here?

I really appreciate your first response.

[geometrylearner]

I have just fixed it by re-install the cudatoolkit=9.2 in conda, the default version of cudatoolkit>=10.xx which is not matched the cudf. I hope it's useful.