Cudf: [BUG] original error: libcuda.so.1: cannot open shared object file: No such file or directory

[stromal]

• My code looks like this

gdf = cudf.read_csv('09_01_02_2perc_400f_22102020.csv')

xgb_params = {
    'nround': 100,
    'max_depth': 4,
    'max_leaves': 2**4,
    'tree_method': 'gpu_hist',
    'n_gpus': 1,
    'loss': 'ls',
    'objective': 'gpu:reg:logistic',
    'max_features': 'auto',
    'criterion': 'friedman_mse',
    'grow_policy': 'lossguide',
    'verbose': True}

model = xgboost.train(xgb_params, dtrain=train)
y_hat = model.predict(train)
auc = roc_auc_score(y['survived'].to_pandas(), y_hat)
print(auc)

• I my code should be using cuda toolkit as it is the rapid install
• conda create -n rapids-0.16 -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.16 python=3.7 cudatoolkit=10.2
• I have cudatoolkit installed at conda list
  
  
  
  • output ... cudatoolkit 10.2.89 h6bb024c_0 nvidia...
  
  
• but when I run the import cudf it gives em the following error message:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-17-e13365c50bc4> in <module>()
----> 1 import cudf

/home/myusername/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cudf/__init__.py in <module>()
      2 from cudf.utils.gpu_utils import validate_setup  # isort:skip
      3 
----> 4 validate_setup()
      5 
      6 import cupy

/home/myusername/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cudf/utils/gpu_utils.py in validate_setup()
     16     import warnings
     17 
---> 18     from rmm._cuda.gpu import (
     19         CUDARuntimeError,
     20         cudaDeviceAttr,

/home/myusername/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/rmm/__init__.py in <module>()
     14 import weakref
     15 
---> 16 from rmm import mr
     17 from rmm._lib.device_buffer import DeviceBuffer
     18 from rmm.rmm import (

/home/myusername/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/rmm/mr.py in <module>()
      1 # Copyright (c) 2020, NVIDIA CORPORATION.
----> 2 from rmm._lib.memory_resource import (
      3     BinningMemoryResource,
      4     CudaMemoryResource,
      5     FixedSizeMemoryResource,

/home/myusername/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/rmm/_lib/__init__.py in <module>()
      1 # Copyright (c) 2019-2020, NVIDIA CORPORATION.
      2 
----> 3 from .device_buffer import DeviceBuffer

ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

After installing theses from here (https://docs.cupy.dev/en/latest/install.html)

python -m pip install -U setuptools pip
conda install -c anaconda cython
conda install -c conda-forge cupy

I got these after importing import cudf

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/__init__.py in <module>()
     20                                 message='can\'t resolve package from __spec__')
---> 21         from cupy import core  # NOQA
     22 except ImportError:

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/core/__init__.py in <module>()
----> 1 from cupy.core import core  # NOQA
      2 from cupy.core import internal  # NOQA
      3 

cupy/core/core.pyx in init cupy.core.core()

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/cuda/__init__.py in <module>()
      4 from cupy._environment import get_cuda_path  # NOQA
----> 5 from cupy.cuda import compiler  # NOQA
      6 from cupy.cuda import device  # NOQA

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/cuda/compiler.py in <module>()
     13 from cupy.cuda import device
---> 14 from cupy.cuda import function
     15 from cupy.cuda import nvrtc

cupy/cuda/function.pyx in init cupy.cuda.function()

cupy/cuda/texture.pyx in init cupy.cuda.texture()

ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-e13365c50bc4> in <module>()
----> 1 import cudf

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cudf/__init__.py in <module>()
      1 # Copyright (c) 2018-2019, NVIDIA CORPORATION.
      2 
----> 3 import cupy
      4 
      5 import rmm

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/__init__.py in <module>()
     40 original error: {}'''.format(exc_info[1]))  # NOQA
     41 
---> 42     six.reraise(ImportError, ImportError(msg), exc_info[2])
     43 
     44 

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/six.py in reraise(tp, value, tb)
    700                 value = tp()
    701             if value.__traceback__ is not tb:
--> 702                 raise value.with_traceback(tb)
    703             raise value
    704         finally:

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/__init__.py in <module>()
     19         warnings.filterwarnings('ignore', category=ImportWarning,
     20                                 message='can\'t resolve package from __spec__')
---> 21         from cupy import core  # NOQA
     22 except ImportError:
     23     # core is a c-extension module.

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/core/__init__.py in <module>()
----> 1 from cupy.core import core  # NOQA
      2 from cupy.core import internal  # NOQA
      3 
      4 
      5 # import class and function

cupy/core/core.pyx in init cupy.core.core()

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/cuda/__init__.py in <module>()
      3 
      4 from cupy._environment import get_cuda_path  # NOQA
----> 5 from cupy.cuda import compiler  # NOQA
      6 from cupy.cuda import device  # NOQA
      7 from cupy.cuda import driver  # NOQA

/home/username/anaconda3/envs/xgboost_env/lib/python3.7/site-packages/cupy/cuda/compiler.py in <module>()
     12 from cupy.cuda import _environment
     13 from cupy.cuda import device
---> 14 from cupy.cuda import function
     15 from cupy.cuda import nvrtc
     16 from cupy.cuda import runtime

cupy/cuda/function.pyx in init cupy.cuda.function()

cupy/cuda/texture.pyx in init cupy.cuda.texture()

ImportError: CuPy is not correctly installed.

If you are using wheel distribution (cupy-cudaXX), make sure that the version of CuPy you installed matches with the version of CUDA on your host.
Also, confirm that only one CuPy package is installed:
  $ pip freeze

If you are building CuPy from source, please check your environment, uninstall CuPy and reinstall it with:
  $ pip install cupy --no-cache-dir -vvvv

Check the Installation Guide for details:
  https://docs.cupy.dev/en/latest/install.html

original error: libcuda.so.1: cannot open shared object file: No such file or directory

Similar to - https://github.com/rapidsai/cudf/issues/441

[beckernick]

Can you confirm that you are actually using the newly created conda environment? Your traceback indicates you're using a different environment called xgboost_env: /home/username/anaconda3/envs/xgboost_env/lib/python3.7/...

[stromal]

@beckernick I am running it in xgboost_env conda environment I am not using conda create -n rapids-0.16 -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.16 python=3.7 cudatoolkit=10.2 because rapids-0.16 never gets created. Who can I just install all these staff with the previous defaults to my current xgboost_env conda environment?

[beckernick]

You can install the RAPIDS libraries into an existing environment by activating the environment and using conda install instead of conda create -n .... However, we recommend creating a fresh environment and including your other desired libraries.

If you are seeing a failure with the conda create... command, could you please include that error?

[stromal]

I have a fresh install of ubuntu 18.04 (previously 20.04 LTS)
conda create -n rapids16 -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.16 python=3.7 cudatoolkit=10.2
Now everything works

[beckernick]

Great 👍 Closing this issue for now. Please feel free to reopen it if you continue to see errors.