Nvidia-docker: libcuda not found

[stefanseefeld]

I'm trying out various images (nvidia/cuda, nvidia/cuda:7.5-devel-centos7, etc.), but can't find libcuda.so* (et al.) in any of them. Is this an oversight, or am I missing something ?

Thanks,

[3XX0]

libcuda is tightly coupled with the driver, it is not shipped as part of the image but mounted at runtime.
If you have a build dependency on it, use the stub file under /usr/local/cuda/lib64/stubs

[stefanseefeld]

How is this mounting supposed to work, i.e. where is the library supposed to be found ? (Or should I add a particular -v option to do the mapping ?

I actually have a runtime dependency for code that expects libcuda.so to be found in the usual places.

Thanks,

[flx42]

You need to use the nvidia-docker wrapper it will take care of mounting the volume and the devices when starting a container.

[stefanseefeld]

I do, yet I don't see libcuda.so anywhere.

[flx42]

$ nvidia-docker run -ti nvidia/cuda
root@4ca5ff31f1dc:/# echo $LD_LIBRARY_PATH
/usr/local/nvidia/lib:/usr/local/nvidia/lib64:
root@4ca5ff31f1dc:/# ls /usr/local/nvidia/lib64
libEGL.so.1               libGLESv1_CM_nvidia.so.361.48  libGLX_nvidia.so.361.48  libnvidia-compiler.so.361.48         libnvidia-glcore.so.361.48  libnvidia-opencl.so.361.48
libEGL_nvidia.so.0        libGLESv2.so.2                 libGLdispatch.so.0       libnvidia-eglcore.so.361.48          libnvidia-glsi.so.361.48    libnvidia-ptxjitcompiler.so.361.48
libEGL_nvidia.so.361.48   libGLESv2_nvidia.so.2          libOpenGL.so.0           libnvidia-encode.so.1                libnvidia-ifr.so.1          libnvidia-tls.so.361.48
libGL.so.1                libGLESv2_nvidia.so.361.48     libcuda.so.1             libnvidia-encode.so.361.48           libnvidia-ifr.so.361.48     libvdpau_nvidia.so.1
libGL.so.1.0.0            libGLX.so.0                    libcuda.so.361.48        libnvidia-fatbinaryloader.so.361.48  libnvidia-ml.so.1           libvdpau_nvidia.so.361.48
libGLESv1_CM.so.1         libGLX_indirect.so.0           libnvcuvid.so.1          libnvidia-fbc.so.1                   libnvidia-ml.so.361.48
libGLESv1_CM_nvidia.so.1  libGLX_nvidia.so.0             libnvcuvid.so.361.48     libnvidia-fbc.so.361.48              libnvidia-opencl.so.1

[stefanseefeld]

Ah, there it is indeed ! (My app was only looking in /usr/lib[64]/, which I assume is wrong.)

Thanks for your help !

[3XX0]

Your app should not depend on a specific path, we do not guarantee the location of drivers libraries.
ld-linux.so is here for that and you should rely on it.

[victorhcm]

A related issue, I was trying to build an OpenCV docker image with CUDA support and cmake wasn't able to find libcuda.so, although I did verify it was there, at /usr/local/cuda/lib64/stubs. If I build it using nvidia-docker instead, will it be able to find it? If it doesn't, a possible workaround would be to symlink libcuda.so to a known library path:

$ ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/libcuda.so
$ ldconfig

I'm also appending my error log here, in case someone also stumbles with the same cryptic error message:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_CUDA_LIBRARY (ADVANCED)
    linked by target "example_gpu_alpha_comp" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_bgfg_segm" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_cascadeclassifier" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_cascadeclassifier_nvidia_api" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_driver_api_multi" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_driver_api_stereo_multi" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_farneback_optical_flow" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_generalized_hough" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_hog" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_houghlines" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_morphology" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_multi" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_optical_flow" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_opticalflow_nvidia_api" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_pyrlk_optical_flow" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_stereo_match" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_stereo_multi" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_super_resolution" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_surf_keypoint_matcher" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_video_reader" in directory /root/opencv/samples/gpu
    linked by target "example_gpu_video_writer" in directory /root/opencv/samples/gpu

[3XX0]

You need to use one of the devel images (see the list of tags available)

[victorhcm]

@3XX0 I'm using nvidia/cuda:7.5-cudnn5-devel, but still no success.

Also, I've just tried to build it with nvidia-docker, but it didn't work; symlinking solved it.

EDIT: Would you like me to share my Dockerfile?

[3XX0]

Then it's probably CMake ignoring LIBRARY_PATH.
You could have CMake point to /usr/local/cuda/lib64/stubs or workaround it with a symlink like you did.

[victorhcm]

If it is in LIBRARY_PATH, should it be printed with ldconfig -p | grep libcuda?

[3XX0]

No, ldconfig/LD_LIBRARY_PATH are used at runtime (see man ld.so).
At build time you should compile against the stubs libcuda.so either using LIBRARY_PATH (which is automatically set in the devel images) or using gcc -L or equivalent (in your case target_link_libraries).

With nvidia-docker the real libcuda.so will be mounted at runtime when your container starts.

[victorhcm]

Oh, I see! Thanks for clarifying, @3XX0.

[flx42]

What's your cmake build command for opencv? I had no problem compiling OpenCV with CUDA support recently (see here)

[victorhcm]

Hey, @flx42, here's an excerpt of my cmake command from my Dockerfile.

RUN \
    cd ~ && \
    git clone https://github.com/Itseez/opencv.git && \
    cd opencv && \
    pwd && \
    git checkout 3.1.0 && \

    cd ~ && \
    git clone https://github.com/Itseez/opencv_contrib.git && \
    cd opencv_contrib && \
    git checkout 3.1.0 && \

    cd /root/opencv && \
    mkdir build && \
    cd build && \
    cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D INSTALL_C_EXAMPLES=OFF \
        -D INSTALL_PYTHON_EXAMPLES=ON \
        -D OPENCV_EXTRA_MODULES_PATH=/root/opencv_contrib/modules \
        -D BUILD_EXAMPLES=ON ..

[flx42]

Thanks @victorhcm, I looked into this problem and this looks like an OpenCV bug, they don't include the stubs directory when searching for libraries.
I just created an issue against OpenCV: https://github.com/Itseez/opencv/issues/6577

[victorhcm]

Awesome, @flx42! Thanks for checking it and finding the error source.

[qiaohaijun]

@victorhcm thx