Dlib: #Cuda doesn't enable installation time dlib

Created on 8 Mar 2019 · 17Comments · Source: davisking/dlib

Dear Davisking,

CUDA doesn't enable dlib installation time What i wrong in following:

tarun@tarun:~/packages/dlib-19.16.0$ sudo python3 setup.py install --record files.txt --yes USE_AVX_INSTRUCTIONS --yes DLIB_USE_CUDA --compiler-flags "-DCUDA_HOST_COMPILER=/usr/bin/gcc-7"
running install
running bdist_egg
running egg_info
writing dlib.egg-info/PKG-INFO
writing dependency_links to dlib.egg-info/dependency_links.txt
writing top-level names to dlib.egg-info/top_level.txt
package init file 'dlib/__init__.py' not found (or not a regular file)
reading manifest file 'dlib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'dlib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Building extension for Python 3.6.7 (default, Oct 22 2018, 11:32:17)
Invoking CMake setup: 'cmake /home/annex/packages/dlib-19.16.0/tools/python -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/annex/packages/dlib-19.16.0/build/lib.linux-x86_64-3.6 -DPYTHON_EXECUTABLE=/usr/bin/python3 -DUSE_AVX_INSTRUCTIONS=yes -DDLIB_USE_CUDA=yes -DCMAKE_CXX_FLAGS=-DCUDA_HOST_COMPILER=/usr/bin/gcc-7 -DCMAKE_BUILD_TYPE=Release'
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v2.2.2
-- Using CMake version: 3.12.4
-- Compiling dlib version: 19.16.0
-- SSE4 instructions can be executed by the host processor.
-- AVX instructions can be executed by the host processor.
-- Enabling AVX instructions
-- Searching for BLAS and LAPACK
-- Searching for BLAS and LAPACK
-- Checking for module 'cblas'
-- No package 'cblas' found
-- Found OpenBLAS library
-- Found LAPACK library
-- Looking for cuDNN install...
-- Found cuDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Disabling CUDA support for dlib. DLIB WILL NOT USE CUDA
-- C++11 activated.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/annex/packages/dlib-19.16.0/build/temp.linux-x86_64-3.6
Invoking CMake build: 'cmake --build . --config Release -- -j4'
Scanning dependencies of target dlib

Cmake Version:

tarun@tarun:~/packages/dlib-19.16.0$ cmake --version
cmake version 3.13.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Graphic card driver:

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1029 G /usr/lib/xorg/Xorg 14MiB |
| 0 1128 G /usr/bin/gnome-shell 56MiB |
| 0 1307 G /usr/lib/xorg/Xorg 175MiB |
| 0 1438 G /usr/bin/gnome-shell 127MiB |
| 0 11533 G ...-token=B4E7D8607DEC3AA5D126D41102A8FE35 47MiB |
| 0 16072 C /usr/lib/libreoffice/program/soffice.bin 105MiB |
+-----------------------------------------------------------------------------+

Please Suggest me where I am doing wrong

Source

tarunbadial

All 17 comments

It's not clear to me why you don't get more error messages about why it's not willing to use cuda. You should look in dlib/CMakeLists.txt for the text "Disabling CUDA support for dlib. DLIB WILL NOT USE CUDA" and add message, printing variables around there to see why it's getting there.

Also, you don't need to say --yes ever, these things are always on by default and --yes is a noop.

davisking on 8 Mar 2019

I also don't think you want to do --compiler-flags "-DCUDA_HOST_COMPILER=/usr/bin/gcc-7". Passing that to GCC doesn't really make any sense.

davisking on 8 Mar 2019

actually I have no idea how to install dlib for cuda .I am follow this http://leadtosilverlining.blogspot.com/2018/08/build-and-install-dlib-with-cuda.html?m=1](link) for installation.

tarunbadial on 8 Mar 2019

Always follow the official instructions: http://dlib.net/compile.html.

That page, and many others like it, are full of cargo cult programming nonsense. You just run python setup.py install and that's it. Obviously you need to install cmake, gcc, cuda, and cudnn first though

davisking on 8 Mar 2019

@davisking thanks for providing this link I have already through out in Ubuntu 18.04 gcc versions 6 and 7 , cuda, cudnn , but I not have idea how to enable dlib for cuda support.

tarunbadial on 8 Mar 2019

Hi,

Also had many such _unexplained_ issues while installing CUDA toolkit and eventually dlib.
I'm totally agree with @davisking that some websites suggest on specifying compiler flags that have no basis. Tried but nothing good happened.

Having no errors in output seems strange, and there is no "Found CUDA" message before "Looking for cuDNN" (this is how I had it in my output) but any way did you try to check logs in: dlib/build/temp.linux-x86_64-3.6/CMakeFiles. There should be CMakeError.log file.

I believe there are a lot of tutorials but after struggling for some time I found this one that worked for me. What most important, it describes how to check if your CUDA toolkit and cuDNN will work before compiling dlib library. You should be able to compile CUDA examples. Actually this what saved me a lot of time. See "Post Installation Actions" in: NVIDIA CUDA TOOLKIT INSTALLATION

Also I would validate that:

CUDA toolkit installation passed successfully by: nvcc -V
LD_LIBRARY_PATH is updated with you cuda libraries locations.

Good luck :)

pavelpe on 10 Mar 2019

You don't need to set LD_LIBRARY_PATH. I always follow the official NVIDIA instructions. They are right here where you download cuda: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

Note that those instructions tell you exactly what to type. If you follow them cuda will be installed and working. This is what I do and it always works. I think the general advice of always following the official install instructions is hugely important. All these random blog posts contain lots of extra steps that either do nothing and just confuse the reader, or actively break your install.

davisking on 10 Mar 2019

You don't need to set LD_LIBRARY_PATH.

I'm not trying to argue what is the correct way but those are actually the official instructions.
It can be found here: CUDA Installation: Post-Installation Actions
Maybe, in most cases, those "Post-Installation Actions" are not necessary but specifically in my case it didn't work until I added the lib path to .profile file.

And no, I didn't use runfile installation method.

pavelpe on 10 Mar 2019

Maybe if you installed via a method other than the .deb file. But the .deb adds the appropriate entries in /etc/ld.so.conf.d so you don't have to set anything in LD_LIBRARY_PATH

Did you install via apt-get install cuda? I don't see how you could do that and not end up with /etc/ld.so.conf.d setup.

davisking on 10 Mar 2019

Did you install via apt-get install cuda? I don't see how you could do that and not end up with /etc/ld.so.conf.d setup.

To be honest, I don't know how I ended up in my situation.
But I did it via downloading the .deb package, opening it with sudo dpkg -i ... and then running apt-get update + apt-get install cuda (following the exact steps of tutorial I attached in my previous messages).

Maybe due to all those _optional_ tutorials online, eventually it screwed up some configurations in Ubuntu. No idea.

Anyway I'm not suggesting it doing my way, but what and how to validate CUDA toolkit installation before running dlib compilation.

pavelpe on 10 Mar 2019

👍1

As per my observation dlib is not supporting gcc version after 6.0. Ubuntu 18.04 by default gcc version 7.5 so I am facing this problem, I was check cuda drivers version 10.1 are properly works,when I run a pycuda coding.

tarunbadial on 10 Mar 2019

@tarunbadial dlib is built against multiple versions of gcc all the time, including gcc 7. I'm using dlib in ubuntu 18.04 right now and it works fine. You just have something wrong with your setup. Also, use the latest version of dlib v19.17.

davisking on 10 Mar 2019

@davisking In the past, I am using the dlib library in CPU its works fine. But now I need to dlib for the cuda, but I seem problem in following screenshot:

dlib Version 19.16

1552115958411

I don't have idea where i am wrong. Please suggest me..

tarunbadial on 11 Mar 2019

I don't have idea where i am wrong. Please suggest me..

The problem as it stated in error message is with your CUDA and gcc you use. It has nothing to do with dlib.
"CUDA failed to compile a simple CUDA program" and reason seems to be gcc version.

I would suggest totally removing this line from your run:
--compiler-flags "-DCUDA_HOST_COMPILER=/usr/bin/gcc-7"

If it doesn't help try:
--compiler-flags "-DCUDA_HOST_COMPILER=/usr/bin/gcc-6"
See related CUDA issue (old!): https://github.com/ethereum-mining/ethminer/issues/731

pavelpe on 11 Mar 2019

Everyone needs to stop setting --compiler-flags "-DCUDA_HOST_COMPILER=<whatever>". That flag isn't even passed to cmake so it can not possibly select the compiler to use. You have to use the --set option of setup.py to pass something like this to cmake if you want cmake to use a different compiler. Not --compiler-flags. This is documented at the top of setup.py.

More importantly, cmake will pick the right compiler on Ubuntu. So you don't have to do this and if you do you are almost certainly doing something wrong.

@tarunbadial Have you done sudo python3 setup.py install --clean yet? Don't give any extra arguments. Just run it like the instructions say. Note that --clean here will delete all the build folders, which is important since you need to remove all the incorrect configurations you have in there from all these other commands.

davisking on 11 Mar 2019

👍1

@davisking Thanks for giving solution finally my problem is resolved actually I was facing some issue in gcc version, but now dlib is properly work with the cuda in following version asap:
Ubuntu 18.04
gcc version 7.8
dlib 19.16
😊

tarunbadial on 14 Mar 2019

🎉1

Same problem. And can't defeat it

root@ubuntich-desktop:/opt/SP/packages/dlib# sudo python3 setup.py install --clean
running install
running bdist_egg
running egg_info
writing tools/python/dlib.egg-info/PKG-INFO
writing dependency_links to tools/python/dlib.egg-info/dependency_links.txt
writing top-level names to tools/python/dlib.egg-info/top_level.txt
package init file 'tools/python/dlib/__init__.py' not found (or not a regular file)
reading manifest file 'tools/python/dlib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'tools/python/build'
writing manifest file 'tools/python/dlib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Removing old directory /opt/SP/packages/dlib/build/temp.linux-x86_64-3.6
Building extension for Python 3.6.9 (default, Apr 18 2020, 01:56:04)
Invoking CMake setup: 'cmake /opt/SP/packages/dlib/tools/python -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/opt/SP/packages/dlib/build/lib.linux-x86_64-3.6 -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=Release'
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python3 (found version "3.6.9")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- pybind11 v2.2.2
-- Using CMake version: 3.10.2
-- Compiling dlib version: 19.20.99
-- SSE4 instructions can be executed by the host processor.
-- Enabling SSE4 instructions
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so
-- Looking for XOpenDisplay in /usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib/x86_64-linux-gnu/libX11.so
-- Searching for BLAS and LAPACK
-- Searching for BLAS and LAPACK
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'cblas'
-- No package 'cblas' found
-- Checking for module 'lapack'
-- No package 'lapack' found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void
-- Check size of void* - done

* No BLAS library found so using dlib's built in BLAS. However, if you
install an optimized BLAS such as OpenBLAS or the Intel MKL your code
will run faster. On Ubuntu you can install OpenBLAS by executing:
sudo apt-get install libopenblas-dev liblapack-dev
Or you can easily install OpenBLAS from source by downloading the
source tar file from http://www.openblas.net, extracting it, and
running:
make; sudo make install *

-- Found CUDA: /usr (found suitable version "9.1", minimum required is "7.5")
-- Looking for cuDNN install...
-- Found cuDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Building a CUDA test project to see if your compiler is compatible with CUDA...

-- * CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA.
-- The output of the failed CUDA test compile is shown below:
-- Change Dir: /opt/SP/packages/dlib/build/temp.linux-x86_64-3.6/dlib_build/cuda_test_build

Run Build Command:"/usr/bin/make"
[ 50%] Building NVCC (Device) object CMakeFiles/cuda_test.dir/cuda_test_generated_cuda_test.cu.o
In file included from /usr/include/host_config.h:50:0,
from /usr/include/cuda_runtime.h:78,
from :0:
/usr/include/crt/host_config.h:121:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
CMake Error at cuda_test_generated_cuda_test.cu.o.cmake:215 (message):
Error generating
/opt/SP/packages/dlib/build/temp.linux-x86_64-3.6/dlib_build/cuda_test_build/CMakeFiles/cuda_test.dir//./cuda_test_generated_cuda_test.cu.o

CMakeFiles/cuda_test.dir/build.make:63: recipe for target 'CMakeFiles/cuda_test.dir/cuda_test_generated_cuda_test.cu.o' failed
make[2]: [CMakeFiles/cuda_test.dir/cuda_test_generated_cuda_test.cu.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/cuda_test.dir/all' failed
make[1]: [CMakeFiles/cuda_test.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: [all] Error 2
**

-- Disabling CUDA support for dlib. DLIB WILL NOT USE CUDA
-- C++11 activated.
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/SP/packages/dlib/build/temp.linux-x86_64-3.6
Invoking CMake build: 'cmake --build . --config Release -- -j2'
Scanning dependencies of target dlib

NVIDIA-SMI 440.44 Driver Version: 440.44 CUDA Version: 10.2
cmake version 3.10.2
Cuda compilation tools, release 9.1, V9.1.85
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

GPU 1060 6gb
CPU Pentium 4500 (WITHOUT AVX)