Lightgbm: [GPU] Error while running cmake

[dvarkiac]

I have downloaded "LightGBM-master" and try to run compile lightgbm but getting the below error.
Installed opencl and boot as well. Currently using CUDA 8 drivers for gpu. using cmake version ( 3.2.2).
Could you please le t me know how to solve this issue.

cd LightGBM-master

cd build/

cmake -DUSE_GPU=1 ..

-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
CMake Error at /opt/cmake-3.2.2-Linux-x86_64/share/cmake-3.2/Modules/FindPackageHandleStandardArgs.cmake:138 (message):
Could NOT find OpenCL (missing: OpenCL_LIBRARY OpenCL_INCLUDE_DIR)
Call Stack (most recent call first):
/opt/cmake-3.2.2-Linux-x86_64/share/cmake-3.2/Modules/FindPackageHandleStandardArgs.cmake:374 (_FPHSA_FAILURE_MESSAGE)
/opt/cmake-3.2.2-Linux-x86_64/share/cmake-3.2/Modules/FindOpenCL.cmake:128 (find_package_handle_standard_args)
CMakeLists.txt:39 (find_package)

=======================
/usr/lib64/nvidia/libnvidia-opencl.so
/usr/lib64/nvidia/libnvidia-opencl.so.1
/usr/lib64/nvidia/libnvidia-opencl.so.375.26
/usr/local/cuda-8.0/targets/x86_64-linux/include/CL/opencl.h

[guolinke]

it misses OpenCL.
Refer to https://github.com/Microsoft/LightGBM/wiki/Installation-Guide#with-gpu-support to get more details.

[dvarkiac]

I followed the same; but i am still getting the same error. whether I need to provide the OPENCL include and Library path specifically. if so how. Please help.

Below is the path of opencl , which came along with cuda driver installation.
/usr/lib64/nvidia/libnvidia-opencl.so
/usr/lib64/nvidia/libnvidia-opencl.so.1
/usr/lib64/nvidia/libnvidia-opencl.so.375.26
/usr/local/cuda-8.0/targets/x86_64-linux/include/CL/opencl.h

[Laurae2]

as you have NVIDIA drivers installed, make sure you have, you can specify manually OpenCL_LIBRARY and OpenCL_INCLUDE_DIR on the cmake line as it seems you know where they are, like how you use -DUSE_GPU=1 for USE_GPU=1.

[huanzhang12]

@dvarkiac CMake can't find OpenCL because there is no libOpenCL.so found in your system. If you are using Ubuntu, you can try to install ocl-icd-libopencl1 and ocl-icd-opencl-dev. Also check if there is a file (filename should include "nvidia", I forget the exact name) inside /etc/OpenCL/vendors/.

[Laurae2]

@dvarkiac If you can't make it work, can you also try using cmake-gui (it's available for Unix-variants) and fiddling with the paths manually?

If your OpenCL library and OpenCL headers are not in the same base directory, typically it causes issues for cmake to find them (like yours with /usr/lib64/nvidia and usr/local/cuda-8.0/targets...).

You need the following to specify manually:

• OpenCL_LIBRARY: that folder has OpenCL.lib or libOpenCL file (or similar name)
• OpenCL_INCLUDE_DIR: that folder has a folder CL with a file called CL.h inside (or similar name)

[guolinke]

@dvarkiac any updates ?

[xin-jin]

I am having the same issue. I have the following installed on my system (ubuntu 16.04):

ii  nvidia-opencl-icd-375                         375.39-0ubuntu0.16.04.1                       amd64        NVIDIA OpenCL ICD
ii  ocl-icd-libopencl1:amd64                      2.2.8-1                                       amd64        Generic OpenCL ICD Loader
ii  ocl-icd-libopencl1:i386                       2.2.8-1                                       i386         Generic OpenCL ICD Loader
ii  ocl-icd-opencl-dev:amd64                      2.2.8-1                                       amd64        OpenCL development files
ii  opencl-headers                                2.0~svn32091-2                                all          OpenCL (Open Computing Language) header files

cmake has the error message:

CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.0")
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.5/Modules/FindOpenCL.cmake:128 (find_package_handle_standard_args)
  CMakeLists.txt:41 (find_package)


-- Configuring incomplete, errors occurred!

[huanzhang12]

@xin-jin Based on the log if seems OpenCL headers were found but the library (.so) is missing. However it seems you already have relevant packages installed. My suggestion:

• check if /usr/lib/x86_64-linux-gnu/libOpenCL.so exists
• check if /etc/OpenCL/vendors/nvidia.icd exists
• clone the repository in a clean folder and retry build
• If still not working, try to install nvidia-libopencl1-375

[xin-jin]

@huanzhang12 Hi, both libOpenCl.so and nvidia.icd exist (in the directories you mentioned), and I also installed nvidia-libopencl1-375 just now. But the same error message appears (build from a clean clone).

[xin-jin]

Wait, it appears that not exactly libOpenCL.so exists:

ls /usr/lib/x86_64-linux-gnu/libOpenCL*
/usr/lib/x86_64-linux-gnu/libOpenCL.so.1  /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0  /usr/lib/x86_64-linux-gnu/libOpenCL.so.1.0.0

I don't know what happened. It used to exist, and the name is changed after installing nvidia-libopencl1-375.

update: I created a soft link named libOpenCL.so, and it still doesn't work.

[huanzhang12]

@xin-jin Can you also try to install nvidia-opencl-dev along with nvidia-libopencl1-375?

[xin-jin]

@huanzhang12 Doesn't work either ... Same error message.

[xin-jin]

OK,
cmake -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so ..
seems to pass the check. However, now boost is not found:

-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.0") 
-- OpenCL include directory:/usr/include
CMake Error at /usr/share/cmake-3.5/Modules/FindBoost.cmake:1677 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.58.0

  Boost include path: /usr/include

  Could not find the following Boost libraries:

          boost_filesystem
          boost_system

  No Boost libraries were found.  You may need to set BOOST_LIBRARYDIR to the
  directory containing Boost libraries or BOOST_ROOT to the location of
  Boost.
Call Stack (most recent call first):
  CMakeLists.txt:44 (find_package)

Something like -DBOOST_LIBRARYDIR=/usr/include/boost doesn't work.

[huanzhang12]

Do you have necessary boost libraries (boost_filesystem and boost_system) installed?
Try apt-get install libboost-system-dev libboost-filesystem-dev

[xin-jin]

@huanzhang12 I have installed libboost-all-dev which includes both system and filesystem.

[huanzhang12]

@xin-jin You can try to set -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu/
It seems your CMake never searches for /usr/lib/x86_64-linux-gnu/, which is weird.
I guess it is a problem with cmake.

[xin-jin]

@huanzhang12 Thanks. Now cmake goes through. But make is now complaining:

[ 98%] Linking CXX shared library ../lib_lightgbm.so
[100%] Linking CXX executable ../lightgbm
[100%] Built target _lightgbm
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_not_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)@GLIBCXX_3.4.20'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::runtime_error::runtime_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char, unsigned long) const@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char, unsigned long) const@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_M_replace(unsigned long, unsigned long, wchar_t const*, unsigned long)@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const@GLIBCXX_3.4.21'
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_last_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
collect2: error: ld returned 1 exit status
CMakeFiles/lightgbm.dir/build.make:695: recipe for target '../lightgbm' failed
make[2]: *** [../lightgbm] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lightgbm.dir/all' failed
make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

[xin-jin]

Seems to be a compiler compatibility problem. I manually specify the version of gcc, and now it works.

So in sum, this is what finally works:
cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu/ -DCMAKE_CXX_COMPILER=g++-4.9 ..

Update: The source of the incompatibility stems from anaconda which includes inside itself a gcc which is outdated (4.8.5). Since anaconda is the first entry in PATH, that gcc is unfortunately used. Removing g++ under anaconda/bin/ also resolves the issue.

[alexmulo]

Hi,

I solved the proble using this line:

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..

This is for all of those using CUDA

[josephgp1]

Hi,

I installed R lightgbm in ubuntu 18.04 LTS using cmake

[ 98%] Linking CXX shared library ../lib_lightgbm.so
[ 98%] Built target _lightgbm
[100%] Linking CXX executable ../lightgbm
[100%] Built target lightgbm

How do i import in R. lightgbm is not in the R library folder.

Thanks