Lightgbm: Build issue with GPU on Ubuntu 16.04

[txjmb]

I tried rebuilding today to get the latest updates with GPU support following the GPU tutorial, and I get this error when running make -j$(nproc):

[ 94%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 98%] Linking CXX executable ../lightgbm
[ 98%] Built target lightgbm
[100%] Linking CXX shared library ../lib_lightgbm.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libboost_filesystem.a(operations.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libboost_filesystem.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
CMakeFiles/_lightgbm.dir/build.make:695: recipe for target '../lib_lightgbm.so' failed
make[2]: *** [../lib_lightgbm.so] Error 1
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/_lightgbm.dir/all' failed
make[1]: *** [CMakeFiles/_lightgbm.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I have tried a couple of recommended fixes for similar errors, but have been unable to resolve this yet. I am on Ubuntu 16.04.

Please let me know if you need any further information.

[Laurae2]

The solution to your problem is in the error message:

• You have the boost filesystem library compiled with absolute addresses, so it is suitable for programs, not libraries
• It tells you to recompile the boost filesystem library with -fPIC, because the compiler can't learn by itself how to translate absolute addresses into relative addresses (it is suitable for programs, not libraries)

When compiling boost with b2 or bjam, use cxxflags=-fPIC

[txjmb]

Thank you. The boost library I am using is the default that installs with the Ubuntu package manager. I will custom compile it per your suggestion. I do think that there is value to the project in understanding what issues users might have compiling on different platforms with the default packages, however. At least we can update the documentation for particular default platform issues.

[huanzhang12]

This problem is introduced by commit fb96b71.
@txjmb No, you don't need to recompile boost. The library installed by Ubuntu works. Just comment out the line set(Boost_USE_STATIC_LIBS ON) in CMakeList.txt, rm -rf build and rebuild

[huanzhang12]

@guolinke I think we don't want static libs on Linux, which can cause compilation troubles like this one. Maybe we just add set(Boost_USE_STATIC_LIBS ON) for Windows?

[guolinke]

@huanzhang12 okay.

[guolinke]

@txjmb can you try the latest code ?

[txjmb]

That fixed it. Thank you! Outstanding library with outstanding support! We are seeing very good results.

Michael

[Ubikas]

Hi @huanzhang12,

It looks like lightGBM doesn't want to let me go easily :) I run into a problem trying to run
sudo make -j

I successfully pass cmake with options
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/include/ ..
what returns
-- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info - done -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features - done -- Try OpenMP C flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED - Success -- Try OpenMP CXX flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED - Success -- Found OpenMP: -fopenmp -- Looking for CL_VERSION_2_0 - found -- Found OpenCL: /usr/local/cuda-8.0/lib64/libOpenCL.so (found version "2.0") -- OpenCL include directory:/usr/include/ -- Boost version: 1.58.0 -- Found the following Boost libraries: -- filesystem -- system -- Configuring done -- Generating done -- Build files have been written to: /usr/local/LightGBM/build
When I run 'sudo make -j' everything goes fine till 98% and then I receive few errors

[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o [ 98%] Linking CXX executable ../lightgbm CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In functionLightGBM::GPUTreeLearner::InitGPU(int, int)':
gpu_tree_learner.cpp:(.text+0x834e): undefined reference to clCreateCommandQueueWithProperties' CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In functionboost::compute::detail::meta_kernel::compile(boost::compute::context const&, std::__cxx11::basic_string, std::allocator > const&)':
gpu_tree_learner.cpp:(.text._ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x3b7): undefined reference to clSetKernelArgSVMPointer' collect2: error: ld returned 1 exit status CMakeFiles/lightgbm.dir/build.make:747: recipe for target '../lightgbm' failed make[2]: *** [../lightgbm] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lightgbm.dir/all' failed make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [100%] Linking CXX shared library ../lib_lightgbm.so [100%] Built target _lightgbm Makefile:127: recipe for target 'all' failed make: *** [all] Error 2
What's more interesting, it creates package!
What should I do to make it work? Any Ideas?
Thanks in advance!
EDIT
If You have got CUDA stuff, you have to compile with cuda. At the moment I'm writing Nvidia supports OpenGL 1.2 so if cmake log shows 2.0 it means something wrong. This is what finally worked for me

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..

[zachmayer]

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ .. fixes the error for me on ubuntu 14 with CUDA

[qifengzhou]

I got the same problem on installing the GPU version on Ubuntu.

'''
qifeng@DLS:~/LightGBM/build$ sudo make -j$(nproc)
Scanning dependencies of target lightgbm
Scanning dependencies of target _lightgbm
[ 1%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt.cpp.o
[ 3%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 4%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o
[ 6%] Building CXX object CMakeFiles/lightgbm.dir/src/io/json11.cpp.o
[ 8%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 9%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 11%] Building CXX object CMakeFiles/lightgbm.dir/src/io/parser.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm.dir/src/io/bin.cpp.o
[ 14%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 16%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 18%] Building CXX object CMakeFiles/lightgbm.dir/src/io/tree.cpp.o
[ 19%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset.cpp.o
[ 22%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
[ 24%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 27%] Building CXX object CMakeFiles/lightgbm.dir/src/io/file_io.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 34%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 36%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset_loader.cpp.o
[ 37%] Building CXX object CMakeFiles/lightgbm.dir/src/io/metadata.cpp.o
[ 39%] Building CXX object CMakeFiles/lightgbm.dir/src/io/config.cpp.o
[ 40%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/metric.cpp.o
[ 44%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 44%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 45%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 49%] Building CXX object CMakeFiles/lightgbm.dir/src/objective/objective_function.cpp.o
[ 50%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 52%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_socket.cpp.o
[ 54%] Building CXX object CMakeFiles/lightgbm.dir/src/network/network.cpp.o
[ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 59%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 60%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linker_topo.cpp.o
[ 62%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 63%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 68%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 70%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 72%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 83%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 86%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 88%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 91%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
make[2]: * No rule to make target '/usr/local/cuda-8.0/lib64/libOpenCL.so', needed by '../lightgbm'. Stop.
make[2]: Waiting for unfinished jobs....
[ 93%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 95%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
make[2]: No rule to make target '/usr/local/cuda-8.0/lib64/libOpenCL.so', needed by '../lib_lightgbm.so'. Stop.
make[2]: Waiting for unfinished jobs....
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lightgbm.dir/all' failed
make[1]: [CMakeFiles/lightgbm.dir/all] Error 2
make[1]: Waiting for unfinished jobs....
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/_lightgbm.dir/all' failed
make[1]: [CMakeFiles/_lightgbm.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: * [all] Error 2
'''
I tried 'sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -', it returns this error:
'CMake Error: The source directory "/home/qifeng/LightGBM/build/-" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.'

I checked the folder, it does exist, what could be the problem?

[Ubikas]

Hi @qifengzhou ,
I had to spend some time until I manged to collect all info to make it right. It's already been some time since my last lightgbm update run, hope script still valid :)

sudo pip uninstall lightgbm
sudo rm -r LightGBM
sudo rm -r lightgbm
sudo git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
sudo mkdir build ; cd build
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
sudo make -j$(nproc)
cd python-package
sudo python setup.py install --precompile

Mind catalogs according to your setup and cuda.

[StrikerRUS]

@Ubikas Thank you for sharing your experience!

I just want to note, that Python installation can take args with paths now. So, the script can be simplified:

sudo pip uninstall lightgbm
sudo pip install lightgbm --install-option=--gpu --install-option="--opencl-include-dir=/usr/local/cuda-8.0/include/" --install-option="--opencl-library=/usr/local/cuda-8.0/lib64/libOpenCL.so"

[qifengzhou]

Hi @Ubikas @StrikerRUS,

Thank you for the instruction!
I followed it and compiled it successfully. However, its still not working, I got an error when running LGB in jupyter notebook:
"[LightGBM] [Warning] Unknown parameter: tree_method
[LightGBM] [Warning] Find whitespaces in feature_names, replace with underlines
[LightGBM] [Warning] Unknown parameter: tree_method
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1"