Lightgbm: Build w/ GPU but got CPU-only ver install in python (Ubuntu 16.04)

Created on 21 Jul 2017 · 17Comments · Source: microsoft/LightGBM

After managed to built a LightGBM w/ GPU on Ubuntu 16.04, it asked to "Recompile with CMake option -DUSE_GPU=1'" while calling LightGBM in python 3.5. However, the binay ./LightGBM/lightgbm is able to enable GPU for computation. GPU is availabe after replacing lib_lightgbm.so in /usr/local/lib/python3.5/dist-packages/lightgbm by one ./LightGBM/ . The lib_lightgbm.so installed within python seems a CPU-only built.

The compiling LightGBM w/ GPU is following by the issue 458 by "sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ .." from https://github.com/Microsoft/LightGBM/issues/458 . Also, pip3 installation did not work for my pc.

Meanwhile, I've notice there are two different sizes of lib_lightgbm.so
2.2MB ./LightGBM/lib_lightgbm.so
1.5MB ./LightGBM/python-package/lightgbm/lib_lightgbm.so
1.5MB ./LightGBM/python-package/build/lib/lightgbm/lib_lightgbm.so
1.5MB .//usr/local/lib/python3.5/dist-packages/lightgbm/lib_lightgbm.so

It seems to me that another CPU-only lib_lightgbm.so is compiled while run "python3 setup.py install". Therefore, I decided to copy the lib_lightgbm.so and so far it works fine.

Environment info

Operating System: Ubuntu 16.04
CPU: i3-7350K / GTX 1080 (375.66)
C++/Python/R version: Python

Error Message:

[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1
Traceback (most recent call last):
File "hybrid.py", line 211, in
clf = lgb.train(params, d_train, 430)
File "/usr/local/lib/python3.5/dist-packages/lightgbm/engine.py", line 168, in train
booster = Booster(params=params, train_set=train_set)
File "/usr/local/lib/python3.5/dist-packages/lightgbm/basic.py", line 1215, in __init__
ctypes.byref(self.handle)))
File "/usr/local/lib/python3.5/dist-packages/lightgbm/basic.py", line 47, in _safe_call
raise LightGBMError(_LIB.LGBM_GetLastError())
lightgbm.basic.LightGBMError: b'GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1'

Steps to reproduce

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
-- Found OpenCL: /usr/local/cuda-8.0/lib64/libOpenCL.so (found version "1.2")
-- OpenCL include directory:/usr/local/cuda-8.0/include
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- filesystem
-- system
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cttsai/DataSci/mlc/LightGBM/build
sudo make -j$(nproc)
cd ../python-package; sudo python3 setup.py install
running install
creating lightgbm/include
creating lightgbm/include/LightGBM
copying ../include/LightGBM/prediction_early_stop.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/dataset_loader.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/dataset.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/application.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/bin.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/c_api.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/tree.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/export.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/lightgbm_R.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/meta.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/metric.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/network.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/objective_function.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/R_object_helper.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/boosting.h -> ./lightgbm/include/LightGBM
creating lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/pipeline_reader.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/random.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/array_args.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/common.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/log.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/threading.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/text_reader.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/utils/openmp_wrapper.h -> ./lightgbm/include/LightGBM/utils
copying ../include/LightGBM/feature_group.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/tree_learner.h -> ./lightgbm/include/LightGBM
copying ../include/LightGBM/config.h -> ./lightgbm/include/LightGBM
creating lightgbm/src
creating lightgbm/src/application
copying ../src/application/application.cpp -> ./lightgbm/src/application
copying ../src/application/predictor.hpp -> ./lightgbm/src/application
copying ../src/main.cpp -> ./lightgbm/src
creating lightgbm/src/objective
copying ../src/objective/regression_objective.hpp -> ./lightgbm/src/objective
copying ../src/objective/multiclass_objective.hpp -> ./lightgbm/src/objective
copying ../src/objective/rank_objective.hpp -> ./lightgbm/src/objective
copying ../src/objective/binary_objective.hpp -> ./lightgbm/src/objective
copying ../src/objective/objective_function.cpp -> ./lightgbm/src/objective
copying ../src/objective/xentropy_objective.hpp -> ./lightgbm/src/objective
creating lightgbm/src/boosting
copying ../src/boosting/dart.hpp -> ./lightgbm/src/boosting
copying ../src/boosting/goss.hpp -> ./lightgbm/src/boosting
copying ../src/boosting/gbdt_prediction.cpp -> ./lightgbm/src/boosting
copying ../src/boosting/gbdt.h -> ./lightgbm/src/boosting
copying ../src/boosting/boosting.cpp -> ./lightgbm/src/boosting
copying ../src/boosting/gbdt.cpp -> ./lightgbm/src/boosting
copying ../src/boosting/rf.hpp -> ./lightgbm/src/boosting
copying ../src/boosting/score_updater.hpp -> ./lightgbm/src/boosting
copying ../src/boosting/prediction_early_stop.cpp -> ./lightgbm/src/boosting
creating lightgbm/src/metric
copying ../src/metric/xentropy_metric.hpp -> ./lightgbm/src/metric
copying ../src/metric/rank_metric.hpp -> ./lightgbm/src/metric
copying ../src/metric/dcg_calculator.cpp -> ./lightgbm/src/metric
copying ../src/metric/metric.cpp -> ./lightgbm/src/metric
copying ../src/metric/regression_metric.hpp -> ./lightgbm/src/metric
copying ../src/metric/multiclass_metric.hpp -> ./lightgbm/src/metric
copying ../src/metric/binary_metric.hpp -> ./lightgbm/src/metric
copying ../src/metric/map_metric.hpp -> ./lightgbm/src/metric
creating lightgbm/src/treelearner
copying ../src/treelearner/gpu_tree_learner.cpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/split_info.hpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/feature_parallel_tree_learner.cpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/leaf_splits.hpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/data_parallel_tree_learner.cpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/serial_tree_learner.cpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/data_partition.hpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/feature_histogram.hpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/tree_learner.cpp -> ./lightgbm/src/treelearner
copying ../src/treelearner/parallel_tree_learner.h -> ./lightgbm/src/treelearner
copying ../src/treelearner/voting_parallel_tree_learner.cpp -> ./lightgbm/src/treelearner
creating lightgbm/src/treelearner/ocl
copying ../src/treelearner/ocl/histogram256.cl -> ./lightgbm/src/treelearner/ocl
copying ../src/treelearner/ocl/histogram64.cl -> ./lightgbm/src/treelearner/ocl
copying ../src/treelearner/ocl/histogram16.cl -> ./lightgbm/src/treelearner/ocl
copying ../src/treelearner/gpu_tree_learner.h -> ./lightgbm/src/treelearner
copying ../src/treelearner/serial_tree_learner.h -> ./lightgbm/src/treelearner
creating lightgbm/src/io
copying ../src/io/parser.hpp -> ./lightgbm/src/io
copying ../src/io/metadata.cpp -> ./lightgbm/src/io
copying ../src/io/dense_bin.hpp -> ./lightgbm/src/io
copying ../src/io/sparse_bin.hpp -> ./lightgbm/src/io
copying ../src/io/ordered_sparse_bin.hpp -> ./lightgbm/src/io
copying ../src/io/bin.cpp -> ./lightgbm/src/io
copying ../src/io/dense_nbits_bin.hpp -> ./lightgbm/src/io
copying ../src/io/tree.cpp -> ./lightgbm/src/io
copying ../src/io/parser.cpp -> ./lightgbm/src/io
copying ../src/io/dataset_loader.cpp -> ./lightgbm/src/io
copying ../src/io/config.cpp -> ./lightgbm/src/io
copying ../src/io/dataset.cpp -> ./lightgbm/src/io
copying ../src/lightgbm_R.cpp -> ./lightgbm/src
copying ../src/c_api.cpp -> ./lightgbm/src
creating lightgbm/src/network
copying ../src/network/socket_wrapper.hpp -> ./lightgbm/src/network
copying ../src/network/linker_topo.cpp -> ./lightgbm/src/network
copying ../src/network/network.cpp -> ./lightgbm/src/network
copying ../src/network/linkers_socket.cpp -> ./lightgbm/src/network
copying ../src/network/linkers.h -> ./lightgbm/src/network
copying ../src/network/linkers_mpi.cpp -> ./lightgbm/src/network
copying ../CMakeLists.txt -> ./lightgbm/
copying ../LICENSE -> ./
Start to compile libarary.
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cttsai/DataSci/mlc/LightGBM/python-package/build_cpp
Scanning dependencies of target _lightgbm
[ 3%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 7%] Building CXX object CMakeFiles/./LightGBM/_lightgbm.dir/src/lightgbm_R.cpp.o
[ 10%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 39%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 42%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 60%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 71%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 89%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 92%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[100%] Linking CXX shared library /home/cttsai/DataSci/mlc/LightGBM/python-package/lightgbm/lib_lightgbm.so
[100%] Built target _lightgbm

Source

cttsai1985

Most helpful comment

@guolinke
Yeah, it was my work around solution. I don’t mind to delete LightGBM one more time today. :)
But this time it works! Thanks you guys @huanzhang12 @guolinke @wxchan working so hard to get this issue solved. The rest part is what did I do this time to remove previous installation and have a new one.

Before this time I remove LightGBM packages in python by a solution found in stacked-overflow:

sudo python3 setup.py install --precompile --record files.txt
cat files.txt | sudo xargs rm -rf

Since the lightgbm lightgbm-2.0.4-py3.5.egg-info folder I would going to replace it later so I deleted both of them anyway this time. Maybe there was something mess around.

sudo python3 setup.py install --precompile --record files.txt
cat files.txt | sudo xargs rm -rf
cd /usr/local/lib/python3.5/dist-packages/
sudo rm -r -f lightgbm lightgbm-2.0.4-py3.5.egg-info/

I re-installed by the following steps:

git clone --recursive https://github.com/Microsoft/LightGBM
cd ./LightGBM
mkdir build; cd build
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
sudo make -j$(nproc)
cd ../python-package; sudo python3 setup.py install --precompile

cttsai1985 on 22 Jul 2017

👍9

All 17 comments

I think you should run python3 setup.py install --precompile after install c++ version, or python3 setup.py install --gpu to direct install gpu version for python. @guolinke can confirm this.

wxchan on 21 Jul 2017

@cttsai1985
We had simplified the python installation. You don't need the build the cli version first.
Refer to: https://github.com/Microsoft/LightGBM/tree/master/python-package

guolinke on 21 Jul 2017

@guolinke Can we check precompile first when run python setup.py install?

wxchan on 21 Jul 2017

@wxchan I don't think so. I think make it independently is better.

guolinke on 21 Jul 2017

It's kind of counter-intuitive. I think most users don't know those --gpu, --precompile options.

wxchan on 21 Jul 2017

@guolinke @wxchan
Thanks for your reply. I followed the instruction below but got more error messages:
https://github.com/Microsoft/LightGBM/tree/master/python-package
Now the installation works for me to enable GPU for python still is the old way. Install a CPU-only version then replace the lib_lightgbm.so . Does it have anything to do with libOpenCL.so?

sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
make -j$(nproc)
sudo python3 setup.py install
cd /usr/local/lib/python3.5/dist-packages/lightgbm
sudo cp ~/LightGBM/lib_lightgbm.so ./lib_lightgbm.so

Here is some log I collected:

Using pip installation:

1.1 sudo pip3 install --no-binary :all: lightgbm

It installed CPU-only version, so I get
lightgbm.basic.LightGBMError: b'GPU Tree Learner was not enabled in this build. Recompile with CMake option -DUSE_GPU=1'

1.2 sudo pip3 install lightgbm --install-option=--gpu

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mnysq7z9/lightgbm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-42d4zwxk-record/install-record.txt --single-version-externally-managed --compile --gpu" failed with error code 1 in /tmp/pip-build-mnysq7z9/lightgbm/

Using python3 setup.py install --precompile or --gpu

Both got the same error message:

Traceback (most recent call last):
File "hybrid.py", line 133, in
import lightgbm as lgb
File "/usr/local/lib/python3.5/dist-packages/lightgbm/__init__.py", line 8, in
from .basic import Booster, Dataset
File "/usr/local/lib/python3.5/dist-packages/lightgbm/basic.py", line 31, in
_LIB = _load_lib()
File "/usr/local/lib/python3.5/dist-packages/lightgbm/basic.py", line 26, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/usr/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
return self._dlltype(name)
File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)

OSError: /usr/local/lib/python3.5/dist-packages/lightgbm/lib_lightgbm.so: symbol clCreateCommandQueueWithProperties, version OPENCL_2.0 not defined in file libOpenCL.so.1 with link time reference

2.1 sudo python3 setup.py install --precompile

running install
Install lib_lightgbm from: ['../lib_lightgbm.so']
running build
running build_py
creating build
creating build/lib
creating build/lib/lightgbm
copying lightgbm/plotting.py -> build/lib/lightgbm
copying lightgbm/__init__.py -> build/lib/lightgbm
copying lightgbm/libpath.py -> build/lib/lightgbm
copying lightgbm/callback.py -> build/lib/lightgbm
copying lightgbm/engine.py -> build/lib/lightgbm
copying lightgbm/sklearn.py -> build/lib/lightgbm
copying lightgbm/compat.py -> build/lib/lightgbm
copying lightgbm/basic.py -> build/lib/lightgbm
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing top-level names to lightgbm.egg-info/top_level.txt
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching 'LICENSE'
warning: no files found matching '.txt'
warning: no files found matching '.so' under directory 'lightgbm'
warning: no files found matching '.dll' under directory 'lightgbm/Release'
warning: no files found matching '' under directory 'lightgbm/include'
warning: no files found matching '' under directory 'lightgbm/src'
warning: no previously-included files matching '.pyo' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
copying build/lib/lightgbm/plotting.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/__init__.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/libpath.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/callback.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/engine.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/VERSION.txt -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/sklearn.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/compat.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
copying build/lib/lightgbm/basic.py -> /usr/local/lib/python3.5/dist-packages/lightgbm
Install lib_lightgbm from: ['../lib_lightgbm.so']
copying ../lib_lightgbm.so -> /usr/local/lib/python3.5/dist-packages/lightgbm
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/plotting.py to plotting.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/__init__.py to __init__.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/libpath.py to libpath.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/callback.py to callback.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/engine.py to engine.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/sklearn.py to sklearn.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/compat.py to compat.cpython-35.pyc
byte-compiling /usr/local/lib/python3.5/dist-packages/lightgbm/basic.py to basic.cpython-35.pyc
running install_data
copying ../lib_lightgbm.so -> /usr/local/lightgbm
running install_egg_info
removing '/usr/local/lib/python3.5/dist-packages/lightgbm-2.0.4-py3.5.egg-info' (and everything under it)
Copying lightgbm.egg-info to /usr/local/lib/python3.5/dist-packages/lightgbm-2.0.4-py3.5.egg-info
running install_scripts

2.2 sudo python3 setup.py install --gpu
The installation log is very long and doing a lot of copy. It may recomple a new binary according to the following message.

Start to compile libarary.
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.0")
-- OpenCL include directory:/usr/include
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- filesystem
-- system
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cttsai/DataSci/mlc/LightGBM/python-package/build_cpp
Scanning dependencies of target _lightgbm
[ 3%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 7%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
[ 10%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 39%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 42%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 60%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 71%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 89%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 92%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[100%] Linking CXX shared library /home/cttsai/DataSci/mlc/LightGBM/python-package/lightgbm/lib_lightgbm.so
[100%] Built target _lightgbm
Install lib_lightgbm from: ['lightgbm/lib_lightgbm.so', '../lib_lightgbm.so']
running build

cttsai1985 on 21 Jul 2017

@cttsai1985
it seems there are some problem of your local opencl env.
Can you run the example by the gpu-version cli LightGBM ?

guolinke on 22 Jul 2017

@huanzhang12
any idea about this error ?

OSError: /usr/local/lib/python3.5/dist-packages/lightgbm/lib_lightgbm.so: symbol clCreateCommandQueueWithProperties, version OPENCL_2.0 not defined in file libOpenCL.so.1 with link time reference

guolinke on 22 Jul 2017

@guolinke as you wish.
../../lightgbm config=train.conf data=binary.train valid=binary.test device=gpu
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Loading weights...
[LightGBM] [Info] Loading weights...
[LightGBM] [Info] Finished loading data in 0.189428 seconds
[LightGBM] [Info] Number of positive: 3716, number of negative: 3284
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 6143
[LightGBM] [Info] Number of data: 7000, number of used features: 28
[LightGBM] [Info] Using GPU Device: GeForce GTX 1080, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 28 dense feature groups (0.19 MB) transfered to GPU in 0.012709 secs. 0 sparse feature groups.
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] Size of histogram bin entry: 12
[LightGBM] [Info] 28 dense feature groups (0.15 MB) transfered to GPU in 0.034523 secs. 0 sparse feature groups.
[LightGBM] [Info] Trained a tree with leaves=63 and max_depth=10
[LightGBM] [Info] Iteration:1, training auc : 0.778676
[LightGBM] [Info] Iteration:1, training binary_logloss : 0.669124
[LightGBM] [Info] Iteration:1, valid_1 auc : 0.716702
[LightGBM] [Info] Iteration:1, valid_1 binary_logloss : 0.674992
[LightGBM] [Info] 10.432616 seconds elapsed, finished iteration 1
[LightGBM] [Info] Trained a tree with leaves=63 and max_depth=11
[LightGBM] [Info] Iteration:2, training auc : 0.80241
[LightGBM] [Info] Iteration:2, training binary_logloss : 0.651141
[LightGBM] [Info] Iteration:2, valid_1 auc : 0.740132
[LightGBM] [Info] Iteration:2, valid_1 binary_logloss : 0.659703
[LightGBM] [Info] 19.234123 seconds elapsed, finished iteration 2
[LightGBM] [Info] Trained a tree with leaves=63 and max_depth=9
[LightGBM] [Info] Iteration:3, training auc : 0.824324
[LightGBM] [Info] Iteration:3, training binary_logloss : 0.635504
[LightGBM] [Info] Iteration:3, valid_1 auc : 0.765649
[LightGBM] [Info] Iteration:3, valid_1 binary_logloss : 0.647252
[LightGBM] [Info] 26.543888 seconds elapsed, finished iteration 3

cttsai1985 on 22 Jul 2017

Two OpenCL libraries are installed on this system. One is provided by Nvidia (version 1.2), and another one is a generic installable client driver (ICD) OpenCL provided by OS (version 2.0). I guess the reason is that LightGBM seems to be linked to the system OpenCL (2.0), but a LD_LIBRARY_PATH setting causes the Nvidia OpenCL to be loaded first, and is incompatible with library that LightGBM was linked with.

The problem with the --gpu flag is that we can't specify OpenCL library path. So first clean up everything, then run sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ .. to build LightGBM manually, and then run sudo python3 setup.py install --precompile. Don't run sudo python3 setup.py install --gpu, as this will link to the default (incorrect) OpenCL library.

@cttsai1985 For further debugging this problem, you can try to run ldd with lib_lightgbm.so and see which libraries it links to. You can also set LD_DEBUG=libs and run lightgbm CLI and see the linker debugging information. Make sure lightgbm is using the same OpenCL library as the one cmake finds during compilation time.

huanzhang12 on 22 Jul 2017

👍2

Thanks @huanzhang12

@cttsai1985 before you using huanzhuang12's solution, you should delete .so file in LightGBM/python-package (or LightGBM/python-package/lightgbm) folder.
and you may need to uninstall lightgbm first by pip uninstall lightgbm

guolinke on 22 Jul 2017

@huanzhang12
To avoid using -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/, can we set some environment variables for OpenCL_LIBRARY and OpenCL_INCLUDE_DIR ?

guolinke on 22 Jul 2017

Thanks @huanzhang12 @guolinke

In fact, the installation instruction mentioned by @huanzhang12 is how I built LightGBM w/ GPU currently. Without addling these paths I can not pass the cmake. That is why I had already followed the issue #458 by "sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ .." to get it worked. And you've all see the results using "sudo python3 setup.py install --precompile" after successful compiled. It is there https://github.com/Microsoft/LightGBM/issues/715#issuecomment-316996166

Regarding to the ldd part:
ldd lib_lightgbm.so

linux-vdso.so.1 =>  (0x00007ffff41fc000)
libOpenCL.so.1 => /usr/local/cuda-8.0/lib64/libOpenCL.so.1 (0x00007f17e64b8000)
libboost_filesystem.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0 (0x00007f17e627f000)
libboost_system.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0 (0x00007f17e607a000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f17e5cf8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f17e59ef000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f17e57cc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f17e55b6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f17e5399000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f17e4fce000)
/lib64/ld-linux-x86-64.so.2 (0x0000558975e03000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f17e4dca000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f17e4bc2000)

cttsai1985 on 22 Jul 2017

@cttsai1985
It can work if you manually replace the lib_lightgbm.so file ?
if yes, using --precomplie should work as well. Maybe you can try to delete the LightGBM folder and re-clone then re-build again.

guolinke on 22 Jul 2017

Before this time I remove LightGBM packages in python by a solution found in stacked-overflow:

sudo python3 setup.py install --precompile --record files.txt
cat files.txt | sudo xargs rm -rf

Since the lightgbm lightgbm-2.0.4-py3.5.egg-info folder I would going to replace it later so I deleted both of them anyway this time. Maybe there was something mess around.

sudo python3 setup.py install --precompile --record files.txt
cat files.txt | sudo xargs rm -rf
cd /usr/local/lib/python3.5/dist-packages/
sudo rm -r -f lightgbm lightgbm-2.0.4-py3.5.egg-info/

I re-installed by the following steps:

git clone --recursive https://github.com/Microsoft/LightGBM
cd ./LightGBM
mkdir build; cd build
sudo cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda-8.0/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda-8.0/include/ ..
sudo make -j$(nproc)
cd ../python-package; sudo python3 setup.py install --precompile

cttsai1985 on 22 Jul 2017

👍9

I successly install lightgbm gpu version like this link. but when I use lightgbm in jupyter lab, there is a

ModuleNotFoundError: No module named 'lightgbm'

How could I solve this problem? do you have any advise? @guolinke