LightGBM and gcc 8 in MacOS: `Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib`

[jrzaurin]

For Mac Users

Version 10.13.4
Python 3.6.5

The 2nd of May gcc-8 was released, which is a major released.

Therefore, if, since then, you run

brew update
brew upgrade

gcc-8 version will be installed:

➜  ~ gcc-8 --version
gcc-8 (Homebrew GCC 8.1.0) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ISSUE

If the lightGBM version you have was built with gcc-7, when you try to load it now you might get:

OSError: dlopen(/usr/local/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib
  Referenced from: /usr/local/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found

SOLUTION

I came around this error by installing it from source doing:

pip uninstall lightgbm
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
export CXX=g++-8 CC=gcc-8
mkdir build ; cd build
cmake ..
make -j4

and then

pip install --no-binary :all: lightgbm

In [1]: import lightgbm

In [2]: lightgbm.__version__
Out[2]: '2.1.1'

By the way, similar issue will happen to xgboost.

If someone thinks this is more for stackoverflow I am happy to remove it from here immediately

Cheers

[guolinke]

okay, I see.
The macos binaries is built by gcc7, which cannot be used for gcc8.
Built from source can solve it.
@StrikerRUS should we update it to the gcc8 version?

[Laurae2]

@guolinke by default this should be now gcc-8 for macOS.

[guolinke]

@Laurae2 thanks.
However, our latest binaries is built before the update of gcc 8. Maybe we should release the new binaries recently.

[StrikerRUS]

Yeah, after this PR https://github.com/Microsoft/LightGBM/pull/1357 Travis artifact for macOS is built by g++-8. However, the latest release on PyPI 2.1.1 contains the version for g++-7.

Since there is an incompatibility between g++-7 and g++-8, I think we should create a note about it and suggest to install 2.1.1 version for those who use g++-7.

Or there is another variant. I'm not a mac user, so I don't know can g++-8 be installed on any version of macOS? If not, then we can build two versions of Python wheel. For instance,
lightgbm-2.1.1-py2.py3-none-macosx_10_9_x86_64.macosx_10_10_x86_64.macosx_10_11_x86_64.macosx_10_12_x86_64.whl with g++-7 and
lightgbm-2.1.1-py2.py3-none-macosx_10_13_x86_64.whl with g++-8.

[StrikerRUS]

Travis images we run aren't the latest version of macOS, but g++-8 can be successfully installed on them. So, the variant with separate Python wheels is unacceptable.

I think we just can document this behavior.

[gouravsb17]

I also found a small hack for OsX High Sierra which has by default gcc 8, and lightgbm required gcc 7.
I build the lightgbm using gcc7 but the error still persisted with
OSError: dlopen(/usr/local/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib
So I decided to investigate what is in the /usr/local/opt/gcc/lib/gcc folder. .... Interestingly it had a folder called 8( as default is gcc8).
Then I tried compiling again the lightgbm with gcc7 but to no help.
Finally, I made a small hack I copied the contents in a similar folder /usr/local/opt/gcc@7/lib/gcc/7 to /usr/local/opt/gcc/lib/gcc/ and voila it worked!!!
sudo cp -a /usr/local/opt/gcc@7/lib/gcc/7 /usr/local/opt/gcc/lib/gcc/

Here is a great doc for lightgbm if u want to use the highPerformance using MPI or GPU or HDFS then refer here:
https://media.readthedocs.org/pdf/lightgbm/latest/lightgbm.pdf

[KaiWangCanada]

Most developers use Mac, and lgb can't be installed on mac? Why did you do that?

[xavierbourretsicotte]

EDIT

The following post solves it for me (from Xgboost)

https://github.com/dmlc/xgboost/issues/1945#issuecomment-290606179

export PATH=/usr/bin:$PATH

Seems to be an issue with my environment rather than the repo... thanks anyways

---

On MAC OSX El Capitan

I am unable to run make .. it seems it doesn't find or like my compiler. When I run which gcc I get
/usr/local/bin/gcc and /usr/local/bin/gcc --version gives

gcc (Homebrew GCC 8.2.0) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Now running the commands in terminal

export CXX=g++-8 CC=gcc-8
mkdir build
cd build
cmake ..

Yields the following message:

-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- Check for working C compiler: /usr/local/bin/gcc-8
-- Check for working C compiler: /usr/local/bin/gcc-8 -- broken
CMake Error at /usr/local/Cellar/cmake/3.12.2/share/cmake/Modules/CMakeTestCCompiler.cmake:52 (message):
The C compiler
"/usr/local/bin/gcc-8"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /Users/User/LightGBM/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_4e2c0/fast"
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cmTC_4e2c0.dir/build.make CMakeFiles/cmTC_4e2c0.dir/build
Building C object CMakeFiles/cmTC_4e2c0.dir/testCCompiler.c.o
/usr/local/bin/gcc-8    -o CMakeFiles/cmTC_4e2c0.dir/testCCompiler.c.o   -c /Users/User/LightGBM/build/CMakeFiles/CMakeTmp/testCCompiler.c
FATAL:/opt/local/bin/../libexec/as/x86_64/as: I don't understand 'm' flag!
make[1]: *** [CMakeFiles/cmTC_4e2c0.dir/testCCompiler.c.o] Error 1
make: *** [cmTC_4e2c0/fast] Error 2

CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:7 (PROJECT)

[fx86]

I'm getting the following error after being able to successfully install lightgbm, even though libomp is already installed. Does anyone have any suggestions ?
~/Downloads/LightGBM/build  python -c "import lightgbm"  18:12:37   master ? /Users/user121/anaconda3/lib/python3.6/site-packages/lightgbm/__init__.py:46: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_9.4.1) compiler. This means that in case of installing LightGBM from PyPI via the ``pip install lightgbm`` command, you don't need to install the gcc compiler anymore. Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler. You can install the OpenMP library by the following command: ``brew install libomp``. "You can install the OpenMP library by the following command: ``brew install libomp``.", UserWarning)

[guolinke]

@fx86 I think it is a warning, not an error, and you are able to run lightgbm, right ?
ping @StrikerRUS to confirm.

[fx86]

Yes, it loads. Any clue why that warning ?

Also, another concern is - once imported into a Jupyter notebook, the module seems to have very basic attributes. Like the following lines show:

import lightgbm as lgb
dir(lgb)

output: ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

[StrikerRUS]

@fx86 Yeah, it's just a UserWarning to inform users about the changed requirements to run LightGBM successfully.

This warning will be removed in the future.
If everything OK for you, ignore this warning.

[liwen611]

For Mac Users

Version 10.13.4
Python 3.6.5

The 2nd of May gcc-8 was released, which is a major released.

Therefore, if, since then, you run

brew update
brew upgrade

gcc-8 version will be installed:

➜  ~ gcc-8 --version
gcc-8 (Homebrew GCC 8.1.0) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ISSUE

If the lightGBM version you have was built with gcc-7, when you try to load it now you might get:

OSError: dlopen(/usr/local/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib
  Referenced from: /usr/local/lib/python3.6/site-packages/lightgbm/lib_lightgbm.so
  Reason: image not found

SOLUTION

I came around this error by installing it from source doing:

pip uninstall lightgbm
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
export CXX=g++-8 CC=gcc-8
mkdir build ; cd build
cmake ..
make -j4

and then

pip install --no-binary :all: lightgbm

```
In [1]: import lightgbm

In [2]: lightgbm.__version__
Out[2]: '2.1.1'
`` By the way, similar issue will happen toxgboost`.

If someone thinks this is more for stackoverflow I am happy to remove it from here immediately

Cheers

when I get to the step $ make -j4
I get the following error:
make: * No targets specified and no makefile found. Stop.
How do I proceed ?

[unnir]

have the same problem on my mac, any updates?

[chivee]

@unnir , would you give a shot on the new options in https://github.com/Microsoft/LightGBM/pull/1975#issuecomment-458096716?

[jander081]

If you don't mind doing a conda install, try:

import sys
!conda install --yes --prefix {sys.prefix} -c conda-forge lightgbm

This resolved the problem for me (run in a jupyter notebook cell)

[fraxedas]

To solve the problem I installed the libomp using brew:
brew install libomp

source: https://stackoverflow.com/a/55958281/1074245

[sshleifer]

I have everything working, and libomp installed but still get the please brew install libomp warning when I import lightgbm on OSX. Any way to fix the warning?

[StrikerRUS]

@sshleifer Please read this https://github.com/microsoft/LightGBM/issues/1369#issuecomment-428971946.

[xiaoyaoyang]

Env OSX, export CXX=g++-9 CC=gcc-9 solved my issue

I think one should do brew install gcc and then brew info gcc to check the version. Mine return

gcc: stable 9.1.0 (bottled), HEAD
GNU compiler collection

so I use gcc-9 instead of gcc-8 in this thread

[josephbak]

To solve the problem I installed the libomp using brew:
brew install libomp

source: https://stackoverflow.com/a/55958281/1074245

This works for me.

[subhasiskhatua]

For Mac OS, this worked for me:
https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html

macOS
On macOS LightGBM can be built using CMake and Apple Clang or gcc.

Apple Clang
Only Apple Clang version 8.1 or higher is supported.

Install CMake (3.12 or higher):

brew install cmake
Install OpenMP:

brew install libomp
Run the following commands:

git clone --recursive https://github.com/microsoft/LightGBM ; cd LightGBM
mkdir build ; cd build

For Mojave (10.14)
cmake
-DOpenMP_C_FLAGS="-Xpreprocessor -fopenmp -I$(brew --prefix libomp)/include"
-DOpenMP_C_LIB_NAMES="omp"
-DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp -I$(brew --prefix libomp)/include"
-DOpenMP_CXX_LIB_NAMES="omp"
-DOpenMP_omp_LIBRARY=$(brew --prefix libomp)/lib/libomp.dylib
..

For High Sierra or earlier (<= 10.13)
cmake ..

make -j4
gcc
Install CMake (3.2 or higher):

brew install cmake
Install gcc:

brew install gcc
Run the following commands:

git clone --recursive https://github.com/microsoft/LightGBM ; cd LightGBM
export CXX=g++-7 CC=gcc-7 # replace "7" with version of gcc installed on your machine
mkdir build ; cd build
cmake ..
make -j4

[saravanansaminathan]

The solution is simple and it is provided in pypi lightgbm
Link : https://pypi.org/project/lightgbm/
Solution:
For _mac_ users,

git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM/python-package
export CXX=g++-7 CC=gcc-7  # macOS users, if you decided to compile with gcc, don't forget to specify compilers (replace "7" with version of gcc installed on your machine)
python setup.py install --openmp-include-dir=/usr/local/opt/libomp/include/ --openmp-library=/usr/local/opt/libomp/lib/libomp.dylib

Note: If you don't have libomp install through brew(brew install libomp)

[BroderickHigby]

So the latest solution seems to not require cmake. You just need libomp and pip:

brew install libomp
pip3 install lightgbm

[StrikerRUS]

Please refer to the latest official installation instructions:

• CLI / dynamic library: https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#macos
• Python: https://github.com/microsoft/LightGBM/tree/master/python-package#installation
• R: https://github.com/microsoft/LightGBM/tree/master/R-package#installation

Locking this conversation to avoid any further confusion. Feel free to open new issue if you still cannot install LightGBM on macOS after reading links above.