Lightgbm: Failed install to Mac OS from RStudio and from terminal

[pbpearman]

I have been trying to install to Mac, and have tried several possible solutions, but nothing seems to work. Following issue 1407, I tried installing from the RStudio console:

> install_github("Microsoft/LightGBM", subdir = "R-package")
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing via 'install.libs.R' to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm
Error in eval(ei, envir) : Copying CMakeLists failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm’
Error in i.p(...) : 
(converted from warning) installation of package ‘/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T//Rtmp9TFE7v/file8de29d49380/lightgbm_2.2.3.tar.gz’ had non-zero exit status

I also tried from within the LightGBM folder and got completely a different error message:

> Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /Users/bgppermp/Downloads/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : 
must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: <Anonymous> ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8 
Execution halted

So, I'm at a dead end in both cases. I have tried to set the CXX and CC flags to (g++ or g++-8) and (gcc and gcc-8). Your suggestions would be appreciated because with the differing error messages, this is now beyond me.

Environment info

Operating System: Mac OS X 10.13.6

CPU/GPU model: 3 GHz 8-Core Intel Xeon E5

C++/Python/R version: R v. 3.5.1

rvm_bin_path=/Users/bgppermp/.rvm/bin
TERM_PROGRAM=Apple_Terminal
GEM_HOME=/Users/bgppermp/.rvm/gems/ruby-2.4.2
SHELL=/bin/bash
TERM=xterm-256color
IRBRC=/Users/bgppermp/.rvm/rubies/ruby-2.4.2/.irbrc
TMPDIR=/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.nAzjTbF0wY/Render
TERM_PROGRAM_VERSION=404
OLDPWD=/Users/bgppermp/Downloads/LightGBM
MY_RUBY_HOME=/Users/bgppermp/.rvm/rubies/ruby-2.4.2
TERM_SESSION_ID=7A5FF598-D214-4F1C-A10E-D177BA13BFB4
USER=bgppermp
_system_type=Darwin
rvm_path=/Users/bgppermp/.rvm
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.wAGuhJZysJ/Listeners
rvm_prefix=/Users/bgppermp
PATH=/usr/local/opt/curl/bin:/usr/local/bin:/Users/bgppermp/.rvm/gems/ruby-2.4.2/bin:/Users/bgppermp/.rvm/gems/ruby-2.4.2@global/bin:/Users/bgppermp/.rvm/rubies/ruby-2.4.2/bin:/usr/local/opt/[email protected]/bin:/Users/bgppermp/anaconda/bin:/opt/local/bin:/opt/local/sbin:/Library/Frameworks/GDAL.framework/Programs:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/ncbi/blast/bin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/opt/X11/bin:~/bin:/Users/bgppermp/.rvm/bin
PWD=/Users/bgppermp/Downloads/LightGBM/R-package/src
LANG=en_US.UTF-8
_system_arch=x86_64
XPC_FLAGS=0x0
BLASTDB=/Volumes/lev_pearman/blastdb
_system_version=10.13
CXX=/usr/local/bin/g++
XPC_SERVICE_NAME=0
rvm_version=1.29.3 (latest)
SHLVL=1
HOME=/Users/bgppermp
LOGNAME=bgppermp
GEM_PATH=/Users/bgppermp/.rvm/gems/ruby-2.4.2:/Users/bgppermp/.rvm/gems/ruby-2.4.2@global
CC=/usr/local/bin/gcc
DISPLAY=/private/tmp/com.apple.launchd.gPxyTK0544/org.macosforge.xquartz:0
RUBY_VERSION=ruby-2.4.2
SECURITYSESSIONID=186a9
_system_name=OSX
_=/usr/bin/env

Error message

Reproducible examples

Steps to reproduce

1.
2.
3.

[pbpearman]

Update. Following this post and adding some links, I got a successful build following this post.
I do:
`bgppermp$ cd R-package/
uhtred:R-package bgppermp$ R CMD INSTALL --build . --no-multiarch

• installing to library ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library’
• installing source package ‘lightgbm’ ...
  ** libs
  make: Nothing to be done for `all'.
  installing via 'install.libs.R' to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm
  Error in eval(ei, envir) : Copying CMakeLists failed
• removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm’
  `
  Can someone enlighten me as to how to fix this, which seems to mean that R can't find CMakeLists?

[Laurae2]

@pbpearman Did you build the package first using the build_r.R script? (it builds and installs, actually)

[pbpearman]

@Laurae2 Yes, I tried, but it doesn't complete.
In the second example in the original post, I showed what happened when using that script from within the LightGBM folder. The fairly cryptic error message was:
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
This might provide another approach to trouble-shooting, but I looked at the build_r.R script and did not find anything I could relate to the error message. I tried again from a fresh clone, but got the same result.

[thebamoor]

Tried to install LightGBM; kept on getting "Copying CMakeLists failed" messages

[ben519]

I too am getting this error, on mac osx 10.14.1 running R 3.5.1. Perhaps this is related to #1702?

Bens-MacBook-Pro-2017:LightGBM Ben$ Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /Users/Ben/Documents/Projects/R/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : 
  must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: <Anonymous> ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8 
Execution halted
Bens-MacBook-Pro-2017:LightGBM Ben$

[Laurae2]

Ping @jameslamb : DLL on Mac or generic error?

[Laurae2]

@pbpearman Can you try lgbdl to see if it provides an another error message? Make sure to use a fresh R session and to close all other R sessions.

[ben519]

Hmm, it seems that the line devtools::document(pkg = "lightgbm_r") in build_r.R is the culprit. If I run the script without that line, lightgbm appears to install successfully.

[Laurae2]

@ben519 Sounds like the R session is not cleaned after generating the documentation because this DLL error appears when trying to install a package with a corresponding .so/.dll already loaded in memory (here, installing LightGBM while LightGBM is being used).

[pbpearman]

@Laurae2, @ben519 Here is what I get when running lgbdl from a fresh R session:
`> lgbdl::lgb.dl(compiler = "gcc")
Cloning into 'LightGBM'...
Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'compute'
Cloning into '/private/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/Rtmpw8LARd/LightGBM/compute'...
Submodule path 'compute': checked out '509ebe4a9282eec8a92c65ce3bbc1925f1fdbe07'
Already on 'master'
Your branch is up to date with 'origin/master'.

• installing source package ‘lightgbm’ ...
  ** libs
  /usr/local/bin/gcc -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I/usr/local/include -O3 -Wall -pipe -pedantic -std=gnu99 -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c R_init.c -o R_init.o
  /usr/local/bin/gcc -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o lightgbm.so R_init.o -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
  ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
  installing via 'install.libs.R' to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm
  -- The C compiler identification is AppleClang 10.0.0.10001145
  -- The CXX compiler identification is AppleClang 10.0.0.10001145
  -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
  -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
  -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CMake Error at /Applications/CMake.app/Contents/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
  Call Stack (most recent call first):
  /Applications/CMake.app/Contents/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /Applications/CMake.app/Contents/share/cmake-3.12/Modules/FindOpenMP.cmake:466 (find_package_handle_standard_args)
  CMakeLists.txt:78 (find_package)

-- Configuring incomplete, errors occurred!
See also "/private/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/Rtmpw8LARd/LightGBM/lightgbm_r/src/build/CMakeFiles/CMakeOutput.log".
make: * No rule to make target `_lightgbm'. Stop.
Error in eval(ei, envir) : Cannot find lib_lightgbm.so

• removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm’
  [1] FALSE
  Warning message:
  In install.packages(file.path(lgb_git_dir, "LightGBM", "lightgbm_r"), :
  installation of package ‘/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T//Rtmpw8LARd/LightGBM/lightgbm_r’ had non-zero exit status`

Also, I note this:
bgppermp$ echo |cpp -fopenmp -dM |grep -i open

define _OPENMP 201511

[jameslamb]

@pbpearman it looks like clang is not finding openmp. Can you humor me and try this as we note in the installation guide?

brew install libomp

I also have never tried from within RStudio but given my experience with that tool, I echo @Laurae2 's concern that something is being left behind.

If you run

Rscript build_r.R

from the command line, this would not be a concern because every Rscript call is a new R session by definition.

[1psmith]

I'm getting the same thing both with the command line and with Rstudio. (Windows install)

C:\LightGBM>Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in C:\LightGBM\lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) :
must specify DLL via a "DLLInfo" object. See getLoadedDLLs()
Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8
Execution halted

[1psmith]

Update: Ran this on my digital ocean Ubuntu 18.04 server and got exactly the same thing.

$ Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /home/pres/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) :
must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8
Execution halted

[Laurae2]

@1psmith Comment the line 49 of build_r.R, I am also getting the error.

[Laurae2]

@1psmith You can try also #1868 fix.

git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
git checkout fbd98d5
Rscript build_r.R

[1psmith]

Update: Ran this on my digital ocean Ubuntu 18.04 server and got exactly the same thing.

$ Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /home/pres/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) :
must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8
Execution halted

@1psmith You can try also #1868 fix.

git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
git checkout fbd98d5
Rscript build_r.R

Yes! This worked perfectly. Thanks!!!

[pbpearman]

@Laurae2 @jameslamb @1psmith Here, for entertainment, I installed libomp anew. Then I try @1psmith 's idea.
`bgppermp$ brew install libomp
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> Updated Formulae
jump laszip mkl-dnn vsts-cli vtk youtube-dl

==> Downloading https://homebrew.bintray.com/bottles/libomp-6.0.1.high_sierra.bo
==> Downloading from https://akamai.bintray.com/04/046f659ad8a2cc336049a1e7f7be0

################################################################## 100.0%

==> Pouring libomp-6.0.1.high_sierra.bottle.tar.gz
Error: The brew link step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink lib/libgomp.a
Target /usr/local/lib/libgomp.a
already exists. You may want to remove it:
rm '/usr/local/lib/libgomp.a'

To force the link and overwrite all conflicting files:
brew link --overwrite libomp

To list all files that would be deleted:
brew link --overwrite --dry-run libomp

Possible conflicting files are:
/usr/local/lib/libgomp.a
/usr/local/lib/libgomp.dylib -> /usr/local/lib/libomp.dylib
/usr/local/lib/libiomp5.dylib -> /usr/local/lib/libomp.dylib
/usr/local/lib/libomp.dylib
==> Caveats
On Apple Clang, you need to add several options to use OpenMP's front end
instead of the standard driver option. This usually looks like
-Xpreprocessor -fopenmp -lomp

You might need to make sure the lib and include directories are discoverable
if /usr/local is not searched:

-L/usr/local/opt/libomp/lib -I/usr/local/opt/libomp/include

For CMake, the following flags will cause the OpenMP::OpenMP_CXX target to
be set up correctly:
-DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include" -DOpenMP_CXX_LIB_NAMES="omp" -DOpenMP_omp_LIBRARY=/usr/local/opt/libomp/lib/libomp.dylib
==> Summary
🍺 /usr/local/Cellar/libomp/6.0.1: 12 files, 1.2MB

bgppermp$ brew link --overwrite libomp
Linking /usr/local/Cellar/libomp/6.0.1... 8 symlinks created

bgppermp$ git checkout fbd98d5
Note: checking out 'fbd98d5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at fbd98d5 Does not need to fix doc. Fixes #1852

bgppermp$ git branch

• (HEAD detached at fbd98d5)
  master
  bgppermp$ Rscript build_r.R

Worked!
Although the link to libomp was present, re-installing and overwriting the old link seems to have fixed the situation. Must have been broken.
Thanks!

`

[ben519]

For reference, I think the root of this issue is discussed here.