Lightgbm: [R-package] I never could make the library run any simple model. The session always exploded

[danielmarcelino]

After installed lightGBM, I never could run a simple model. The R session always abort after I press enter to fit the model, even the very simple ones as those provided in the demo section. I've two MacBook pro with different power and operating systems, but in none of them I could use the library. In a server, running a linux it was so simple to make it work properly though.

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.2

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] lightgbm_2.3.1 R6_2.4.1      

loaded via a namespace (and not attached):
[1] compiler_3.6.3    tools_3.6.3       data.table_1.12.6 jsonlite_1.6.1  

[jameslamb]

@danielmarcelino Could you provide us with a bit more information?

How did you install LightGBM's R package? Looking for the specific commands you ran and which compiler you used (if you know it).

If you start from a completely clean session, do you still see this issue?

I see you're running version 2.3.1. Is it possible for you to upgrade to the current version on master? We have had a lot of activity on the R package since 2.3.1 was released in November and it's possible that we've already fixed the issue you're encountering.

[danielmarcelino]

Hi @jameslamb, thanks for asking.
Yes, the issue always happens even using a completely new session. The following it the installation verbose. Am I doing anything wrong? I just repeated the process with the 2.3.2 and got the same screen after trying to fit a toy model.

packageVersion("lightgbm")
[1] ‘2.3.2’

daniels-MacBook-Pro-2:~ marcelino$ cd LightGBM
daniels-MacBook-Pro-2:LightGBM marcelino$ Rscript build_r.R
* checking for file ‘lightgbm_r/DESCRIPTION’ ... OK
* preparing ‘lightgbm’:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory ‘lightgbm/src/compute/test’ is empty
* looking to see if a ‘data/datalist’ file should be added
* building ‘lightgbm_2.3.2.tar.gz’

* installing to library ‘/Library/Frameworks/R.framework/Versions/3.6/Resources/library’
* installing *source* package ‘lightgbm’ ...
** using staged installation
** libs
installing via 'install.libs.R' to /Library/Frameworks/R.framework/Versions/3.6/Resources/library/00LOCK-lightgbm/00new/lightgbm
-- The C compiler identification is AppleClang 11.0.3.11030032
-- The CXX compiler identification is AppleClang 11.0.3.11030032
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
R version passed into FindLibR.cmake: 3.6.3
-- Found LibR: /Library/Frameworks/R.framework/Versions/3.6/Resources  
-- LIBR_EXECUTABLE: /usr/local/bin/R
-- LIBR_INCLUDE_DIRS: /Library/Frameworks/R.framework/Versions/3.6/Resources/include
-- LIBR_CORE_LIBRARY: /Library/Frameworks/R.framework
-- Found OpenMP_C: -Xclang -fopenmp (found version "3.1") 
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "3.1") 
-- Found OpenMP: TRUE (found version "3.1")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /private/var/folders/nv/ntvjypdj4tqbgznkrcfhzm240000gn/T/RtmpyhwypF/R.INSTALL180625293b148/lightgbm/src/build
Scanning dependencies of target _lightgbm
[  3%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 34%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 37%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 40%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 43%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 56%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 59%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 62%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 68%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 71%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 84%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
[100%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
[1] "Found library file: /var/folders/nv/ntvjypdj4tqbgznkrcfhzm240000gn/T//RtmpyhwypF/R.INSTALL180625293b148/lightgbm/src/lib_lightgbm.so to move to /Library/Frameworks/R.framework/Versions/3.6/Resources/library/00LOCK-lightgbm/00new/lightgbm/libs"
[1] "Removing 'build/' directory"
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (lightgbm)

[jameslamb]

Thanks! This looks exactly like our recommended approach to installing the R package right now, and I don't see any obvious issues in the logs.

Can you share the exact R code that you're running which is resulting in the error? I can kind of see it in your first screenshot but I want too be sure there aren't other things you've run in the console in that session (just trying to rule things out).

I believe we haven't tested with Catalina yet, so maybe there is something Catalina-specific that is causing this.

I want to find the issue but I also want to be sure you can get back to work...do you have gcc available? (you can check by running gcc --version)

If you do, you could try rebuilding with gcc? On my Mac, I do this in a terminal (per our docs)

export CXX=g++-8
export CC=gcc-8
Rscript build_r.R

I use g++-8 because that's the version that I have. You may need to change the command above to whatever version you have.

[danielmarcelino]

Sure, thanks fro replaying. Please consider that I upgraded from Mojave to Catalina. After that I've seen many mac users telling that they can't compile a C program on a Mac after upgrading to Catalina. Perhaps Apple just made our lives more difficult since then.

daniels-MacBook-Pro-2:~ marcelino$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
daniels-MacBook-Pro-2:~ marcelino$ 

The reproduction example is this:

library(lightgbm)
data(agaricus.train, package = "lightgbm")
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
data(agaricus.test, package = "lightgbm")
test <- agaricus.test
dtest <- lgb.Dataset.create.valid(dtrain, test$data, label = test$label)
params <- list(objective = "regression", metric = "l2")
valids <- list(test = dtest)
model <- lgb.train(
  params = params
  , data = dtrain
  , nrounds = 10L
  , valids = valids
  , min_data = 1L
  , learning_rate = 1.0
  , early_stopping_rounds = 5L
)

[danielmarcelino]

After rebuilding with the following parameters the problem was apparently solved. I'll do some testing tomorrow, but at least it worked now. Thanks.

export CXX=g++-9
export CC=gcc-9
Rscript build_r.R

[jameslamb]

@danielmarcelino thanks very much! My best guess is that we have a Catalina-specific issue 😬 .

I'm glad that gcc is working for you! I'll need to experiment on Catalina and see if I can find the issue. Thanks very much for your bug report.

[jameslamb]

@danielmarcelino have you been having issues with other C++ projects on your Mac? A friend of mine said he's been unable to build R packages with C++ code on Catalina for a bit, and this fixed it: https://stackoverflow.com/questions/59071881/problems-with-c-and-gems-on-osx-catalina/59072909#59072909

[danielmarcelino]

Yes, I had. I saw some error mensagens while installing Rcpp package, for instance. But the issue with LightGBM was more subtle once it was perfect compiled and installed, but was not working. Thanks a lot for you guidance here.

[StrikerRUS]

@jameslamb Hmm, seems that it is a general Catalina problem. Nothing can be done from LightGBM side. Can we close this issue?

[jameslamb]

@danielmarcelino I'm going to close this issue for now, since it does seem like an issue with Catalina and since you were able to use g++ to get a successful installation.

I do have an experimental setup that ignores CMake and uses the toolchain commonly used by CRAN, and a few who've tested it have reported success on Catalina. It's here if you want to try: https://github.com/jameslamb/LightGBM/pull/15

./build-cran-package.sh
R CMD INSTALL lightgbm_2.3.2.tar.gz

It is VERY experimental and not something we're officially supporting yet, but when we eventually add it here it may help.

^ @StrikerRUS I'm mentioning this just to document that I think any incompatibility with Catalina might be limited to the installation path that uses CMake (Rscript build_r.R) and doesn't mean that the package we prepare for CRAN will fail for Catalina users.