Lightgbm: [R-package] Installation issue for LightGBM in R 4.0.1 (Linux)

Created on 11 Jul 2020 · 16Comments · Source: microsoft/LightGBM

I followed the exact installation guide, but I get the following error at the very end for Light GBM in R 4.0.1 in Linux.

I have R6 installed but not sure what the issue could be.

[100%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
Found library file: /tmp/Rtmp5AUfVk/R.INSTALLbafd4d2aa1db/lightgbm/src/lib_lightgbm.so to move to /mnt/home/u044338/R/x86_64-pc-linux-gnu-library/4.0/lightgbm/libs
Removing 'build/' directory
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
Warning: package ‘R6’ was built under R version 4.0.1
Warning: namespace ‘processx’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Error: package or namespace load failed for ‘R6’ in rbind(info, getNamespaceInfo(env, "S3methods")):
 number of columns of matrices must match (see arg 2)
Error : package ‘R6’ could not be loaded
ERROR: lazy loading failed for package ‘lightgbm’

r-package

Source

sriharitn

All 16 comments

Thanks for using LightGBM! Could you please share some additional information?

operating system and version
compiler and version
the full log from installation (everything printed when you ran Rscript build_r.R)

If you can share that information, I'll look into this and try to see what's going on.

jameslamb on 11 Jul 2020

Please see below.

operating system and version:

platform       x86_64-pc-linux-gnu         
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
version.string R version 4.0.1 (2020-06-06)
nickname       See Things Now

Compiler and version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
Full Log from installation:

Rscript build_r.R
WARNING: ignoring environment value of R_HOME
* checking for file ‘/mnt/home/abc/LightGBM/lightgbm_r/DESCRIPTION’ ... OK
* preparing ‘lightgbm’:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory ‘lightgbm/src/compute/test’ is empty
* looking to see if a ‘data/datalist’ file should be added
* building ‘lightgbm_2.3.2.tar.gz’

* installing to library ‘/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0’
* installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- R version passed into FindLibR.cmake: 3.5.0
-- Found LibR: /usr/lib64/R  
-- LIBR_EXECUTABLE: /usr/bin/R
-- LIBR_INCLUDE_DIRS: /usr/include/R
-- LIBR_CORE_LIBRARY: /usr/lib64/R/lib/libR.so
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/Rtmp5AUfVk/R.INSTALLbafd4d2aa1db/lightgbm/src/build
Building lib_lightgbm
Scanning dependencies of target _lightgbm
[  3%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 16%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 19%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 22%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 29%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 45%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 48%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 51%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 54%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 61%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 70%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 74%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 83%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[100%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
Found library file: /tmp/Rtmp5AUfVk/R.INSTALLbafd4d2aa1db/lightgbm/src/lib_lightgbm.so to move to /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm/libs
Removing 'build/' directory
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
Warning: package ‘R6’ was built under R version 4.0.1
Warning: namespace ‘processx’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Error: package or namespace load failed for ‘R6’ in rbind(info, getNamespaceInfo(env, "S3methods")):
 number of columns of matrices must match (see arg 2)
Error : package ‘R6’ could not be loaded
ERROR: lazy loading failed for package ‘lightgbm’
* removing ‘/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm’
Error in .run_shell_command(install_cmd, install_args) : 
  Command failed with exit code: 1
Execution halted

sriharitn on 11 Jul 2020

Thanks! Did you recently update from an older version of R? Could you show me the output of these commands?

Rscript --version

Rscript --vanilla -e "print(.libPaths())"

I see other issues with a similar error, that seems to be caused by mixing R versions in your installed packages.

If you have old packages leftover from previous R versions, I think the fix would be to try this

remove.packages("R6")
install.packages("R6", repos = "http://cran.r-project.org")

jameslamb on 11 Jul 2020

👍1

If that doesn't work, you should also consider updating gcc (if possible). We expect LightGBM's core library to be compilable with gcc >=4.8.2, but that is not necessarily true for all R packages.

CRAN uses gcc 9.3 and 10.1 for testing Linux packages (https://cran.r-project.org/web/checks/check_flavors.html#r-devel-linux-x86_64-debian-gcc), so it's possible for a release of a package like R6 to make it onto CRAN even if it can't be built with gcc 4.x

jameslamb on 11 Jul 2020

Yes, I recently upgraded R and Rstudio from 3.5 to 4.0.

Please see below running Rscript.

Rscript --version
R scripting front-end version 3.5.0 (2018-04-23)

Rscript --vanilla -e "print(.libPaths())"

WARNING: ignoring environment value of R_HOME
[1] "/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0"
[2] "/mnt/home/libraries/library_2019-01-01"             
[3] "/usr/lib64/R/library"

sriharitn on 11 Jul 2020

👀1

Ah, then yes this is your problem. The output of Rscript --version tells me you still have R 3.5 installed.

The .libPaths() output tells me that you have multiple libraries of R packages. I'm guessing they are a mix of R 3.5 and R 4.0 packages.

I bet that if you run ls /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0, you will not see R6 listed.

I believe that if you remove R 3.5.0 (and those old library directories), then this should work as expected for you. I think that this problem is not specific to LightGBM, and that having this mix of versions is going to cause problems for you with other R packages.

If you want to keep R 3.5.0 around for some reason, I think you can force the installation of LightGBM (and other R packages!) to use R 4.0 by doing the following:

set environment variable R_HOME to a location for R 4.0. It is probably at a place like /usr/local/bin/R
make sure the location of R 4.0 is earlier on your PATH environment variable than R 3.5

jameslamb on 11 Jul 2020

👍1

Thank you for quick response.

I'll check with my Linux admin on Monday on changing the environment variable.

When I check the location below, I do see R6 installed for R 4.0.1. Strange.

ls /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0

askpass     cli         data.table  fansi      htmlwidgets  lgbdl      pillar       promises   remotes     sessioninfo  utf8     yaml
assertthat  clipr       desc        fs         httr         lifecycle  pkgbuild     ps         rex         stringi      vctrs
backports   commonmark  devtools    gh         ini          magrittr   pkgconfig    purrr      rlang       stringr      whisker
base64enc   covr        digest      git2r      jsonlite     markdown   pkgload      R6         roxygen2    sys          withr
BH          crayon      DT          glue       knitr        memoise    praise       rcmdcheck  rprojroot   testthat     xfun
brew        crosstalk   ellipsis    highr      later        mime       prettyunits  Rcpp       rstudioapi  tibble       xml2
callr       curl        evaluate    htmltools  lazyeval     openssl    processx     rematch2   rversions   usethis      xopen

sriharitn on 11 Jul 2020

oh interesting! Maybe the issue is in a dependency of R6 then. Either way, the issue definitely seems to be related to R 3.5 and R 4.0 mixed on the same system (see those issues I linked).

If you don't have admin access, you can try to work around this by passing a direct path. So instead of

Rscript build_r.R

you can try

/usr/local/bin/Rscript build_r.R

Where instead of /usr/local/bin/Rscript, put whatever location Rscript (for R 4.0) is installed

jameslamb on 11 Jul 2020

👍1

That makes sense. Thanks again.

sriharitn on 12 Jul 2020

I'm going to close this issue since it seems that it's not related to LightGBM directly. If you fully upgrade to R 4.0 and still experience this problem, please comment here and I'll re-open it.

Thanks for using LightGBM, and enjoy your weekend!

jameslamb on 12 Jul 2020

👍1

Hi James, Unfortunately I reran with Rscript with 4.0.1 I still get the same error.

/opt/R/4.0.1/bin/Rscript build_r.R 
WARNING: ignoring environment value of R_HOME
* checking for file ‘/mnt/home/abc/LightGBM/lightgbm_r/DESCRIPTION’ ... OK
* preparing ‘lightgbm’:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory ‘lightgbm/src/compute/test’ is empty
* looking to see if a ‘data/datalist’ file should be added
* building ‘lightgbm_2.3.2.tar.gz’

WARNING: ignoring environment value of R_HOME
* installing to library ‘/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0’
* installing *source* package ‘lightgbm’ ...
** libs
installing via 'install.libs.R' to /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- R version passed into FindLibR.cmake: 3.5.0
-- Found LibR: /usr/lib64/R  
-- LIBR_EXECUTABLE: /usr/bin/R
-- LIBR_INCLUDE_DIRS: /usr/include/R
-- LIBR_CORE_LIBRARY: /usr/lib64/R/lib/libR.so
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/RtmpDFqpms/R.INSTALL18ee2de00d8f/lightgbm/src/build
Building lib_lightgbm
Scanning dependencies of target _lightgbm
[  3%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
[  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 16%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 19%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 22%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 29%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 45%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 48%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 51%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 54%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 61%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 70%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 74%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 77%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 80%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 83%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[100%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
Found library file: /tmp/RtmpDFqpms/R.INSTALL18ee2de00d8f/lightgbm/src/lib_lightgbm.so to move to /mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm/libs
Removing 'build/' directory
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
Warning: package ‘R6’ was built under R version 4.0.1
Warning: namespace ‘processx’ is not available and has been replaced
by .GlobalEnv when processing object ‘’
Error: package or namespace load failed for ‘R6’ in rbind(info, getNamespaceInfo(env, "S3methods")):
 number of columns of matrices must match (see arg 2)
Error : package ‘R6’ could not be loaded
ERROR: lazy loading failed for package ‘lightgbm’
* removing ‘/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0/lightgbm’
Error in .run_shell_command(install_cmd, install_args) : 
  Command failed with exit code: 1
Execution halted

sriharitn on 13 Jul 2020

I still believe this issue is the result of mixing installations of R versions, per the links I shared.

Can you please share the new output of these commands?

Rscript --version

Rscript --vanilla -e "print(.libPaths())"

What happens if you just run this?

Rscript -e "library(R6)"

If your R version has been upgraded, have tried running this command I suggested?

remove.packages("R6")
install.packages("R6", repos = "http://cran.r-project.org")

jameslamb on 13 Jul 2020

👍1

you may be right on the multiple versions of R. Here is the output of

/opt/R/4.0.1/bin/Rscript --version
R scripting front-end version 4.0.1 (2020-06-06)

```
/opt/R/4.0.1/bin/Rscript --vanilla -e "print(.libPaths())"

[1] "/mnt/home/abc/R/x86_64-pc-linux-gnu-library/4.0"
[2] "/opt/R/4.0.1/lib64/R/library"

when I run this

/opt/R/4.0.1/bin/Rscript -e "library(R6)"
```
nothing happens. the command runs smoothly.

Yes, I removed and reinstalled R6 package.

I'm trying to work with my Linux Admin to resolve this and he seems to think that it has to do with have the right R paths. Hopefully, I would be able to resolve this and will surely report back.

Thanks again.

sriharitn on 13 Jul 2020

hmmmm ok, that is good that you're able to target the 4.0.1!

Look for the value of the R_LIBS and R_HOME environment variables, those will be clues.

Now that I think about it, I know why this might not still work for you if you have multiple versions of R installed...even if you use /specific/path/to/Rscript, this command will use whichever R is found by PATH:

https://github.com/microsoft/LightGBM/blob/61b3c30842810205bc95736e6e66baa790b098cd/build_r.R#L128,L142

Sorry, I didn't think of that before! You could try editing build_r.R and replacing those "R" with /opt/R/4.0.1/bin/ as a short-term fix. For example:

.run_shell_command("/opt/R/4.0.01/bin/R", c("CMD", "build", TEMP_R_DIR, "--keep-empty-dirs"))

Or (as a better fix) if you're allowed to edit environment variables, you could do

export PATH="/opt/R/4.0.1:${PATH}"
Rscript build_r.R

jameslamb on 13 Jul 2020

👍1

Hi James, thanks for all your help. It finally worked, you can close the issue. Basically, you are right, It was the conflict in R packages that caused the issue. So editing the environment variable really helped.

export PATH="/opt/R/4.0.1/bin:${PATH}"

sriharitn on 17 Jul 2020

great! Thanks for using LightGBM, come back and open a new issue if you run into other problems with the R package.

jameslamb on 17 Jul 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings