Lightgbm: Rstudio on Windows throws Fatal error, on Mac works fine.

Created on 16 Sep 2019  路  8Comments  路  Source: microsoft/LightGBM 
R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> lgbdl::lgb.dl(compiler = "vs")

C:\Users\turgut.abdullayev\Downloads>C:

C:\Users\turgut.abdullayev\Downloads>cd C:\Users\TURGUT~1.ABD\AppData\Local\Temp\RtmpOuxWf7 

C:\Users\TURGUT~1.ABD\AppData\Local\Temp\RtmpOuxWf7>git clone --recursive https://github.com/Microsoft/LightGBM 
Cloning into 'LightGBM'...
Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'compute'
Cloning into 'C:/Users/turgut.abdullayev/AppData/Local/Temp/RtmpOuxWf7/LightGBM/compute'...
Submodule path 'compute': checked out '36c89134d4013b2e5e45bc55656a18bd6141995a'

C:\Users\TURGUT~1.ABD\AppData\Local\Temp\RtmpOuxWf7>cd LightGBM 

C:\Users\TURGUT~1.ABD\AppData\Local\Temp\RtmpOuxWf7\LightGBM>git checkout master 
Your branch is up to date with 'origin/master'.
Already on 'master'
* installing *source* package 'lightgbm' ...
** using staged installation
** libs
installing via 'install.libs.R' to C:/Users/turgut.abdullayev/Documents/R/R-3.6.1/library/00LOCK-lightgbm_r/00new/lightgbm
CMake Error at CMakeLists.txt:7 (PROJECT):
  Generator

    Visual Studio 16 2019

  could not find any instance of Visual Studio.



-- Configuring incomplete, errors occurred!
See also "C:/Users/turgut.abdullayev/AppData/Local/Temp/RtmpOuxWf7/LightGBM/lightgbm_r/src/build/CMakeFiles/CMakeOutput.log".
CMake Error at CMakeLists.txt:7 (PROJECT):
  Generator

    Visual Studio 15 2017

  could not find any instance of Visual Studio.



-- Configuring incomplete, errors occurred!
See also "C:/Users/turgut.abdullayev/AppData/Local/Temp/RtmpOuxWf7/LightGBM/lightgbm_r/src/build/CMakeFiles/CMakeOutput.log".
-- Selecting Windows SDK version  to target Windows 10.0.17763.
CMake Error at CMakeLists.txt:7 (PROJECT):
  Failed to run MSBuild command:

    C:/Program Files (x86)/MSBuild/14.0/bin/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 14.0.23107.0
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 9/16/2019 10:37:25 AM.
    Project "C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\build\CMakeFiles\3.15.0-rc3\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\build\CMakeFiles\3.15.0-rc3\VCTargetsPath.vcxproj(14,2): error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
    Done Building Project "C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\build\CMakeFiles\3.15.0-rc3\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

    "C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\build\CMakeFiles\3.15.0-rc3\VCTargetsPath.vcxproj" (default target) (1) ->
      C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\build\CMakeFiles\3.15.0-rc3\VCTargetsPath.vcxproj(14,2): error MSB4019: The imported project "C:\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

        0 Warning(s)
        1 Error(s)

    Time Elapsed 00:00:00.04


  Exit code: 1



-- Configuring incomplete, errors occurred!
See also "C:/Users/turgut.abdullayev/AppData/Local/Temp/RtmpOuxWf7/LightGBM/lightgbm_r/src/build/CMakeFiles/CMakeOutput.log".
CMake Error at C:/Program Files/CMake/share/cmake-3.15/Modules/CMakeMinGWFindMake.cmake:12 (message):
  sh.exe was found in your PATH, here:

  C:/Rtools/bin/sh.exe

  For MinGW make to work correctly sh.exe must NOT be in your path.

  Run cmake from a shell that does not have sh.exe in your PATH.

  If you want to use a UNIX shell, then use MSYS Makefiles.

Call Stack (most recent call first):
  CMakeLists.txt:7 (PROJECT)


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Check for working C compiler: C:/Rtools/mingw_64/bin/gcc.exe
-- Check for working C compiler: C:/Rtools/mingw_64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Rtools/mingw_64/bin/g++.exe
-- Check for working CXX compiler: C:/Rtools/mingw_64/bin/g++.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.0") 
-- Found OpenMP_CXX: -fopenmp (found version "4.0") 
-- Found OpenMP: TRUE (found version "4.0")  
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/turgut.abdullayev/AppData/Local/Temp/RtmpOuxWf7/LightGBM/lightgbm_r/src/build
Scanning dependencies of target _lightgbm
[  3%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\c_api.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\lightgbm_R.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\application\application.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\boosting\boosting.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\boosting\gbdt.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\boosting\gbdt_model_text.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\boosting\gbdt_prediction.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\boosting\prediction_early_stop.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\bin.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\config.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 34%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\config_auto.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 37%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\dataset.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 40%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\dataset_loader.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 43%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\file_io.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\json11.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /* Copyright (c) 2013 Dropbox, Inc.
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\metadata.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\parser.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 56%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\io\tree.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 59%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\metric\dcg_calculator.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 62%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\metric\metric.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\network\linker_topo.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 68%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\network\linkers_mpi.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 71%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\network\linkers_socket.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\network\network.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\objective\objective_function.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\data_parallel_tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 84%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\feature_parallel_tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\gpu_tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\serial_tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.obj
C:\Users\turgut.abdullayev\AppData\Local\Temp\RtmpOuxWf7\LightGBM\lightgbm_r\src\src\treelearner\voting_parallel_tree_learner.cpp:1:0: warning: -fPIC ignored for target (all code is position independent)
 /*!
 ^
cc1plus.exe: warning: unrecognized command line option "-Wno-ignored-attributes"
[100%] Linking CXX shared library ..\lib_lightgbm.dll
[100%] Built target _lightgbm
Found library file: C:/Users/TURGUT~1.ABD/AppData/Local/Temp/RtmpOuxWf7/LightGBM/lightgbm_r/src/lib_lightgbm.dll to move to C:/Users/turgut.abdullayev/Documents/R/R-3.6.1/library/00LOCK-lightgbm_r/00new/lightgbm/libs/x64** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'lightgbm'
    finding HTML links ... done
    agaricus.test                           html  
    agaricus.train                          html  
    bank                                    html  
    dim                                     html  
    dimnames.lgb.Dataset                    html  
    getinfo                                 html  
    lgb.Dataset                             html  
    lgb.Dataset.construct                   html  
    lgb.Dataset.create.valid                html  
    lgb.Dataset.save                        html  
    lgb.Dataset.set.categorical             html  
    lgb.Dataset.set.reference               html  
    lgb.cv                                  html  
    lgb.dump                                html  
    lgb.get.eval.result                     html  
    lgb.importance                          html  
    lgb.interprete                          html  
    lgb.load                                html  
    lgb.model.dt.tree                       html  
    lgb.plot.importance                     html  
    lgb.plot.interpretation                 html  
    lgb.prepare                             html  
    lgb.prepare2                            html  
    lgb.prepare_rules                       html  
    lgb.prepare_rules2                      html  
    lgb.save                                html  
    lgb.train                               html  
    lgb.unloader                            html  
    lgb_shared_params                       html  
    lightgbm                                html  
    predict.lgb.Booster                     html  
    readRDS.lgb.Booster                     html  
    saveRDS.lgb.Booster                     html  
    setinfo                                 html  
    slice                                   html  
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (lightgbm)
[1] TRUE

sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.6.1   tools_3.6.1      lgbdl_0.0.0.9000

Hi, I run my code snippet both on my Mac and Windows. Mac works fine but on Windows, RStudio throws a fatal error... What could be the possible reason of error?
But your examples for LightGBM on github work fine in both systems. So, probably this issue is related to my dataset.
AFAIR @Laurae2 once advised that integer and missing values could be the main reason. But again, Mac does not throw any kind of errors for NA and integers despite that I filled NAs with 0s and encoded integers to numeric on Windows.

bug r-package
Source
picture henry090

All 8 comments

@henry090 Does LightGBM crash with the default example?

library(lightgbm)
data(agaricus.train, package = "lightgbm")
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
data(agaricus.test, package = "lightgbm")
test <- agaricus.test
dtest <- lgb.Dataset.create.valid(dtrain, test$data, label = test$label)
params <- list(objective = "regression", metric = "l2")
valids <- list(test = dtest)
model <- lgb.train(params,
                   dtrain,
                   100,
                   valids,
                   min_data = 1,
                   learning_rate = 1,
                   early_stopping_rounds = 10)
Laurae2 picture Laurae2 on 29 Sep 2019
👀1

@henry090 Does LightGBM crash with the default example?

library(lightgbm)
data(agaricus.train, package = "lightgbm")
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
data(agaricus.test, package = "lightgbm")
test <- agaricus.test
dtest <- lgb.Dataset.create.valid(dtrain, test$data, label = test$label)
params <- list(objective = "regression", metric = "l2")
valids <- list(test = dtest)
model <- lgb.train(params,
                   dtrain,
                   100,
                   valids,
                   min_data = 1,
                   learning_rate = 1,
                   early_stopping_rounds = 10)

Yes, as I mentioned, it works fine. This dataset causes fatal error on windows. https://www.kaggle.com/c/ieee-fraud-detection

I figured out that when I read 10k rows, it does not throw fatal error. When load 100k rows it thrwos an error.
Here is the code:

library(tidyquant)
library(data.table)
library(lightgbm)
setwd("C:/Users/turgut.abdullayev/Downloads/ieee-fraud-detection")

train_trs <- fread('train_transaction.csv',na.strings = '',nrows = 100000) 
train_iden <- fread('train_identity.csv',na.strings = '',nrows = 100000) 

test_trs <- fread('test_transaction.csv',na.strings = '',nrows = 100000) 
test_iden <- fread('test_identity.csv',na.strings = '',nrows = 100000) 
print('reading is done')

train <- left_join(train_trs,train_iden) %>% select(-c(TransactionID))
test <- left_join(test_trs,test_iden) %>% select(-c(TransactionID))

print('join is done')
total  = bind_rows(train,test)

total = total %>% select(c(isFraud,TransactionDT,TransactionAmt,ProductCD,card1,card2,card3,card4,card5,card6,addr1,addr2,dist1,dist2,P_emaildomain,R_emaildomain,C3,C5,C7,D1,D3,D8,D9,D10,D11,D12,D13,D14,D15,M1,M2,M3,M4,M5,M6,M7,M8,M9,V1,V2,V3,V4,V6,V7,V8,V11,V12,V20,V24,V25,V27,V35,V37,V44,V46,V47,V53,V55,V56,V62,V65,V66,V75,V77,V81,V83,V86,V98,V107,V109,V110,V112,V115,V116,V118,V119,V120,V121,V122,V123,V124,V131,V135,V138,V140,V142,V147,V149,V159,V161,V162,V166,V169,V170,V171,V173,V174,V175,V184,V187,V188,V189,V190,V194,V195,V196,V200,V205,V206,V207,V208,V209,V210,V216,V220,V222,V223,V224,V227,V231,V234,V235,V238,V240,V241,V250,V252,V253,V260,V262,V270,V271,V273,V281,V282,V283,V284,V285,V286,V289,V290,V291,V294,V296,V297,V300,V305,V311,V313,V319,V321,id_01,id_02,id_03,id_04,id_05,id_06,id_07,id_08,id_09,id_10,id_11,id_12,id_13,id_14,id_15,id_16,id_17,id_18,id_19,id_20,id_21,id_22,id_23,id_24,id_25,id_26,id_27,id_28,id_29,id_30,id_31,id_32,id_33,id_34,id_35,id_36,id_37,id_38,DeviceType,DeviceInfo))

rm(train_trs,train_iden,test_trs,test_iden,train,test)

total$id_30 %>% gsub(.,replacement = '',pattern = '[[:punct:]]|[[:digit:]]') %>% 
  trimws() ->total$id_30

total %>% tidyr::separate(P_emaildomain,c('pdomain','ptype_org','pcountry_domain'),sep='.')->total

total %>% tidyr::separate(R_emaildomain,c('rdomain','rtype_org','rcountry_domain'),sep='.')->total

total$id_31 %>% as.character() %>%  gsub(.,replacement = '',pattern = '[[:digit:]]|\\.') %>% 
  trimws()  %>% stringr::str_to_lower() ->total$id_31

total %>% tidyr::separate(id_31,c('id_31_1st','id_31_2nd'),sep="/| ") ->total

total$id_31_2nd = total$id_31_2nd %>% gsub(.,replacement = '',pattern = '[[:punct:]]') 

total %>% tidyr::separate(id_33,c('length','width'),'x') ->total

total%>% mutate(length = as.numeric(length),width=as.numeric(width))->total

total %>% tidyr::separate(DeviceInfo,c('device1','device2'),sep=" |/|:|-") ->total

total[total=='']=NA

total = total %>% mutate_if(is.character,as.factor)

total <- total %>% arrange(TransactionDT) %>% 
  mutate(hr = floor( (TransactionDT / 3600) %% 24 ),
         weekday = floor( (TransactionDT / 3600 / 24) %% 7)
  ) %>% select(-TransactionDT) 

train = total %>% filter(!is.na(isFraud)) 
test= total %>% filter(!is.na(isFraud)) 

rules <- lgb.prepare_rules(data = train)

train <- rules$data

test <- lgb.prepare_rules(data = test, rules = rules$rules)$data

set.seed(500)
pct = floor(nrow(train)*0.8)
tr_0 = train[1:pct,]
tr1 = train[(1+pct):nrow(train),]

#tr_0[is.na(tr_0)]=-999
#tr1[is.na(tr1)]=-999

dtrain <- lgb.Dataset(data.matrix(tr_0  %>% select(-isFraud)), 
                  label = tr_0  %>% pull(isFraud))

dtest <-lgb.Dataset(data.matrix(tr1  %>% select(-isFraud)), 
                    label = tr1  %>% pull(isFraud))

p <- list(boosting_type = 'gbdt',
          objective = "binary" ,
          metric = "AUC",
          boost_from_average = "false",
          learning_rate = 0.008,
          num_leaves = 197,
          min_gain_to_split = 0,
          feature_fraction = 0.3,
          # feature_fraction_seed = 666666,
          bagging_freq = 1,
          bagging_fraction = 0.7,
          # min_sum_hessian_in_leaf = 0,
          min_data_in_leaf = 100,
          lambda_l1 = 0,
          lambda_l2 = 0
)

model<- lgb.train(data = dtrain,
                  params= p, 
                  nrounds=5000, 
                  valids = list(val1 = dtrain ,val2 = dtest), 
                  metric="auc",
                  obj = "binary",
                  eval_freq = 200, 
                  early_stopping_rounds=150
)
henry090 picture henry090 on 30 Sep 2019

Hi, I noticed that after fresh installation for the 1st time LightGBM works (it runs the training process successfully). 2nd time it throws a fatal error (when you restart RStudio).

henry090 picture henry090 on 2 Oct 2019

@henry090 thanks for the report and apologies for our delay. I will see if I can reproduce this.

jameslamb picture jameslamb on 25 Oct 2019

@henry090 thanks for the report and apologies for our delay. I will see if I can reproduce this.

I tested another dataset from Kaggle competition and it works fine. The dataset which I mentioned earlier has 400 or even more columns. Maybe it is related to the number of columns.

henry090 picture henry090 on 25 Oct 2019

@henry090 I tried to reproduce this today.

Short Answer

I think that if you stop using lgb.dl() and run this code instead, you'll get a successful installation with Visual Studio and you'll be able to train a model on the IEEE dataset using the code you provided.

git clone --recursive https://github.com/jameslamb/LightGBM
cd LightGBM
git fetch origin r/msvc-support
git checkout r/msvc-support
Rscript build_r.R

Long Answer

I put some work in today to try to reproduce the issue you reported. The rest of this issue details that attempt. Sorry it took us so long to get back to you! This issue just slipped through.

my environment

Because you said in https://github.com/microsoft/LightGBM/issues/2412#issuecomment-537349922 that sometimes the code succeeds the first time and fails the second time, for each test I installed the R package, loaded it, ran the code, removed all objects and restarted the session, then tried again.

  • R 3.6.1
  • RStudio 1.2.5019
  • Windows 10
  • Visual Studio 16 2019
  • output of sessionInfo():

sessionInfo (click me)

R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] dplyr_0.8.5                lightgbm_2.3.2             R6_2.4.1                  
 [4] data.table_1.12.8          tidyquant_1.0.0            quantmod_0.4.17           
 [7] TTR_0.23-6                 PerformanceAnalytics_2.0.4 xts_0.12-0                
[10] zoo_1.8-7                  lubridate_1.7.8           

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3       compiler_3.6.1   pillar_1.4.3     tools_3.6.1      uuid_0.1-4       jsonlite_1.6.1  
 [7] tibble_3.0.1     lifecycle_0.2.0  lattice_0.20-38  pkgconfig_2.0.3  rlang_0.4.5      cli_2.0.2       
[13] rstudioapi_0.11  curl_4.3         httr_1.4.1       stringr_1.4.0    generics_0.0.2   vctrs_0.2.4     
[19] tidyselect_1.0.0 grid_3.6.1       glue_1.3.2       fansi_0.4.1      Quandl_2.10.0    purrr_0.3.3     
[25] tidyr_1.0.2      magrittr_1.5     ellipsis_0.3.0   assertthat_0.2.1 quadprog_1.5-8   stringi_1.4.6   
[31] crayon_1.3.4

I used your code from https://github.com/microsoft/LightGBM/issues/2412#issuecomment-536399457 (thanks for providing that!). I had to add a library(dply) that I think you missed (since left_join() is used but not exported from those three libraries library'd in in your script).

R code I ran (click me)

library(tidyquant)
library(data.table)
library(dplyr)
library(lightgbm)

# Get training data by clicking "download all" at
# https://www.kaggle.com/c/ieee-fraud-detection/data
#
# unzip the .zip archive downloaded, so that all the .csv files are in a
# "data/" directory under the working directory you run this R code from
DATA_DIR <- file.path(getwd(), "data")

train_trs <- data.table::fread(
    file.path(DATA_DIR, 'train_transaction.csv')
    , na.strings = ''
    , nrows = 100000
) 
train_iden <- data.table::fread(
    file.path(DATA_DIR, 'train_identity.csv')
    , na.strings = ''
    , nrows = 100000
) 
test_trs <- data.table::fread(
    file.path(DATA_DIR, 'test_transaction.csv')
    , na.strings = ''
    , nrows = 100000
) 
test_iden <- data.table::fread(
    file.path(DATA_DIR, 'test_identity.csv')
    , na.strings = ''
    , nrows = 100000
)
print('reading is done')

train <- left_join(train_trs,train_iden) %>% select(-c(TransactionID))
test <- left_join(test_trs,test_iden) %>% select(-c(TransactionID))

print('join is done')
total  = bind_rows(train,test)

total = total %>% select(c(isFraud,TransactionDT,TransactionAmt,ProductCD,card1,card2,card3,card4,card5,card6,addr1,addr2,dist1,dist2,P_emaildomain,R_emaildomain,C3,C5,C7,D1,D3,D8,D9,D10,D11,D12,D13,D14,D15,M1,M2,M3,M4,M5,M6,M7,M8,M9,V1,V2,V3,V4,V6,V7,V8,V11,V12,V20,V24,V25,V27,V35,V37,V44,V46,V47,V53,V55,V56,V62,V65,V66,V75,V77,V81,V83,V86,V98,V107,V109,V110,V112,V115,V116,V118,V119,V120,V121,V122,V123,V124,V131,V135,V138,V140,V142,V147,V149,V159,V161,V162,V166,V169,V170,V171,V173,V174,V175,V184,V187,V188,V189,V190,V194,V195,V196,V200,V205,V206,V207,V208,V209,V210,V216,V220,V222,V223,V224,V227,V231,V234,V235,V238,V240,V241,V250,V252,V253,V260,V262,V270,V271,V273,V281,V282,V283,V284,V285,V286,V289,V290,V291,V294,V296,V297,V300,V305,V311,V313,V319,V321,id_01,id_02,id_03,id_04,id_05,id_06,id_07,id_08,id_09,id_10,id_11,id_12,id_13,id_14,id_15,id_16,id_17,id_18,id_19,id_20,id_21,id_22,id_23,id_24,id_25,id_26,id_27,id_28,id_29,id_30,id_31,id_32,id_33,id_34,id_35,id_36,id_37,id_38,DeviceType,DeviceInfo))

rm(train_trs,train_iden,test_trs,test_iden,train,test)

total$id_30 %>% gsub(.,replacement = '',pattern = '[[:punct:]]|[[:digit:]]') %>% 
    trimws() ->total$id_30

total %>% tidyr::separate(P_emaildomain,c('pdomain','ptype_org','pcountry_domain'),sep='.')->total

total %>% tidyr::separate(R_emaildomain,c('rdomain','rtype_org','rcountry_domain'),sep='.')->total

total$id_31 %>% as.character() %>%  gsub(.,replacement = '',pattern = '[[:digit:]]|\\.') %>% 
    trimws()  %>% stringr::str_to_lower() ->total$id_31

total %>% tidyr::separate(id_31,c('id_31_1st','id_31_2nd'),sep="/| ") ->total

total$id_31_2nd = total$id_31_2nd %>% gsub(.,replacement = '',pattern = '[[:punct:]]') 

total %>% tidyr::separate(id_33,c('length','width'),'x') ->total

total%>% mutate(length = as.numeric(length),width=as.numeric(width))->total

total %>% tidyr::separate(DeviceInfo,c('device1','device2'),sep=" |/|:|-") ->total

total[total=='']=NA

total = total %>% mutate_if(is.character,as.factor)

total <- total %>% arrange(TransactionDT) %>% 
    mutate(hr = floor( (TransactionDT / 3600) %% 24 ),
           weekday = floor( (TransactionDT / 3600 / 24) %% 7)
    ) %>% select(-TransactionDT) 

train = total %>% filter(!is.na(isFraud)) 
test= total %>% filter(!is.na(isFraud)) 

rules <- lightgbm::lgb.prepare_rules(data = train)

train <- rules$data

test <- lightgbm::lgb.prepare_rules(data = test, rules = rules$rules)$data

set.seed(500)
pct = floor(nrow(train)*0.8)
tr_0 = train[1:pct,]
tr1 = train[(1+pct):nrow(train),]

dtrain <- lightgbm::lgb.Dataset(data.matrix(tr_0  %>% select(-isFraud)), 
                      label = tr_0  %>% pull(isFraud))

dtest <- lightgbm::lgb.Dataset(data.matrix(tr1  %>% select(-isFraud)), 
                    label = tr1  %>% pull(isFraud))

p <- list(
    boosting_type = 'gbdt',
    objective = "binary" ,
    metric = "AUC",
    boost_from_average = "false",
    learning_rate = 0.008,
    num_leaves = 197,
    min_gain_to_split = 0,
    feature_fraction = 0.3,
    bagging_freq = 1,
    bagging_fraction = 0.7,
    min_data_in_leaf = 100,
    lambda_l1 = 0,
    lambda_l2 = 0
)

model <- lgb.train(data = dtrain,
                  params= p, 
                  nrounds=5000, 
                  valids = list(val1 = dtrain ,val2 = dtest), 
                  metric="auc",
                  obj = "binary",
                  eval_freq = 200, 
                  early_stopping_rounds=150
)

Results with installing from GitHub (master branch)

I ran remove.packages("lightgbm") in R to be sure I had no lingering installations left, then installed with the following code:

git clone --recursive --depth=1 [email protected]:microsoft/LightGBM.git
cd LightGBM
Rscript build_r.R

Although I have Visual Studio installed, because of the bug reported in #2995 nd #2997, the installation failed back to using mingw (see below)

image

First attempt: SUCCESS

image

Second attempt: SUCCESS

image

Results with installing from GitHub (r/msvc-support branch)

We have some known issues with Visual Studio builds for the R package right now. We have a PR out to fix it (#2963) but it is stuck waiting for a review. I tested with this alternative branch, since that solved the issues reported in #2995 and #2997 .

I ran remove.packages("lightgbm") in R to be sure I had no lingering installations left, then installed with the following code:

git clone --recursive https://github.com/jameslamb/LightGBM
cd LightGBM
git fetch origin r/msvc-support
git checkout r/msvc-support
Rscript build_r.R

On that branch, Visual Studio is found and used correctly.

image

First attempt: SUCCESS

image

Second attempt: SUCCESS

image

Results with lgb.dl() installation

I ran remove.packages("lightgbm") in R to be sure I had no lingering installations left, then installed with the following R code:

remotes::install_github("Laurae2/lgbdl")
lgbdl::lgb.dl(compiler = "vs")

This attempt failed. There have been a lot of changes to the LightGBM R package since the last update to lgbdl (December 2018), so I think that installation path may be broken.

image

Conclusion

I don't believe we have an underlying problem with the LightGBM R package where, for example, the IEEE dataset you mentioned is too large in some dimension for you to train on. I believe you're facing some other installation issue that has been fixed since you first opened this issue.

Could you please try the fix suggested in #2995 and #2997 ? (Sorry that we have not merged it to master yet!)

git clone --recursive https://github.com/jameslamb/LightGBM
cd LightGBM
git fetch origin r/msvc-support
git checkout r/msvc-support
Rscript build_r.R

I hope that helps! If not, please answer these questions:

  1. What was the specific error that you saw?
  2. In https://github.com/microsoft/LightGBM/issues/2412#issuecomment-537349922 you said "restart RStudio"...did you rm(list = ls()) and restart your session? If you close and re-open RStudio, by default everything you'd defined in your global environment will still be left behind, which can lead to strange behavior.
jameslamb picture jameslamb on 25 Apr 2020

Hi, Currently due to quarantine I cannot access my laptop. Please, close this issue because I will not be able to check. If I can, I will inform you. Thank you so much James.

henry090 picture henry090 on 28 Apr 2020

Thank you @henry090 If I'm right about #2963 possibly fixing this issue, maybe by the time you're able to access your laptop again, this will be fixed 馃榾

Thanks for using LightGBM and taking the time to open this issue. Stay safe and have a good week!

jameslamb picture jameslamb on 29 Apr 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tbenthompson picture tbenthompson  路  3Comments
JoshuaC3 picture JoshuaC3  路  3Comments
jameslamb picture jameslamb  路  3Comments
chivee picture chivee  路  3Comments
BuGTEa picture BuGTEa  路  3Comments