Xgboost: R package compiled from source fails to install

[adatum]

Compiled from source for GPU support with steps from https://xgboost.readthedocs.io/en/latest/build.html#installing-r-package-with-gpu-support :

git clone --recursive https://github.com/dmlc/xgboost -b v1.2.1
cd xgboost; mkdir build; cd build
cmake .. -DUSE_CUDA=ON -DR_LIB=ON -DCMAKE_C_COMPILER=cuda-gcc -DCMAKE_CXX_COMPILER=cuda-g++
make install -j

Compiling succeeds but the package installation fails:

[...]
[100%] Built target xgboost
Install the project...
-- Install configuration: "Release"
-- Installing: /home/[USER]/src/xgboost/build/R-package-install/R-package/src/xgboost.so
-- Set runtime path of "/home/[USER]/src/xgboost/build/R-package-install/R-package/src/xgboost.so" to ""
Command: /usr/bin/R -q -e deps = setdiff(c('data.table', 'magrittr', 'stringi'), rownames(installed.packages()))\  if(length(deps)>0) install.packages(deps, repo = 'https://cloud.r-project.org/')
Command: /usr/bin/R CMD INSTALL --no-multiarch --build /home/[USER]/src/xgboost/build/R-package-install/R-package
CMake Error at RPackageInstall.cmake:16 (message):
  out: , err: Warning: invalid package
  ‘/home/[USER]/src/xgboost/build/R-package-install/R-package’

  Error: ERROR: no packages specified

  , res: 1
Call Stack (most recent call first):
  RPackageInstall.cmake:34 (check_call)
  cmake_install.cmake:89 (include)


make: *** [Makefile:105: install] Error 1

Contents of directory /home/[USER]/src/xgboost/build/R-package-install/:

$ ls -R
.:
R-package

./R-package:
src

./R-package/src:
Makevars  Makevars.win  xgboost.so

My workaround is to install xgboost from CRAN, then copy the GPU-enabled compiled xgboost.so from the build directory to the R package directory (replacing the xgboost.so from the CRAN installation). i.e. copy /home/[USER]/src/xgboost/build/R-package-install/R-package/src/xgboost.so to /home/[USER]/R/x86_64-redhat-linux-gnu-library/4.0/xgboost/libs/

This works, but how can I fix the installation failure?

Running Fedora 33 x86_64.

[hcho3]

I've never seen the error before, and the same installation steps work on my machine. It might be better to document the workaround in the install doc.

[adatum]

@hcho3 The installation part has never worked for me. I had a similar error since a couple of years ago while figuring out how to compile from source with GPU support: https://github.com/dmlc/xgboost/issues/3594#issuecomment-414028276

[hcho3]

GPU installation has been a pain point for the R package, despite the efforts in the recent years. What makes it tricky is that the issue would often come up in certain machines but not on my machine, so it is hard to reproduce. Without a reliable way to reproduce the bug on our end, it is hard for us to resolve it.

Let me file a pull request to document the suggested workaround (copy .so file).

[hcho3]

If you can build a Docker image where this issue occurs, it will help us fix the issue.

[adatum]

Is the workaround actually ok in that it won't cause conflicts? Also notice the latest CRAN version of xgboost shows v1.2.0.1 but the .so file is compiled from the v1.2.1 tag.

I'm not familiar with building Docker images, but if you can provide instructions on what to do in this case, I can give it a try. Alternatively if you can create a Docker image, I could try it on my machine to see if it fails.

[hcho3]

@adatum The v1.2.0.1 version is equivalent to the v1.2.1 tag.

Can you provide more details about your computing environment? For example, which R version are you using? Also the version of CMake and CUDA?

[hcho3]

@adatum Alternatively, can you set up an SSH access to your machine? If you are interested, send me an e-mail at [email protected].

[adatum]

The v1.2.0.1 version is equivalent to the v1.2.1 tag.

Good to know. And there aren't other potential issues by dropping in the GPU-enabled compiled .so into the CPU-only package from CRAN?

R version 4.0.3 (2020-10-10)
cmake version 3.18.4
CUDA Version: 11.1

I'll think about SSH access, but in the meantime let me know if you need any other system details.

[hcho3]

And there aren't other potential issues by dropping in the GPU-enabled compiled .so into the CPU-only package from CRAN?

There should not be an issue.

[adatum]

There should not be an issue.

Good to know.

I have sent you an email.

[hcho3]

6423 will fix the issue.

[sherlockyGH]

hey @hcho3 I have a very similar issue with installing the latest xgboost on my RStudio server on AWS EC2.
The compilation stops and is stuck for hours at one point forcing me to Stop my EC2 instance everytime.

I have detailed my issue at https://stackoverflow.com/questions/64946612/xgboost-installation-stuck-on-rstudio-server-on-aws-ec2. However from whatever Googling I did, it seems the issue is about gcc version or something. I am complete newbie when it comes to aws, rstudio server, linux, everything. Please help ..

[hcho3]

@sherlockyGH Your issue is not the same issue as here, since in your case, XGBoost fails to compile. I am moving your comment to #6424. Let's continue discussion there.