Xgboost: Fail when building xgboost using 64-bit mingw g++

[Willamette-OR]

I would like to compile xgboost for Anaconda Python 3.x in 64-bit Windows 10, so I followed the steps here - https://xgboost.readthedocs.org/en/latest/build.html#building-on-windows and here - Install xgboost under python with 64-bit msys failing. Using the 64-bit mingw g++, I got the following error message when "make -j4" using mingw64.mk:

g++: error: dmlc-core/libdmlc.a: No such file or directory
g++: error: rabit/lib/librabit_empty.a: No such file or directory
Makefile:120: recipe for target 'lib/libxgboost.dll' failed
mingw32-make: *** [lib/libxgboost.dll] Error 1

More details of the log messages can be found here:
log.txt

Can you please let me know what's going on and how I can resolve this?

Thanks a lot.

[baibaiw5]

You can copy mingw32-make.exe to git directory(eg. C:\Program Files\Git\usr\bin),and rename it to make.exe.Then rerun make -j4 in git shell.

[jfpuget]

You need to build dmlc-core and rabit directly

cd dmlc-core
make -j4

cd ../rabit
make -j4

but this one fails in my installation, trying to understand why.

[jfpuget]

OK, I found it.

Using mingw64, you need to have mingw32-make in your path. Alias it to make as explained in the installation guide.
You then go to xgboost directory and do the following
cd dmlc-core
make -j4
cd ../rabit
make lib/librabit_empty -j4
cd ..
cp make/mingw64.mk config.mk
make -j4

[jfpuget]

typo:
in the previous comment, one line should read
make lib/librabit_empty.a -j4
instead of
make lib/librabit_empty -j4

[Willamette-OR]

Thanks a lot jfpuget!! It finally works with your steps above!

[jfpuget]

I wrote complete instructions here: https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_For_Anaconda_on_Windows?lang=en

[thirdwing]

@jfpuget Maybe you can help to improve our instructions. Any PR will be truly welcome.

[tqchen]

@jfpuget This looks great, can you make a PR to improve the old instruction. Thanks!

[jfpuget]

Sure, will do ASAP (couple of days as I am just back from vacation and people have thought of me at work ;) )

[Zhongjiong]

Hi all. When 'make -j4' in the root directory, I have this problem. Do you have some advice? Thx

A subdirectory or file build already exists.
Error occurred while processing: build.
Makefile:97: recipe for target 'build/logging.o' failed
make: ** [build/logging.o] Error 1
make: ** Waiting for unfinished jobs....
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/common/common.o' failed
make: ** [build/common/common.o] Error 1
The syntax of the command is incorrect.
Makefile:97: recipe for target 'build/c_api/c_api_error.o' failed
make: ** [build/c_api/c_api_error.o] Error 1

[Zhongjiong]

I got it...I was using git cmd.. not git bash... That's why I encountered the problem.

[khotilov]

I can also share my notes on how I build xgboost R and python packages under windows.

Some of my workarounds were similar to Jean-François', but there are some subtle differences because I'm using the mingw build toolchain bundled with Rtools. Essentially, windows is a mess with a huge variety of mingw-based bundles out there nowadays of varying degree of completeness, stability and up-to-dateness. Even in my situation that I describe below there are three mingw-based toolsets that have to work together:

• a minimal no-compiler set of command line gnu tools with windows git;
• a gcc-4.7 compiler but no-make set with Anaconda;
• and a fairly usable set which has gcc-4.6.3 and make that comes with Rtools.

The latter one is sufficient for the C++11 needs of xgboost, and I don't see any reason so far to involve yet another mingw distribution into this mess. In such situations it's critical to be very careful about what goes into PATH and in what order.

A few notes on Rtools:

I use the latest Rtools bundle version 3.3. However (IMPORTANT), during Rtools installation, I do not install the 3.3 toolchain with gcc 4.9 - it still has some issues. The 3.2 64-bit toolchain is sufficient and works fine. Since I use R more, I have Rtools' paths added to my system PATH, but not Miniconda's. Whenever I need python while not using Anaconda's shortcuts, I just set the needed paths manually.

R package build is fairly trivial:

• git the source
• launch R

setwd('wherever xgboost R-package directory is')
install.packages('.', repos = NULL, type='source')
setwd('tests')
source('testthat.R')

Building xgboost exe & libraries needs some workarounds:

• assuming git-bash, within xgboost root directory
• make sure that which gcc points to /c/Rtools/gcc-4.6.3/bin/gcc (depending on where Rtools were installed into)
• edit rabit/Makefile and add the -m64 flag to the CFLAGS definition line (shouldn't it be there in the first place?): export CFLAGS = -m64 -O3 -msse2 $(WARNFLAGS)
• the following should work now (need to run make in multiple stages here due to some mingw bundle interactions/quirks/bugs):

cp make/mingw64.mk config.mk
cp make/mingw64.mk dmlc-core/config.mk
cd rabit
make lib/librabit_empty.a -j4
cd ../dmlc-core
make -j4
cd ..
make -j4

Python package:

export PATH=/c/Miniconda/Scripts:/c/Miniconda:/C/Miniconda/MinGW/bin:$PATH
cd python-package/
python setup.py install
python tests/python/test_basic_models.py

[jfpuget]

You are right: if you already have a working g++ compiler, then you don't need to install yet another one.

[tqchen]

with https://github.com/dmlc/xgboost/pull/1071, things should be easier

[khotilov]

BTW, I've submitted #1071 which would allow to simply run mingw32-make.exe -j from the project root to compile everything at once, when using that recent Mingw distribution. The make in Rtools still needs workarounds...

@jfpuget I'm curious: did you have any specific reason to choose the POSIX threads and SEH exception handling option? E.g., I've just tried installing Win32 and SJLJ (just because that was what gcc in Rtools used) and that seems to work.

An interesting observation on gcc 5.3: it seems to generate the came 64 bit code no matter whether there was the -m64 flag set for the rabit library or not. That's why you had no error linking it. Did gcc switch to -m64 as a default at some version?

On modification of the PATH within python: if you have just modified the system PATH, Python from anaconda in windows sometimes (depending on how you start it?) would only pick it up after a reboot/re-login. I didn't check it right now, but I had some similar issues some time ago.

[jfpuget]

@khotilov I just left the default settings in the mingw64-gw installer.

Modifying the system path wasn't good enough, even if I rebooted python (actually, I restarted the kernel from within a notebook). I'll check again why I needed to modify the os path variable inside Python.

My goal was to have xgboost up and running, hence I stopped investigating when I was done.

[thirdwing]

@khotilov I am not sure which MinGW-w64 with GCC 5.3 you are using.

But in some distribution (maybe most), there is no multiarch support.

[yunzhou]

In xgboost\dmlc-core\include\dmlc\base.h
line 113

if (defined MINGW32) || (defined MINGW64)

//#define fopen64 std::fopen

endif

if you are using MINGW64, you should comment this fopen64 define since
mingw64 has defined its own fopen64 in the stdio.h

[qingdatascience]

@yunzhou Thank you. After messed around for a whole night, your trick really did the job.

[qingdatascience]

Actually my errors did say something wrong with the 'base.h'.

[harishneela]

@jfpuget : I tried installing using your link but i am getting following error
Makefile:136: recipe for target 'xgboost' failed

I am sure i did everything as per your article. Can you please help me? Thanks

[jfpuget]

Hi,
 
there is no way you can get help unless you provide more information.  What is the error during the build?  What you provide is not the error message, it is the summary.
 
Please send the log of the build without any editing.
 
Best regards/Cordialement,JFJean-Francois Puget, PhDDistinguished Engineer, Machine Learning and Optimization, IBM AnalyticsMobile: [phone] Email: j-f.[email protected]: http://bit.ly/jfpuget Twitter: @JFPuget
 
 
----- Original message -----From: harishneela [email protected]: dmlc/xgboost [email protected]: Jean Francois Puget/France/IBM@IBMFR, Mention [email protected]: Re: [dmlc/xgboost] Fail when building xgboost using 64-bit mingw g++ (#1049)Date: Wed, Nov 2, 2016 7:51 PM 
@jfpuget : I tried installing using your link but i am getting following errorMakefile:136: recipe for target 'xgboost' failed
I am sure i did everything as per your article. Can you please help me? Thanks
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 Sauf indication contraire ci-dessus:/ Unless stated otherwise above:
Compagnie IBM France
Siège Social : 17 avenue de l'Europe, 92275 Bois-Colombes Cedex
RCS Nanterre 552 118 465
Forme Sociale : S.A.S.
Capital Social : 657.364.587 €
SIREN/SIRET : 552 118 465 03644 - Code NAF 6202A

[df865017]

Mac 安装 xgboost

1、升级brew

由于把Mac升级到10.12后，brew没有更新，我在更新brew的时候发现几个坑，现分析一下。不要直接使用 brew update 更新brew，会报错。使用 brew upgrade 进行软件更新，在更新其他软件的同时会主动更新brew。

2、安装最新版本的gcc (gcc-6)

brew install gcc --without-multilib

注意：如果部升级 brew 就不能安装 gcc。

3、从Git上下载源码

Git clone --recursive https://github.com/dmlc/xgboost

下载后 cd xgboost; cp make/minimum.mk ./config.mk; make -j4 编译xgboost。到目前为止，只是编译了xgboost而已。(多尝试几次，可能下次就成功了哈)

4、安装python版xgboost

因为xgboost分Python版和R语言版。由于R的性能瓶颈，我使用的python，因此安装python版xgboost。

进入xgboost目录下，然后再进入xgboost的 python-package 目录，使用命令：cd python-package/

在 python-package 目录下执行：sudo python setup.py install，如果不加sudo则会报错：

5.完成，我自己安装成功了，希望对其他人有用哈

[abhijithch]

@jfpuget just curious to know if you are aware that these instructions are valid on a POWER8 machine. I am trying to build XGBoost on POWER8 and centos 7. I am getting the same error as the OP.

[jfpuget]

I don't know.

[elessar001]

Hello @jfpuget

Thank you for your blog. However I need some guidance to complete the installation. I get an error once I type "python setup.py install"

It says "cannot find xgboost library in the candidate path". Does this mean I have an error in installation prior to this point. I did not get any errors when I followed the steps.

Kind Regards

[zhengwang522]

I followed jfpuget's instructions to make for submodules, then here is an error after the last make,

dmlc-core/libdmlc.a: error adding symbols: Archive has no index; run ranlib to add one
collect2.exe: error: ld returned 1 exit status
Makefile:174: recipe for target 'xgboost' failed
make: * [xgboost] Error 1

Anyone has ideas?

Thanks!

[sajitroshan]

@jfpuget I tried following your instruction on my windows OS several times, but I get this error every time:
$ cd dmlc-core
$ mingw32-make -j4
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o line_split.o src/io/line_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio_split.o src/io/recordio_split.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o input_split_base.o src/io/input_split_base.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o io.o src/io.cc
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o local_filesys.o src/io/local_filesys.cc
In file included from include/dmlc/./././parameter.h:25:0,
from include/dmlc/././registry.h:14,
from include/dmlc/./data.h:16,
from include/dmlc/threadediter.h:20,
from src/io/cached_input_split.h:16,
from src/io.cc:13:
include/dmlc/././././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/./././parameter.h:513:7: required from here
include/dmlc/././././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast(&val)->~T();
^~~~
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/./././parameter.h:872:9: required from here
include/dmlc/././././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast(&val);
^~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o data.o src/data.cc
include/dmlc/././././optional.h: In instantiation of 'const T& dmlc::optional::operator() const [with T = int]':
include/dmlc/././././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/./././parameter.h:877:13: required from here
include/dmlc/././././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return reinterpret_cast(&val);
^~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o recordio.o src/recordio.cc
In file included from src/data/././text_parser.h:11:0,
from src/data/./libsvm_parser.h:13,
from src/data/disk_row_iter.h:19,
from src/data.cc:12:
include/dmlc/omp.h:15:81: note: #pragma message: Warning: OpenMP is not available, project will be compiled into single-thread code. Use OpenMP-enabled compiler to get benefit of multi-threading.
"Use OpenMP-enabled compiler to get benefit of multi-threading.")
^
In file included from include/dmlc/././parameter.h:25:0,
from include/dmlc/./registry.h:14,
from include/dmlc/data.h:16,
from src/data.cc:5:
include/dmlc/./././optional.h: In instantiation of 'dmlc::optional::~optional() [with T = int]':
include/dmlc/././parameter.h:513:7: required from here
include/dmlc/./././optional.h:60:7: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
reinterpret_cast(&val)->~T();
^~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::value() const [with T = int]':
include/dmlc/././parameter.h:872:9: required from here
include/dmlc/./././optional.h:106:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return *reinterpret_cast(&val);
^~~~~~~~~
include/dmlc/./././optional.h: In instantiation of 'const T& dmlc::optional::operator() const [with T = int]':
include/dmlc/./././optional.h:134:11: required from 'std::ostream& dmlc::operator<<(std::ostream&, const dmlc::optional&) [with T = int; std::ostream = std::basic_ostream]'
include/dmlc/././parameter.h:877:13: required from here
include/dmlc/./././optional.h:97:12: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
return *reinterpret_cast(&val);
^~~~~~~~~
g++ -c -O3 -Wall -Wno-unknown-pragmas -Iinclude -std=c++0x -fPIC -DDMLC_USE_HDFS=0 -DDMLC_USE_S3=0 -DDMLC_USE_AZURE=0 -msse2 -o config.o src/config.cc
ar cr libdmlc.a line_split.o recordio_split.o input_split_base.o io.o local_filesys.o data.o recordio.o config.o

[jfpuget]

You are compiling with g++, not mingw.

[sajitroshan]

@jfpuget
Based on your prescription available at https://www.ibm.com/developerworks/community/blogs/jfp/entry/Installing_XGBoost_For_Anaconda_on_Windows?lang=en, I carried out the following steps:

1) I installed Git for windows.
2) I launched Git Bash from start menu
3) I changed directory to /c/Users/Roshan/code
4) I ran the commands:
$ git clone --recursive https://github.com/dmlc/xgboost
$ cd xgboost
$ git submodule init
$ git submodule update

5) I downloaded MinGW-W64 from the url mentioned in your post
6) I checked and found mingw32-make in "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin"
7) I added "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev0\mingw64\bin" to my path system variable
8) I ran the command "which mingw32-make" which returned "/c/Program Files/mingw-w64/x86_64-7.1.0-posix-seh-rt_v5-rev0/mingw64/bin/mingw32-make"
9) I ran the command: alias make='mingw32-make'
10) I changed the directory to "/c/Users/Roshan/code/xgboost/dmlc-core"
11) Then, I ran the command: make -j4

How did I end up compiling with g++ instead of mingw. I followed each one of your instructions, almighty. Why am I still burning in hell?

[sajitroshan]

@jfpuget I cleaned up everything and tried all of these steps again. This time, I installed mingw in C drive (as one of the websites recommended avoiding Program Files, as this folder has a space in its name). While installing, I also selected the version as 5.3.0 (instead of the default 7 something). But that did not help either.

[sajitroshan]

@qingdatascience : What specifically did you do to make it work?

[datascienceqing]

I did not remember. It looks like just editing the bash script file like @yunzhou suggested. If it did not work for you, I would suggest you to try LightGBM, which is from MS and faster.

[sajitroshan]

Many thanks to @datascienceqing as well as @jfpuget

Since I could not install xgboost, I installed LightGBM. I followed the instructions provided at the following URL. The entire installation (start to finish) took less than 30 minutes:
https://github.com/Microsoft/LightGBM/tree/master/python-package

I have windows OS. I do not have Visual Studio. I could install LightGBM in 5 steps:
1) Download and install cmake from https://cmake.org/download/
2) Download and install MS Build from https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
3) Download and install Git from https://git-for-windows.github.io/?cm_mc_uid=41805252005114998995219&cm_mc_sid_50200000=1500774547
4) Start Git Bash
5) Run these commands:
git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM/python-package
python setup.py install

[sajitroshan]

I could finally install xgboost, after several rounds of trial and error. The instructions at the following location was very helpful: http://www.picnet.com.au/blogs/guido/post/2016/09/22/xgboost-windows-x64-binaries-for-download/

Thanks to @gatapia @jfpuget and @qingdatascience