Xgboost: ImportError: No module named numpy.distutils.core (Ubuntu xgboost installation)

[sidsvash26]

I am a new user to Ubuntu and recently downloaded Ubuntu 14.04 desktop version alongside Windows 10.

My PC configs are: 4 GB RAM, 64 bit

I installed Anaconda using the following command from the terminal (after downloading from website):
bash Anaconda3-2.5.0-Linux-x86_64.sh

Installation was successful.

Now in my ubuntu terminal, I followed steps as instructed on http://xgboost.readthedocs.org/en/latest/build.html#building-on-ubuntu-debian

1. Installed a recent GNU C++ compiler -->successful
2. git clone --recursive https://github.com/dmlc/xgboost cd xgboost; make -j4
   Building was also successful and I could build 'libxgboost.so'
3. sudo apt-get install python-setuptools (Successful)
4. cd python-package
5. sudo python setup.py install

The 5. command returns an error after a number of lines. The error is as follows:

'ImportError: No module named numpy.distutils.core'

Can anyone suggest how to get rid of this error so that I can install xgboost ?

The full log after step 5. is as follows:

_sidvash@sidvash-HP-Pavilion-g6-Notebook-PC:~/xgboost/python-package$ sudo python setup.py install
Install libxgboost from: ['/home/sidvash/xgboost/python-package/xgboost/../../lib/libxgboost.so']
running install
running bdist_egg
running egg_info
writing requirements to xgboost.egg-info/requires.txt
writing xgboost.egg-info/PKG-INFO
writing top-level names to xgboost.egg-info/top_level.txt
writing dependency_links to xgboost.egg-info/dependency_links.txt
reading manifest file 'xgboost.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '_' under directory 'xgboost/include'
warning: no files found matching '_' under directory 'xgboost/src'
warning: no previously-included files matching 'xgboost/build/_' found anywhere in distribution
warning: no previously-included files matching 'xgboost/_.o' found anywhere in distribution
warning: no previously-included files matching '_.pyo' found anywhere in distribution
warning: no previously-included files matching '_.pyc' found anywhere in distribution
writing manifest file 'xgboost.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/libpath.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/VERSION -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/build-python.sh -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/__init__.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/training.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/compat.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/core.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/sklearn.py -> build/bdist.linux-x86_64/egg/xgboost
copying build/lib.linux-x86_64-2.7/xgboost/plotting.py -> build/bdist.linux-x86_64/egg/xgboost
byte-compiling build/bdist.linux-x86_64/egg/xgboost/libpath.py to libpath.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/init.py to init.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/training.py to training.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/compat.py to compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/core.py to core.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/sklearn.py to sklearn.pyc
byte-compiling build/bdist.linux-x86_64/egg/xgboost/plotting.py to plotting.pyc
installing package data to build/bdist.linux-x86_64/egg
running install_data
copying /home/sidvash/xgboost/python-package/xgboost/../../lib/libxgboost.so -> build/bdist.linux-x86_64/egg/xgboost
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xgboost.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating 'dist/xgboost-0.4-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing xgboost-0.4-py2.7.egg
removing '/usr/local/lib/python2.7/dist-packages/xgboost-0.4-py2.7.egg' (and everything under it)
creating /usr/local/lib/python2.7/dist-packages/xgboost-0.4-py2.7.egg
Extracting xgboost-0.4-py2.7.egg to /usr/local/lib/python2.7/dist-packages
xgboost 0.4 is already the active version in easy-install.pth

Installed /usr/local/lib/python2.7/dist-packages/xgboost-0.4-py2.7.egg
Processing dependencies for xgboost==0.4
Searching for scipy
Reading https://pypi.python.org/simple/scipy/
Best match: scipy 0.17.0
Downloading https://pypi.python.org/packages/source/s/scipy/scipy-0.17.0.zip#md5=28a4fe29e980804db162524f10873211
Processing scipy-0.17.0.zip
Writing /tmp/easy_install-2YPTnH/scipy-0.17.0/setup.cfg
Running scipy-0.17.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-2YPTnH/scipy-0.17.0/egg-dist-tmp-z4NE5c
Traceback (most recent call last):
File "setup.py", line 39, in
url='https://github.com/dmlc/xgboost')
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 73, in run
self.do_egg_install()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 96, in do_egg_install
cmd.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 381, in run
self.easy_install(spec, not self.no_deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 597, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 648, in install_item
self.process_distribution(spec, dist, deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 694, in process_distribution
[requirement], self.local_index, self.easy_install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain
return installer(requirement)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 646, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 834, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1040, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1025, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50, in run_setup
lambda: execfile(
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 100, in run
return func()
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52, in
{'file':setup_script, 'name':'main'}
File "setup.py", line 265, in

File "setup.py", line 253, in setup_package

ImportError: No module named numpy.distutils`_

[brickerino]

Are you sure you are using right python?

Try

which python

or

readlink -f `which python`

to determine whether you are using Anaconda or not.

[sidsvash26]

Hi,

Thanks for commenting.

which python returns the following:
/home/sidvash/anaconda3/bin/python

[sidsvash26]

@brickerino : I have just updated the full log after step 5 in the description. Please see if that helps.

[phunterlau]

The message shows you are not having numpy, please check if your PYTHON_PATH is pointing to the anaconda package path. Also, be sure conda install numpy if not yet

[sidsvash26]

@phunterlau :
I checked the python_path using the follwing command:
python -c "import sys; print(sys.path)"

It returns:
['', '/home/sidvash/anaconda3/lib/python35.zip', '/home/sidvash/anaconda3/lib/python3.5', '/home/sidvash/anaconda3/lib/python3.5/plat-linux', '/home/sidvash/anaconda3/lib/python3.5/lib-dynload', '/home/sidvash/anaconda3/lib/python3.5/site-packages', '/home/sidvash/anaconda3/lib/python3.5/site-packages/Sphinx-1.3.5-py3.5.egg', '/home/sidvash/anaconda3/lib/python3.5/site-packages/cryptography-1.0.2-py3.5-linux-x86_64.egg', '/home/sidvash/anaconda3/lib/python3.5/site-packages/setuptools-19.6.2-py3.5.egg']

This looks fine to me. However, I see in my first post that the path "/usr/lib/python2.7/dist-packages" appears in a number of lines in the error and I am not sure if this should also be there in the python path or not. Could you suggest anything?

I also ran "conda install numpy" which returned the following:

Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: ......................
Package plan for installation in environment /home/sidvash/anaconda3:

The following packages will be downloaded:

| package | build |
| --- | --- |
| wheel-0.29.0 | py35_0 82 KB |

The following packages will be UPDATED:

wheel: 0.26.0-py35_1 --> 0.29.0-py35_0

Proceed ([y]/n)? y

Fetching packages ...
wheel-0.29.0-p 100% |################################| Time: 0:00:01 50.38 kB/s
Extracting packages ...
[ COMPLETE ]|###################################################| 100%
Unlinking packages ...
[ COMPLETE ]|###################################################| 100%
Linking packages ...
[ COMPLETE ]|###################################################| 100

Tried sudo python setup.py install after installing this but no luck, still the same error

[phunterlau]

well, you have python from the system and anaconda, and python 2 3 both, and you have system's setuptool, so some env has messed up. i can suggest creating a virtualenv and install numpy scikit-learn and xgboost

[sidsvash26]

@phunterlau : Thanks for your comment. I really appreciate your help.

I am a novice on Ubuntu and don't want to mess up things further.

Could you please tell me a step by step process to do this?

I know that I can create a virtualenv using the following: (installing virtualenv is not recommended with conda)
conda create -n <env_name> python=3.5 anaconda

And then how should I install numpy scikit-learn and xgboost on this env ?

I can activate the env by:
source activate <env_name>

Should I just then proceed with further installations as follows:
pip install numpy
pip install scikit-learn

And for xgboost I should repeat all these steps?
git clone --recursive https://github.com/dmlc/xgboost cd xgboost; make -j4
cd python-package; sudo python setup.py install

[sidsvash26]

Okay, so I did the following but still getting the exact same error:
conda create -n snakes python=3.5 numpy pandas scikit-learn
sidvash:~$ source activate snakes
discarding /home/sidvash/anaconda3/bin from PATH
prepending /home/sidvash/anaconda3/envs/snakes/bin to PATH

cd xgboost/python-package
sudo python setup.py install

For some reason, the installation still keeps referring to "/usr/lib/python2.7/dist-packages"

[sidsvash26]

I got it sorted out by following the below steps:

1. Remove current dir named "xgboost" which I earlier downloaded from github.
2. pip install xgboost

This worked like a charm. I didn't try this earlier as it was mentioned on the github site that you should install from github for latest version.

I posted this question on stack overflow and someone suggested this. Here's the link:
http://stackoverflow.com/questions/35332300/importerror-no-module-named-numpy-distutils-core-ubuntu-xgboost-installation/35493714#35493714

Anyway, thanks a lot for your help guys: @brickerino @phunterlau

[phunterlau]

Glad that pip install xgboost helped :-)

[knguyen0105]

If 'pip install xgboost' doesn't work for you, this issue can also be fixed by doing

sudo -s
python setup.py install 

The problem is 'sudo python' will use system's Python instead of anacoda's python.

More detail: http://stackoverflow.com/questions/30626160/anaconda-python-not-available-from-sudo

[ashish1302]

@knguyen0105 Good worked for me....thanks man....:)

[ashish1302]

@knguyen0105 giving error
"/home/ashish/anaconda2/bin/../lib/libgomp.so.1: version `GOMP_4.0' not found (required by /home/ashish/anaconda2/lib/python2.7/site-packages/xgboost/./lib/libxgboost.so)"

can anybody help?

[brando90]

what is PYTHON_PATH suppose to return?

[TiagoMHCSantana]

@knguyen0105 It worked just fine for me! Thanks!

[zebrabug]

Thanks knguyen0105, Summary for Ubuntu, anaconda 2:

1. http://xgboost.readthedocs.io/en/latest/build.html
2. sudo -s
   python setup.py install
   instead of
   cd python-package; sudo python setup.py install

[RSjakson]

＠knguyen0105， really thanks a lot