Incubator-mxnet: Is there a conflict between anaconda and mxnet?

[dreamquster]

I had installed anaconda2(python2.7) in Ubuntu16.04. Then, I compile and installe mxnet. But when I use anaconda python interpret execute scripts such as:

import mxnet as mx

I get the following errors

  File "/home/dknight/softwares/pycharm-community-2016.3/helpers/pydev/pydevd.py", line 1596, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/dknight/softwares/pycharm-community-2016.3/helpers/pydev/pydevd.py", line 974, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/dknight/PycharmProjects/mxnet-learn/mxnet-digit-recognize.py", line 40, in <module>
    import mxnet as mx
  File "/home/dknight/OpenSource/Github/mxnet/python/mxnet/__init__.py", line 7, in <module>
    from .base import MXNetError
  File "/home/dknight/OpenSource/Github/mxnet/python/mxnet/base.py", line 43, in <module>
    _LIB = _load_lib()
  File "/home/dknight/OpenSource/Github/mxnet/python/mxnet/base.py", line 35, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/home/dknight/anaconda2/lib/python2.7/ctypes/__init__.py", line 440, in LoadLibrary
    return self._dlltype(name)
  File "/home/dknight/anaconda2/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/dknight/anaconda2/bin/../lib/libgomp.so.1: version `GOMP_4.0' not found (required by /home/dknight/OpenSource/Github/mxnet/python/mxnet/../../lib/libmxnet.so)

With the help of locate libgomp.so, I found multiple the same name library. Is there a conflict between anaconda and mxnet?

python git:(master) ✗ locate libgomp.so
/home/dknight/anaconda2/lib/libgomp.so
/home/dknight/anaconda2/lib/libgomp.so.1
/home/dknight/anaconda2/lib/libgomp.so.1.0.0
/home/dknight/anaconda2/pkgs/libgcc-4.8.5-2/lib/libgomp.so
/home/dknight/anaconda2/pkgs/libgcc-4.8.5-2/lib/libgomp.so.1
/home/dknight/anaconda2/pkgs/libgcc-4.8.5-2/lib/libgomp.so.1.0.0
/usr/lib/gcc/x86_64-linux-gnu/5/libgomp.so
/usr/lib/x86_64-linux-gnu/libgomp.so.1
/usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0

[piiswrong]

anaconda has custom openmp libs. You can probably work around it by tweaking LD_LIBRARY_PATH

otherwise set USE_OPENMP=0 in mxnet/config.mk

[RamonYeung]

I encounted the same error while trying to build MXnet on Anaconda.

1st try

I followed this blog post to solve the problem by using virtual env feature@conda.

# Create a virtual environment for Mxnet
conda create -n mxnet python=2.7
source activate mxnet

sudo apt-get update
sudo apt-get install -y build-essential git libatlas-base-dev libopencv-dev
git clone –recursive https://github.com/dmlc/mxnet
edit make/config.mk
set cuda= 1, set cudnn= 1, add cuda path

cd mxnet
make clean_all
make -j8 # num of CPU cores
conda install -c anaconda numpy=1.11.1
cp -r ../mxnet/python/mxnet .
cp ../mxnet/lib/libmxnet.so mxnet/

# Quick Test
python example/image-classification/train_mnist.py

I was able to run example/image-classification/train_mnist.py

Since virtual env means a seperate environment from the root one, any other pkgs can not be used before installed(linked).

And then wired part. I used conda install -n mxnet anaconda to install all the rest pkgs.
I run the train_mnist.py as soon as it finished, and OSError occurs!

2nd try

Anaconda says they have included the Intel Math Kernel Library (MKL) optimizations (version 11.3.1) for improved performance.

As @piiswrong mentioned, I found a file related both LD_LIBRARY_PATH and MKL.

conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

or set USE_OPENMP=0 in mxnet/config.mk do not help, at least in my case.

Conclusion

1st try can solve the problem, any other pkg you need can be linked from Anaconda root env(if installed).
I believe problem occurs when mxnet is trying to link a system/local lib named libgomp.so which customized by Anaconda.

[dytan]

I encountered this problem too, but after doing ' conda install libgcc' , everything got worked.

[RogerBorras]

I have the same problem and with this solution I can't install the GPU mxnet version on ubuntu 16.04. The solution for conda users is to provide a gpu conda installer like tensorflow....

[Stonesjtu]

@dytan this works for me on ubuntu 16.04, can't figure out the reason.

[wguo68]

I can't run mxnet on ubuntu 14.04 too.

[ysh329]

@dytan 's method is invalid for me. Because when I type "conda install libgcc", screen prints below message: (However I found another solution)

yuens@Spark:~/Software/mxnet/example/ssd$ conda install libgcc
Fetching package metadata ...An unexpected error has occurred.
Please consider posting the following information to the
conda GitHub issue tracker at:

    https://github.com/conda/conda/issues



Current conda install:

               platform : linux-64
          conda version : 4.3.8
       conda is private : False
      conda-env version : 4.3.8
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.12.4
       root environment : /usr/local/lib/anaconda2  (read only)
    default environment : /usr/local/lib/anaconda2
       envs directories : /home/yuens/.conda/envs
                          /usr/local/lib/anaconda2/envs
          package cache : /home/yuens/.conda/envs/.pkgs
                          /usr/local/lib/anaconda2/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False
             user-agent : conda/4.3.8 requests/2.12.4 CPython/2.7.13 Linux/4.4.0-64-generic debian/stretch/sid glibc/2.23
                UID:GID : 1000:1000



`$ /usr/local/lib/anaconda2/bin/conda install libgcc`




    Traceback (most recent call last):
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/exceptions.py", line 617, in conda_exception_handler
        return_value = func(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/cli/main.py", line 137, in _main
        exit_code = args.func(args, p)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/cli/main_install.py", line 80, in execute
        install(args, parser, 'install')
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/cli/install.py", line 210, in install
        unknown=index_args['unknown'], prefix=prefix)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 120, in get_index
        index = fetch_index(channel_priority_map, use_cache=use_cache)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 445, in fetch_index
        repodatas = _collect_repodatas(use_cache, urls)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 433, in _collect_repodatas
        repodatas = _collect_repodatas_serial(use_cache, urls)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 401, in _collect_repodatas_serial
        for url in urls]
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 141, in func
        res = f(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 385, in fetch_repodata
        mod_etag_headers.get('_mod'))
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/conda/core/index.py", line 191, in fetch_repodata_remote_request
        timeout=timeout)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/sessions.py", line 501, in get
        return self.request('GET', url, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
        resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/sessions.py", line 641, in send
        r.content
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/models.py", line 781, in content
        self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/models.py", line 703, in generate
        for chunk in self.raw.stream(chunk_size, decode_content=True):
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 432, in stream
        data = self.read(amt=amt, decode_content=decode_content)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 380, in read
        data = self._fp.read(amt)
      File "/usr/local/lib/anaconda2/lib/python2.7/httplib.py", line 597, in read
        s = self.fp.read(amt)
      File "/usr/local/lib/anaconda2/lib/python2.7/socket.py", line 384, in read
        data = self._sock.recv(left)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 250, in recv
        return self.recv(*args, **kwargs)
      File "/usr/local/lib/anaconda2/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 246, in recv
        [self.socket], [], [], self.socket.gettimeout())
    error: (4, 'Interrupted system call')

[ysh329]

Although @dytan 's method is invalid for me, we can confirm that this problem is about gcc version conflict between system and Anaconda2(or 3). More concretely spearking, gcc version of current system(Ubuntu 16.04, 64bit) is 5.4 which support GOMP protocol 4.0, and MXNet compiling is achieved by gcc5.4. Anaconda2(or 3) uses gcc4.8 compiling which support GOMP protocol 3.0.

After I tried below commands, it's okay to import mxnet in Ubuntu Shell:

yuens@Spark:~/Software/mxnet/example/ssd$ sudo ln -sf /usr/lib/x86_64-linux-gnu/libgomp.so.1 /usr/local/lib/anaconda2/lib/libgomp.so.1.0.0
yuens@Spark:~/Software/mxnet/example/ssd$ sudo ln -sf /usr/lib/x86_64-linux-gnu/libgomp.so.1 /usr/local/lib/anaconda2/pkgs/libgcc-4.8.5-2/lib/libgomp.so.1.0.0
yuens@Spark:~/Software/mxnet/example/ssd$ sudo ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/local/lib/anaconda2/pkgs/libgcc-4.8.5-2/lib/libstdc++.so.6
yuens@Spark:~/Software/mxnet/example/ssd$ sudo ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/local/lib/anaconda2/lib/libstdc++.so.6yuens@Spark:~/Software/mxnet/example/ssd$ python
Python 2.7.13 |Anaconda 4.3.0 (64-bit)| (default, Dec 20 2016, 23:09:15) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import mxnet
>>> exit()

At a word, you should make four soft link:

# libgomp.so
ln -sf /usr/lib/x86_64-linux-gnu/libgomp.so.1 /opt/anaconda3/lib/libgomp.so.1.0.0
ln -sf /usr/lib/x86_64-linux-gnu/libgomp.so.1 /opt/anaconda3/pkgs/libgcc-4.8.5-2/lib/libgomp.so.1.0.0

# libstdc++.so
ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /opt/anaconda3/pkgs/libgcc-4.8.5-2/lib/libstdc++.so.6
ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /opt/anaconda3/lib/libstdc++.so.6

In each ln -sf shell command, front directory is in system, later directory is in Anaconda(You should look up where is your anaconda directory firstly).

Thanks to this link in MXNet compiling part!

[johnsonice]

@dytan 's method works for me as well. i just created a new conda virtual environment, build mxnet, install libgcc. then everything works.

[yajiedesign]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!