Xgboost: OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.

[symPhysics]

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.

Environment info

Operating System: Mac OSX Sierra 10.12.1

Compiler:

Package used (python):

xgboost version used: xgboost 0.6a2

If you are using python package, please provide

1. The python version and distribution:
   Pythong 2.7.12
2. The command to install xgboost if you are not installing from source
   pip install xgboost

Steps to reproduce

1. from xgboost import XGBClassifier
   import numpy as np
   import matplotlib.pyplot as plt
   x = np.array([[1,2],[3,4]])
   y = np.array([0,1])
   clf = XGBClassifier(base_score = 0.005)
   clf.fit(x,y)
   plt.hist(clf.feature_importances_)

   What have you tried?

   See the error message:
   "OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.
   OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/."

I tried:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

It can do the job for me. But it is kind of ugly.

---

I know it might be not the problem of xgboost, but I'm pretty sure this problem happened after I upgrade xgboost using 'pip install xgboost'. I post the issue here to see if someone had the same problem as me. I have very little knowledge about OpenMP. Please help!
Thanks in advance!

[staubda]

I'm having the same issue when I try to use both xgboost and matplotlib.

OS X 10.10.5
Python 2.7.12
xgboost 0.6
matplotlib 1.5.1

[jimthompson5802]

I've encountered the same problem with xgboost and matplotlib. Same symptom...kernel dies with the same error message as reported by @symPhysics

Mac OS X 10.12.3

Python 3.5.2 |Anaconda custom (x86_64)| (default, Jul 2 2016, 17:52:12)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin

xgboost 0.6a2

matplotlib 2.0.0

[jimthompson5802]

FWIW...this seems to be a MacOS specific problem. I was not able to recreate the problem on a CentOS Linux VM.

[ilovejs]

At least for Mac,

brew uninstall libiomp clang-omp

as long as u got gcc v5 from brew it come with openmp

follow steps in:
https://github.com/dmlc/xgboost/tree/master/python-package

We can close this issue now.

[deltaoui]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

[JamieGainer]

For me the issue (on a Mac) was fixed with
pip uninstall xgboost
followed by
pip install xgboost

[Toooodd]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

thanks, it works!!!

[gpetty]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

[shengwei-xu]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

thanks a lot, it works.

[csuagui]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

Thanks too, it works for me too.

[M-Alami]

@gpetty You sir are a true hero. Thank you!

[ghost]

@gpetty It works! THANKSSS :))

[christabella]

Strangely conda install nomkl didn't work for me (don't have libiomp clang-omp installed on Brew either), only
import os os.environ['KMP_DUPLICATE_LIB_OK']='True'
could work

[SumNeuron]

brew uninstall libiomp clang-omp

Error: No such keg: /usr/local/Cellar/libiomp

[kangkang59812]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

thanks a lot, even though i dont know what is nomkl. It confused me a few days when i run a CNN-RNN model using tensorflow.When i used os.environ['KMP_DUPLICATE_LIB_OK']='True' , it worked, but the program run so slow.You got the right ans.

[Bjvanminnen]

My issues were completely unrelated to xgboost, but I got here via google so that I'd share for the sake of others.

I am using keras and matplotlib installed via conda. Setting KMP_DUPLICATE_LIB_OK just changed my experience from a warning to an exception/crash. Install nomkl didn't change anything for me. Eventually I downgraded my version of matplotlib and that fixed things for me
conda install matplotlib=2.2.3

[esvhd]

For people googling this error, the best solution that I found is listed here.

The site lists two method. I prefer method 1, which is to go into your anaconda3/lib directory and remove the old libiomp5.dylib file. Nice and simple.

[leong1016]

For people googling this error, the best solution that I found is listed here.

The site lists two method. I prefer method 1, which is to go into your anaconda3/lib directory and remove the old libiomp5.dylib file. Nice and simple.

Please be really careful if you want to try method 1. Everything crashed after I deleted the file and I have to reinstall openmp. Same problem reported in the original csdn link (in Chinese).

[Cuica20]

Thanks!!
For me on MAC:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

[sam1902]

Had the same problem after switching from homebrew/virtualenv installed numpy/keras/matplotlib/tensorflow to miniconda installed. Solved by running conda install nomkl though I reckon keras (and numpy) wont run as fast as they could if they don't make use of Intel's Math Kernel Lib but at least my code still run so ¯\_(ツ)_/¯

[jms-dipadua]

@sam1902 that worked for me too -- updating the os.environ variable did not. thanks!

[TVI-BIZ]

have same issue on my Mac
Just run - conda update -n base conda
it install and update needed dependences

[wangcj05]

Got the same issue on my mac, tried suggested method and it works

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

[bwilfley]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

I used anaconda to do the install of nomkl. That fixed the problem.

Also, the attempt to do "brew uninstall..." gave the "No such keg" error.

[mfansler]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

It should be noted that this is strongly discouraged by the OpenMP devs. From their error message:

The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results.

[georgeu2000]

conda install nomkl worked for me.

Can anyone explain what this does and why it fixes the problem?

[jscriptcoder]

@georgeu2000, I'm wondering about the same. It did also fix the problem for me. I'm using Numpy, Matplotlib, Keras and Tensorflow as backend. I don't seem to understand what that package does. Anyone could explain?

Thanks a lot.

[trivialfis]

I saw some benchmarks between openblas and mkl before, mkl is not superior. Is there any reason for using mkl over openblas? Explaining something without the source code at hand is not very motivating.

[lvwarren]

On MacOS Mojave:

conda install nomkl

works but it has to downgrade:

  mkl_fft                             1.0.10-py36h5e564d8_0 --> 1.0.1-py36h917ab60_0
  mkl_random                           1.0.2-py36h27c97d8_0 --> 1.0.1-py36h78cc56f_0
  numexpr                              2.6.9-py36h7413580_0 --> 2.6.9-py36hafae301_0
  numpy                               1.15.4-py36hacdab7b_0 --> 1.15.4-py36h926163e_0
  numpy-base                          1.15.4-py36h6575580_0 --> 1.15.4-py36ha711998_0
  scikit-learn                        0.20.2-py36h27c97d8_0 --> 0.20.2-py36hebd9d1a_0
  scipy                                1.2.1-py36h1410ff5_0 --> 1.2.1-py36h1a1e112_0

[ZhuoerLyu]

My issues were completely unrelated to xgboost, but I got here via google so that I'd share for the sake of others.

I am using keras and matplotlib installed via conda. Setting KMP_DUPLICATE_LIB_OK just changed my experience from a warning to an exception/crash. Install nomkl didn't change anything for me. Eventually I downgraded my version of matplotlib and that fixed things for me
conda install matplotlib=2.2.3

Hey buddy, you saved my ass

[gitjonez]

Had the same problem after switching from homebrew/virtualenv installed numpy/keras/matplotlib/tensorflow to miniconda installed. Solved by running conda install nomkl though I reckon keras (and numpy) wont run as fast as they could if they don't make use of Intel's Math Kernel Lib but at least my code still run so ¯_(ツ)_/¯

^^^^ This is what did the trick for me. Thanks!

[Isaac009]

Thanks!!
For me on MAC:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

WORKS FOR ME THANKS

[jolespin]

In my conda lib, I have multiple libgomp:

I got the expected error:

OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

This definitely helps:

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Here are all of my versions:

(µ_env) jespinozlt-osx:lib jespinoz$ pwd
/Users/jespinoz/anaconda/envs/µ_env/lib
(µ_env) jespinozlt-osx:lib jespinoz$ ls -lhtr | grep "omp"
-rw-rw-r--    4 jespinoz  tigr   165B Oct 28  2016 libgomp.spec
-rw-rw-r--    4 jespinoz  tigr   108K Oct 28  2016 libgomp.a
-rwxrwxr-x   11 jespinoz  tigr    62K Oct 28  2016 libgomp.1.dylib
-rwxrwxr-x    6 jespinoz  tigr   489K Oct 27 17:30 libomp.dylib
-rwxrwxr-x    5 jespinoz  tigr   176K Mar 14 13:10 libiompstubs5.dylib
-rwxrwxr-x    5 jespinoz  tigr   296K Mar 14 13:10 libiomp5_db.dylib
-rwxrwxr-x    5 jespinoz  tigr   2.7M Mar 14 13:10 libiomp5.dylib
lrwxr-xr-x    1 jespinoz  tigr    15B Apr 10 14:51 libgomp.dylib -> libgomp.1.dylib

Should I just remove one of them? If so, should I keep libiomp5.dylib instead of libgomp.1.dylib and just symlink?

[yangzhou95]

OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.

OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

[hzitoun]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

Does fix the problem. Thanks

[binkod2000]

I ran into this problem on a fresh install of anaconda on an a brand new Mac book pro in Anaconda trying to use tensor-flow and keras. I used. 'conda update --all' and it fixed the problem.

[preet1881]

brew uninstall libiomp clang-omp

Error: No such keg: /usr/local/Cellar/libiomp

i have the same error how did you solve it , and conda insatll nomkl is not working either
its giving the error
UnavailableInvalidChannel: The channel is not accessible or is invalid.
channel name: torch
channel url: https://conda.anaconda.org/torch
error code: 404

You will need to adjust your conda configuration to proceed.
Use conda config --show channels to view your configuration's current state,
and use conda config --show-sources to view config file locations.

[hxuaj]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

I ran into the same problem and I use keras, numpy and tensorflow 1.14.
conda install nomkl does solve my problem! Thank you @gpetty

[sameershmedianet]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with
'conda install nomkl'.

This worked for me as well. I was getting error while running LightGBM package, it was erroring out after a certain number of rows or certain number of features.

I'm guessing it has something to do with memory usage or parallelising process. It would be great if someone can explain what exactly 'libomp' does and why 'nomkl' solves it. Thank you @gpetty.

[ppmdatix]

Strangely conda install nomkl didn't work for me (don't have libiomp clang-omp installed on Brew either), only

os.environ['KMP_DUPLICATE_LIB_OK']='True'

could work

Hello
Same for me but
conda install -c anaconda nomkl
worked for me
Hope it helps

[hoveidar]

Hey all,

although using os.environ['KMP_DUPLICATE_LIB_OK']='True' is absolutely frowned upon, but eventually, I had to use it cause conda install -c anaconda nomkl didn't work for me. Any other ideas?

[mayhuifu]

use the os.environ['KMP_DUPLICATE_LIB_OK']='True' works but was not recommended. use Conda install nomkl looks running endless, has to kill it. use conda update -n base Conda works and it updated the lib

[xhRhapsody]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

Thank you very much! It works for me!

[JackGuoFranklin]

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

it works for me,thank you .

[MakeWater]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

Thanks Bro！ It worked for me!

[MakeWater]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

Thanks Bro！ It worked for me!

I just test tensorflow object_detection API and run the object_detection_tutorial.ipynb file

[trypythy]

i'm on a 2010 MBP, running the python(3.7) file on anaconda; so I guess it was something to do with my 'old' system - i have no clue on what territories i'm treading here - tried uninstalling xgboost, realised it's not installed, tried installing it, it just got stuck; tried conda install nomkl, and then my python script itself wouldn't run (illegal instruction: 4); uninstalled nomkl; finally worked when i used

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

i still don't know how and why it worked, but hey, it works now

[sharonwoo]

Thanks for this; conda update --all was what worked for me. Running Mac OS Mojave with Python 3.7 Anaconda (not the new install... dates from last year).

[trivialfis]

Reopening, if someone has any insight to the issue, please share.

[CarcajadaArtificial]

We can close this issue now.

LMAO

[Bue-von-hon]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

it works for me thx

[1ps0]

For me, this combination worked:

brew uninstall libiomp clamp-omp
conda uninstall intel-openmp -n base
conda install -c intel openmp -n myenv
conda install nomkl -n myenv

The primary conflict came from the brew libiomp but then later conflicts with the conda base env and my python 3.7 env over openmp sent me in circles until i sorted out the dependency chain/fallback. The channel source distinction made a huge difference.

[j4m3sb0nd007]

I tried this and the error stopped !

import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

It works for me too

[cheetah90]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

Thanks! It works for me as of today! Can anyone explain why this works?

[hcho3]

We now provide a binary wheel for Mac OSX, which links to /usr/local/lib/libomp.dylib. Install XGBoost by running

python -m pip install xgboost

[apneaHomosapiens]

Thank you this solved my case. My Mac did not have libiomp and clamp-omp, and I did not install nomkl.
So I only do these two steps.

conda uninstall intel-openmp
conda install -c intel openmp

For me, this combination worked:

brew uninstall libiomp clamp-omp
conda uninstall intel-openmp -n base
conda install -c intel openmp -n myenv
conda install nomkl -n myenv

The primary conflict came from the brew libiomp but then later conflicts with the conda base env and my python 3.7 env over openmp sent me in circles until i sorted out the dependency chain/fallback. The channel source distinction made a huge difference.

[xbiggyl]

(macOS Catalina 10.15.5)

Going through the installed package in my env, I noticed that there was an update for mkl.

So I was able to solve my case by updating mkl.

conda install -c intel mkl

[chegini]

you can run this comment conda install nomkl

[caffett]

I had the same error on my Mac with a python program using numpy, keras, and matplotlib. I solved it with 'conda install nomkl'.

What is the magic behind this?

[CatNofishing]

I had the same error on my window10.

[vickykhan89]

os.environ['KMP_DUPLICATE_LIB_OK']='True', I tried this command and the problem solve thanks to @Toooodd

[sjoerdvansteenkiste]

I found an alternative solution to this problem here, which is to preload the OpenMP runtime using the LD_PRELOAD variable:

LD_PRELOAD=<path_to_libiomp5.so> <path_to your_executable>

This eliminates multiple loadings of libiomp, and makes all the components use this specific version of OpenMP.

[powarvinayak]

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Followed by

conda install nomkl

Worked for me. Thanks!