Kratos: Clang and MKL

[philbucher]

following the discussion in #7362

As of now we do explicitly not support MKL in the LinearSolversApp when compiling with Clang. Reason is OpenMP related, but I am not 100% sure about the details / what we can do about it.

I tried to run it when compiling with Clang but it gives the following runtime error:

OMP: Error #15: Initializing libiomp5.so, but found libomp.so.5 already initialized. OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. Aborted (core dumped)

also it gives the following compiler warning which already indicates that sth is off:
~
clang: warning: -Wl,--no-as-needed: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -lpthread: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -lm: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: -ldl: 'linker' input unused [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-L/home/philippb/intel/compilers_and_libraries_2018.3.222/linux/mkl/lib/intel64' [-Wunused-command-line-argument]
~

I am not sure how to proceed, anyone has an idea/suggestion?

@shahrokh-shayegan discussed this with @oberbichler some time ago but I don't remember the details, maybe they can comment

Here are some references:

• https://software.intel.com/en-us/mkl-windows-developer-guide-calling-intel-mkl-functions-from-multi-threaded-applications
• https://software.intel.com/en-us/articles/using-threaded-intel-mkl-in-multi-thread-application
• https://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-using-intel-mkl-with-threaded-applications
• https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications
• https://software.intel.com/en-us/mkl-macos-developer-guide-intel-mkl-specific-environment-variables-for-openmp-threading-control
• https://software.intel.com/en-us/mkl-macos-developer-guide-mkl-domain-num-threads

@KratosMultiphysics/technical-committee this will also be interesting how to handle once we move to C++11 based parallelism. => not only for MKL, same will also apply for other external libs that use OpenMP => like AMGCL!

FYI @vgudzulic

[vgudzulic]

@philbucher Thanks for the update. Just to note: I tried compiling LinearSolversApplication & mkl with llvm's clang (clang 10) on mac (ofc I disabled the error check in CMakeLists.txt that prevents that), and I got same warnings as you, but no openmp error. And I can use pardiso solver.

[roigcarlo]

Just for the reccord I also been able to compile clang + mkl and running the code with openmp (through KMP_DUPLICATE_LIB_OK). What I cannot assure is that results are 100% accurate.

[philbucher]

Just for the reccord I also been able to compile clang + mkl and running the code with openmp (through KMP_DUPLICATE_LIB_OK). What I cannot assure is that results are 100% accurate.

same here, I can also run it with this.

Googling it seems like there is no better solution and many other users of MKL have the same problem

Should we then just remove the error message?

7477 corrects the compiler flags (then the warnings disappear)

[roigcarlo]

I would remove the error and leave it as a warning. Probably if someone is compiling specifically MKL they know more or less what they are doing

[pooyan-dadvand]

Is it possible to link against the mkl openmp lib when we are enabling the mkl?

[roigcarlo]

It is possible, but its complicated. You can remove the -fopenmp from Kratos linking stage (but not the building) and then forcefully add -liomp5.

What this does is use to force gcc to use the compatibility layer of intel's implementation, which may or may not work and I don't know which version of OpenMP supports.

Explained here: https://stackoverflow.com/questions/25986091/telling-gcc-to-not-link-libgomp-so-it-links-libiomp5-instead
It's quite old but situation remains the same (or worst in any case so...)

[pooyan-dadvand]

So what is saying it to use the MKL for gcc instead of the one for icc. Is it also the source of our problem?

[roigcarlo]

So what is saying it to use the MKL for gcc instead of the one for icc. Is it also the source of our problem?

I don't think I am following you here. The root of the problem is that you cannot use MKL + GOMP. There is no "MKL for gcc", I would expect that the LinearSolvers app is compatible with any linear algebra lib like BLAS, so in that sense if you don't want to use MKL you have alternatives, but if you chose to use it you have to stick to intel's compiler.

If I recall correctly there is a amd alternative (ACML something like that) which is built againts GOMP with the same extended interface as MKL because the performance of intel mkl in amd processors is.. subpair. Maybe we can make use of that.

[pooyan-dadvand]

I was referring to this in the link:

That's the reason why MKL's multithreaded driver comes in two flavours - an Intel one and a GNU one. That the latter is missing on some machines is probably a defect of the installation.

@roigcarlo as you said, I also think we should try this to use the same library rather than suppressing the error and preparing ourself for a mysterious failure afterward