Kratos: Issue with compiling Kratos on Mac

[vgudzulic]

Description
Hello everyone,
I am trying to compile Kratos on a Mac using different compilers. Right now, I can compile Kratos only using LLVM's clang compiler. It does not compile with g++, intel, and apple clang.

The issue with g++ and intel seems to lie in linker flag "-Wl,--no-undefined" defined in global CMakeLists.txt, and the flag "--no-as-needed" defined in CMakeLists.txt of LinearSolversApplication.

Unfortunately, even though both intel and g++ compile and link, only intel works - namely there is a segfault when I attempt to run Kratos compiled with g++:

Vladislavs-MBP:scripts vladislav$ python3 
Python 3.8.5 (default, Jul 21 2020, 18:31:18) 
Type "help", "copyright", "credits" or "license" for more information.
>>> import KratosMultiphysics
 |  /           |
 ' /   __| _` | __|  _ \   __|
 . \  |   (   | |   (   |\__ \
_|\_\_|  \__,_|\__|\___/ ____/
           Multi-Physics 9.0."Dev"-3d080b5f68-Release
Compiled with OpenMP support.
Maximum OpenMP threads: 8.
Segmentation fault: 11

When I try to compile with apple clang, I get an error:

Undefined symbols for architecture x86_64:
  "_omp_get_max_threads", referenced from:
      Kratos::ComputeNodalGradientProcess<true>::ClearGradient() in compute_nodal_gradient_process.cpp.o
      Kratos::ComputeNodalGradientProcess<true>::PonderateGradient() in compute_nodal_gradient_process.cpp.o
      Kratos::ComputeNodalGradientProcess<false>::ClearGradient() in compute_nodal_gradient_process.cpp.o
      Kratos::ComputeNodalGradientProcess<false>::PonderateGradient() in compute_nodal_gradient_process.cpp.o
      Kratos::ComputeNodalNormalDivergenceProcess<true>::ClearDivergence() in compute_nodal_normal_divergence_process.cpp.o
      Kratos::ComputeNodalNormalDivergenceProcess<true>::PonderateDivergence() in compute_nodal_normal_divergence_process.cpp.o
      Kratos::ComputeNodalNormalDivergenceProcess<false>::ClearDivergence() in compute_nodal_normal_divergence_process.cpp.o
      ...
ld: symbol(s) not found for architecture x86_64

because the CMake can't find corresponding OpenMP libraries.

I am not sure which compilers are supported on Mac, I could find only a script with LLVM's clang. However, I would like to use LinearSolversApplication with intel MKL and for that, I need to compile with either intel or gnu compilers. I can of course manually change the flag in my version, but maybe someone else also has the same problem so I wanted to share this.

Scope

• Compilation
• KratosCore
• LinearSolversApplication

To Reproduce
Set the corresponding compiler.
When "-Wl,--no-undefined" is replaced by "-Wl,-undefined,error" in global CMakeLists.txt, and "--no-as-needed" is removed from CMakeLists.txt of LinearSolversApplication it should compile with both g++ and intel.

Expected behavior
Should compile and link.

Environment

• OS: macOS Catalina 10.15.6
• g++: g++ 10.2.0
• icpc: icpc (ICC) 19.1.2.258 20200623
• clang: Apple clang version 11.0.3 (clang-1103.0.32.62)
• llvm clang: clang version 10.0.1

Additional info
If you need more details, please let me know.

Thanks in advance!

[roigcarlo]

Hi @vgudzulic

We have only been able to successfully compile Kratos in Mac with LLVM's Clang. Ping @ipouplana as he has more experience with this topic.

I might be wrong on that as I am not an expert in with Mac's but Apple and LLVM clang's should be essentially the same, with the only difference being LLVM's one comes with more build-in libs (libomp among them) so you should also be able to compile with apple-clang if you install OpenMP for it (it does not come by default with Xcode). I recommend you to use brew:

brew install libomp

As for intel's MKL, you should in principle be able to link it regardless the compiler you use. What problem is giving you with specifically clang?

[ipouplana]

It is just as @roigcarlo said. The only difference between Apple and LLVM clangs should be that LLVM comes with omp library already installed...
Regarding the rest of compilers, I haven't tried them recently, but I think I once tried to compile g++ but I also had issues...

[vgudzulic]

Hi @roigcarlo and @ipouplana,

Thanks for your answers!
The problem with clang is when I set a CMake flag: -DUSE_EIGEN_MKL=ON,
I get the following error:

**** configuring KratosLinearSolversApplication ****
-- MKLROOT = /opt/intel/compilers_and_libraries_2020.2.258/mac/mkl
CMake Error at applications/LinearSolversApplication/CMakeLists.txt:102 (message):
  Clang does not yet support MKL

[roigcarlo]

I've just tried in my build env (clang 10.0.0) and seems that works just fine bypassing that check.

The only problem I find is that sees that Intel MKL has its own version of OpenMP 5 built in, and it can conflict with the version that the compiler is using ( which is not a problem for example with Intel compiler). Basically this means that the code can compile but is gonna produce a nasty warning at runtime complaining about libomp5 trying to initialize while libomp is already initialized.

@vgudzulic If this is the problem, I believe that replacing the -fopenmp flag in clang with -fopenmp=/opt/intel/mkl/lib/intel64/libomp5.so (you should use the one inside the MKL you are using) should solve any potential incompatibility and should let you ignore that error safely.

I will not have access to a Mac machine until Monday so I am afraid I cannot test if this solution works until then.

Also would be good to have a confirmation from @philbucher and @oberbichler. I am correct on this or there is another reason to disable MKL with clang? If this work we should probably modify the CMakeList.txt on our side, as this is not a Mac exclusive issue.

[vgudzulic]

@roigcarlo Unfortunately, I was not able to do what you have suggested. First, I am a bit confused about where the -fopenmp flag is defined (obviously in '${CMAKE_CXX_FLAGS}' but I could not find it in Kratos, only in some external libraries)
I added the flag to the configure script, and it now reads:
' -DCMAKE_CXX_FLAGS="${CMAKE_CXX_FLAGS} -msse3 -std=c++11 -march=native -Wno-unknown-pragmas -fopenmp=/opt/intel/lib/libomp5.dylib" \ ' but this does not work.

There is however a way to make intel compiler work:
In global CMakeLists.txt change the problematic line for the case of OS=Darwin from -Wl,--no-undefined to -Wl,-undefined,error:

if(${CMAKE_C_COMPILER} MATCHES "icc.*$")
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC  -funroll-loops  ") #-lpthread -wd654 -wd10010  ")
    set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS} -fPIC -funroll-loops  ") #-lpthread -wd654 -wd10010  ")
    #set(CMAKE_SHARED_LINKER_FLAGS "-Wl,--no-undefined")
    set(CMAKE_SHARED_LINKER_FLAGS "-Wl,-undefined,error")
    message("additional default options were set for intel compiler")
    message("CMAKE_CXX_FLAGS = ${CMAKE_CXX_FLAGS}")
    message("CMAKE_C_FLAGS = ${CMAKE_C_FLAGS}")
  endif(${CMAKE_C_COMPILER} MATCHES "icc.*$")

and add a condition in CMakeLists.txt of LinearSolversApplication for the case of OS=Darwin not to include --no-as-needed flag:

if( NOT MSVC )
            if(NOT ${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
                set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m64 -L$ENV{MKLROOT}/lib/intel64 -Wl,--no-as-needed -lpthread -lm -ldl" )
            else(NOT ${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
                set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m64 -L$ENV{MKLROOT}/lib/intel64 -lpthread -lm -ldl" )
            endif(NOT ${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
        endif()

This works for me with intel compiler. It would be really great if you could check it when you have access to Mac and maybe add to the repository if it works well.

Regards

[roigcarlo]

-fopenmp=/opt/intel/lib/libomp5.dylib" \ ' but this does not work.

Yep sorry, seems that the option is gone since clang 3.8.

If you managed to compile it with intel with those changes it should be safe to use. I will check it on my machine on Monday as well to conform that test, etc... are working. If that's the case we will update the CMake with those changes.

Thx!

[philbucher]

(brief answer I am on vacation)

Intel provides a tool for selecting the linking options for OS/compiler
Unfortunately it doesn't provide this for Clang

@oberbichler and I tried it briefly but at the time we didn't get it to work. (there are lots of options)

we can take a look again

[philbucher]

I made a separate issue for the MKL stuff, see #7458

[roigcarlo]

Closing this, we can follow up in the new issue.