Kratos and the Intel Compiler

[philbucher]

This issue is a collection and documentation of other issues, PRs, problems and thoughts that I and others are having since we switched to pybind.
Here is a brief description of the problems and the status:

• Kratos requires the Intel compiler version 2018, otherwise it won't even compile. Theoretically pybind requires only 2017, but then Kratos doesn't compile, see #2000. If an older compiler version is used a proper error message is issued now, rather than just crashing at some point, see #2404
• The Intel Compiler requires GCC in the background, currently the only tested and working version is 5.4

Current problems:

• Currently OMP has to be disabled (by adding KRATOS_EXCLUDE_OPENMP=ON) in the configure, otherwise it crashes in OMP loop, even when running only one thread
• compiling with the C-Flag lifcore (required for the ExternalSolversApp) makes Kratos crash in random places EDIT: I was trying to us static libs (.a), according to @roigcarlo this can be solved using shared libs of blas&lapack (*.so)

Other issues, PRs related to the Intel Compiler: #1971 , #1973 , #1988 , #2108 , #2112 , #2158, #2679

FYI @KratosMultiphysics/technical-committee @msandre

[msandre]

One more point:

• It worked for me with OMP only if there are no empty loops. This was found by observing it crashes as it enters the build loop for conditions. As soon as I add 1 condition to the model part it worked.

[pooyan-dadvand]

Thanks for putting together all these information. Have you contacted to Intel for these bugs. Especially the OpenMP one?

[philbucher]

I was in contact with the computing center abt the crashes, but at that time I didn't know they only happen for empty loops
They told me to make a small example, which is not really possible in Kratos. Therefore they could not really support

@msandre I think we could ask them again regarding the empty loops

[roigcarlo]

@philbucher Can we close this? as far as I know it works again with recent versions of intel. but you are more an active user intel so I leave it you

[philbucher]

@roigcarlo last time I tried it was not working for me, but maybe I am the only one (I used 2018)
if it works for others then please go ahead and close it :+1:

[roigcarlo]

Mm not sure if it works its just someone told me that worked for him, but can't remember who. let's keep it open for a while then

[AndreasWinterstein]

Here an update from my side. I compiled Kratos with the Intel compiler 2018 with gcc 7 in the background and Intel MPI 2018 on two clusters at Leibniz Super Computing Center here in Munich (LRZ) and ran some simulations. I also compared to a compilation with pure gcc 7 and OpenMPI 3.1.2. The performance with the Intel compiler and Intel MPI is better on those kinds clusters, which is reasonable since they are optimized for Intel. I also did some scaling studies.

It works in both cases with OpenMP switched explicitly off. Currently it is not working when OpenMP is switched on. But I am in any case not sure if a hybrid parallelism results in more performance then pure MPI.

I think we can close this issue and open a new one if there are further insights or needs for discussion.

[pooyan-dadvand]

Thanks for the comment!

@KratosMultiphysics/altair we cannot not move to Intel compiler yet without OpenMP. Or we should contact the support.

[philbucher]

in the branch that updates pybind => https://github.com/KratosMultiphysics/Kratos/pull/2877 I was able to get the tests passing in FullDebug, but not in Release (there it still segfaults)
To quote @roigcarlo:

Probably is an optimization flag that is messing up the code from openmp...
Maybe we could try to compile with the custom mode and adding the flags manually, for example everything except the -OX ( I think its 03 in release for intel) to see what happens

I have not yet had the time to go through the compiler flags and test which one is the bad one

But maybe someone else can compile this branch in FullDebug and see if it works

[pooyan-dadvand]

I have got the license but haven't install it yet. I will try to do it these days

[philbucher]

Now that the new version of pybind is merged I will have another look at it in the next days
( since I anyway do the testing for #3895 )

[pooyan-dadvand]

I also should test it....

[philbucher]

in the branch that updates pybind => #2877 I was able to get the tests passing in FullDebug, but not in Release (there it still segfaults)
To quote @roigcarlo:

Probably is an optimization flag that is messing up the code from openmp...
Maybe we could try to compile with the custom mode and adding the flags manually, for example everything except the -OX ( I think its 03 in release for intel) to see what happens

I have not yet had the time to go through the compiler flags and test which one is the bad one

But maybe someone else can compile this branch in FullDebug and see if it works

I have tested again and still arrive to the same conclusion

I am now trying to change the compiler-flags, but I cannot find them...
Can anyone please tell me where they are (e.g. the optimization-flags)

I guess I could also do it through a custom-build-mode but I don't know how ...

@roigcarlo do you thnk these flags have an influence? Edit no they don't :/

[philbucher]

4092 finally solves this issue by disabling ipo (linktime-optimization, called LTO for other compilers)

@roigcarlo suggested this also earlier, but before the pybind-update in #2877 it did not work...

[philbucher]

4092 solved the issue with not being able to compile with OMP enabled

unfortunately it still crashes randomly when running on more than one thread.
For me this happened in Release and FullDebug :/

[uksajja]

@philbucher, May I know if you are compiling with Intel on Windows/Linux?

[philbucher]

Only linux

[uksajja]

Thanks, I will apply your fix and test.
Is there anyone tried to compile with Intel on Windows?

[pooyan-dadvand]

@philbucher do you have any small test which gives the problem? It would be great to have such a test so @uksajja would report it to the Intel support.

[philbucher]

just run the regular tests ...
E.g. in StructuralMechanics or FluidDynamics

[uksajja]

Hi @philbucher @pooyan-dadvand,
cmake files may be missing -std=c++11 flag for intel. I am able to compile and run example with Intel compiler version 2018 update 5, after adding this flag in CmakeLists.txt and pybind11Tools.cmake. Can you please give it a try? This is not a proper fix and probably be applied correctly if it works.

changes.zip

[philbucher]

Thanks @uksajja I will check your changes

How many threads did you use?

[uksajja]

20 threads

[philbucher]

Does it fail without your modifications?

[pooyan-dadvand]

It wasn't compiling for him. Seems that the -std=c++11 flags is only defined for GCC

https://github.com/KratosMultiphysics/Kratos/blob/0b38a20ae8c1168234029b06b64a3b5f3c9b4614/CMakeLists.txt#L89

but no for Intel

[uksajja]

Without changes in cmake files,
Intel-Release - compiled
Intel-Debug - Did not compile - Complained about pybind11Tools.cmake - pybind11 requires C++11 support

[philbucher]

ok thx I will try you proposed changes
Will take me some days though

[philbucher]

I tried again on a newer cluster, same problem, still have to disable OMP to get it running :/

[philbucher]

In the scope of #6496 I did a lot of testing with the Intel compiler and it seems like this problem no longer exists

I tested with & without OMP, Release & FullDebug and different numbers of threads

I had the feeling it might have sth to do with #6379 but it also worked without...
In any case I will close this, in case problems reappear we can reopen

[pooyan-dadvand]

So do we compatible also with Intel MPI?

[philbucher]

So do we compatible also with Intel MPI?

that always worked, @AndreasWinterstein was heavily using it over the last months

[pooyan-dadvand]

Perfect! So would you please tell me which version of IntelMPI you are using?

[philbucher]

So I am using this compiler, but only NON-MPI
~
-- The C compiler identification is Intel 19.0.5.20190815
-- The CXX compiler identification is Intel 19.0.5.20190815
-- Check for working C compiler: /lrz/sys/intel/studio2019_u5/compilers_and_libraries_2019.5.281/linux/bin/intel64/icc
-- Check for working C compiler: /lrz/sys/intel/studio2019_u5/compilers_and_libraries_2019.5.281/linux/bin/intel64/icc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /lrz/sys/intel/studio2019_u5/compilers_and_libraries_2019.5.281/linux/bin/intel64/icpc
-- Check for working CXX compiler: /lrz/sys/intel/studio2019_u5/compilers_and_libraries_2019.5.281/linux/bin/intel64/icpc -- works
~

@AndreasWinterstein I guess you are using the same version but can you please post your setup too?