Alpaka: std::decay_t causes internal compiler errors in Intel compiler

[krzikalla]

In TaskKernelCpuOmp2Blocks.hpp the lines 102 and 122 read

[this](typename std::decay<TArgs>::type const & ... args)

The std::decay in these lines produces an internal compiler in all Intel.2018 versions.
In alpaka 0.3.5 these lines (there in ExecCpuOmp2Blocks.hpp) just read

[this](TArgs const & ... args)

Unfortunatly we can't switch to Intel.2019 or a newer version yet (since this causes other internal compiler errors in our code). However, if I just remove the std::decay in alpaka 0.4.1 I get a bunch of compilation errors. What is this std::decay for? If I want to remove it, what other changes do I need to make?

This is somehow related to #843

[psychocoderHPC]

For documentation I like to link @ax3l 's comment that it is now possible to add icc to travis https://github.com/alpaka-group/alpaka/issues/493#issuecomment-622222393

[psychocoderHPC]

What is this std::decay for?

You can best see it in the cppreference documentation. It removes references, universal references and is doing some pointer transformations.

As you wrote I do not think that you can simply remove std::decay.

If I want to remove it, what other changes do I need to make?

Depends on the compiler errors.

[psychocoderHPC]

@krzikalla Is it possible to compile all alpaka tests with icc 2018?

[sbastrakov]

I do not remember all details, but as far as I remember that decay is indeed needed, we discussed with @psychocoderHPC at that time.

[krzikalla]

@psychocoderHPC:
CMake didn't complete ("Could NOT find alpaka (missing: alpaka_INCLUDE_DIR)")
To reformulate my initial question in präziserem Deutsch: Warum steht std::decay in alpaka an diesen Stellen? Wofür wurde es damals dort hingeschrieben?
@sbastrakov :
It was introduced as part of a patch, which can be traced back to #843. But why?

I mean, you should by no means change it - Intel 2018 is outdated. However maybe I find a hacky workaround for our problem.

[psychocoderHPC]

CMake didn't complete ("Could NOT find alpaka (missing: alpaka_INCLUDE_DIR)")

Remove all temporary cmake files. Than set the environment variable alpaka_ROOT. I am not sure this happend.

To reformulate my initial question in präziserem Deutsch: Warum steht std::decay in alpaka an diesen Stellen? Wofür wurde es damals dort hingeschrieben?

Because TArgs can be a reference or an array[]. decay avoids that the arguments you push to the kernel will not match the definitions of the kernel arguments.

It was introduced as part of a patch, which can be traced back to #843. But why?

The changes was made to fix the issue that fixed size array e.g. double[4] was not handled correctly. I refactored all kernel calls in alpaka to use perfectly forwarding and avoid that arguments are copied more often than needed.

I mean, you should by no means change it - Intel 2018 is outdated. However maybe I find a hacky workaround for our problem.

A workaround would be nice. For that we need first reproduce the issue and than understand whats going on. I looked again into the code and can not find anything we are doing wrong.

[krzikalla]

Remove all temporary cmake files. Than set the environment variable alpaka_ROOT. I am not sure this happend.

Doesn't work, same error. A documentation on how to build examples and tests would be good.

[krzikalla]

This issue is getting serious now. There is no recent Intel compiler we can work with.

icpc 2017.8, 2018.1, 2018.3: working with alpaka 0.3.5, failing with alpaka 0.4.1 and alpaka 0.5.0.
icpc 2019.2, 2020.1: working with alpaka 0.3.5 and 0.4.1, failing with alpaka 0.5.0.

We don't know the exact reason, the error message is always the same:

../src/Matrix/LinesHelper.cpp(211): internal error: assertion failed at: "shared/cfe/edgcpfe/lower_name.c", line 9640
compilation aborted for ../src/Matrix/LinesHelper.cpp (code 4)

Sorry, that's the whole message.
But we can't abandon icpc.

[sbastrakov]

Thanks for reporting. We've also just learned about issues with Intel compilers in another project using alpaka. And iirc it's exactly the same message.

I will be looking into that next week. So far am also puzzled.

[sbastrakov]

@krzikalla it seems the internal compiler error of the Intel compiler occurs when there is a kernel with no parameters (except for the standard acc), when one does alpaka::kernel::exec or alpaka::kernel::createTaskKernel for it (both call the same internal alpaka stuff). I did not have time yet to look into what exactly goes wrong here.

[sbastrakov]

With @psychocoderHPC we narrowed it down, and it seems the Intel internal compiler error occurs bacause of std::decay with empty template pack. (So as was complained about originally).
I will try to wrap it up in a special decay that for Icc handles the empty pack specially and explicitly does nothing. But that may be on my next work day, Wednesday. Or if someone wants to try this suggestion, they are welcome. I just used the helloWorld example and Serial accelerator to reproduce the issue.

[ax3l]

In parallel: reported as intel bug 04727055 at https://supporttickets.intel.com
(had moderate success with those recently, but an ICE is a good thing to report.) Always open vendor issues, everyone!

I also added a PR drafting CI, as soon as the bug herein is fixed: #1066

@sbastrakov good luck today with this! :crossed_fingers:

[sbastrakov]

Thanks @ax3l ! Your link somehow does not open to me. Or rather it tries to redirect to a login page i guess and gets stuck in what looks like infinite loop. However, may be rather consistent to some other Intel software :)

[ax3l]

Same here, try to open it in "private mode" of your browser.

[sbastrakov]

That words, thanks.

[sbastrakov]

@krzikalla so now alpaka develop tests and examples compile with the Intel compiler for us. Could you check that it works for you as well ?

[ax3l]

@sbastrakov Intel support needs the following information, please. Can you please use Alpaka 0.5.0 and provide:

Could you please attach the preprocessed LinesHelper.cpp? You can do that by appending -E compiler option to the existing command. Also please share the command used to compile this compilation unit so that I can reproduce this issue on my end.

[ax3l]

So far I drafted:

prerequisites: please have a recent version of CMake (3.15+) and Boost (1.65.1+; only the header-only libs are needed) installed

# on my Ubuntu 18.04 LTS
git clone --branch 0.5.0 https://github.com/alpaka-group/alpaka.git
mkdir alpaka/build
cd alpaka/build

. /opt/intel/inteloneapi/setvars.sh
CXX=$(which icpc) cmake .. -Dalpaka_BUILD_EXAMPLES=OFF -DBUILD_TESTING=ON -DALPAKA_ACC_GPU_CUDA_ENABLE=OFF

The primary problem reported herein is visible via:

make VERBOSE=ON <target>

If you set -Dalpaka_BUILD_EXAMPLES=ON one uncovers another icpc internal compiler error:

  make VERBOSE=ON
  # alpaka/example/helloWorld/src/helloWorld.cpp(203):
  #   internal error: assertion failed at: "lower_name.c", line 9761 in record_substitution_for_type

At linking stage I can find another set of problems for some tests and examples:

  ld: cannot find /opt/intel/inteloneapi/compiler/2021.1-beta07/linux/bin/intel64/../../compiler/lib/intel64_lin/i_ofldbegin_target.o: No such file or directory

@sbastrakov which <target> can I put from our tests/examples that shows the std::decay problem? Do we need to apply #1070 and reset #1074? Should we create a branch for this for intel to test on?
Do you see the same linker errors that I see?

[sbastrakov]

Hello @ax3l .

Sorry, I do not understand which file are they talking about in
Could you please attach the preprocessed LinesHelper.cpp?
I can't find / open your ticket either so get the context. (Earlier I meant that the website opens in private mode, but did not try to find the ticket then).

I am a bit puzzled with your last message, there may be some confusion or idk. Let me reiterate the information I've got.
So far we have learned in this issue discussion that with ICC, applying std::decay_t may lead to ICE in one case. Namely, when using it in template parameter pack expansion expression and that pack is empty. This is exactly what happens with alpaka 0.5.0 release and the helloWorld example, as it has no kernel arguments other than acc and so uses an empty pack internally and causes Intel ICE.

And this is the internal error: assertion failed at: "lower_name.c", you are getting, and I am also getting on Hemera with 0.5.0. So in my understanding this thing is the ICE we were hunting down. And now there is a workaround (hinted privately) implemented in #1074 to not use std::decay_t in this case, but manually do std::decay<...>::type. Note that std::decay_t works fine with Intel for one type, or non-empty pack expansions. PR #1070 is just a utility for it to fix Intel compiler detection.

With the current alpaka dev, I am no longer getting ICEs on Hemera with both examples and unit tests when using the 2019 versions installed there. @krzikalla told me that the dev builds for him for 2019 and 2020 Intel versions, but not earlier ones. I did not check it yet, and so do not know if this is the same issue or different.

I never encountered the linker issue you are describing. So it can be unrelated to the decay_t problems.

[ax3l]

And this is the internal error: assertion failed at: "lower_name.c", you are getting, and I am also getting on Hemera with 0.5.0. So in my understanding this thing is the ICE we were hunting down.

Thanks, that's the info that I needed confirmed as it was not yet posted here.

LinesHelper.cpp

I think that was not alpaka specific but in the OPs code.

[ax3l]

As a follow-up, you might want to add ICX/ICPX support to CI.

ICX/ICPX is the next-gen Intel compiler based on LLVM, it's similar to the host-side of DPC++ and where new developments go. Here is a template to derive from: https://github.com/ECP-WarpX/WarpX/blob/0c8d5f61d92309b9a3df6e985f1a9d3da5edfd96/.github/workflows/ubuntu.yml#L81-L117 (apt lines)
Note this bug shipped in the latest oneAPI and fixed in CMake master.