Alpaka: Replace alpaka metaprogramming parts with boost-1.66 mp11

[BenjaminW3]

The boost-1.66 library mp11 should be able to replace the alpaka metaprogramming utilities completely (because the alpaka ones have been implemented based on the same blog posts leading to mp11).
This would increase the required boost version to at least boost-1.66.

[psychocoderHPC]

Note: We are currently testing mp11 together with nvcc from CUDA 9.1.
As soon as #include <boost/mp11.hpp> is used in a CUDA program it will not compile.

@theZiz

[BenjaminW3]

This is really bad news.
Are you trying to break it down to the header/construct that causes the compiler error? If it is something we do not need, we may be lucky and use only the working sub-headers.

[psychocoderHPC]

It looks like CUDA 8 is working but not CUDA 9. Alex creates an issue for CUDA and for mp11 directly.

[theZiz]

This is what I figured out and wrote to Nvidia:

I am trying to port my application to use the recently added boost::mp11 C++11 metaprogramming library. However just including the template library using nvcc 9.1 without even using it at all fails to compile with the error:

boost/mp11/utility.hpp:141:184: error: expansion pattern ‘boost::mp11::detail::mp_no_type’ contains no argument packs.

Line 141 in boost/mp11/utility.hpp looks like this:

135: // mp_quote
136: template<template<class...> class F> struct mp_quote
137: {
138:     // the indirection through mp_defer works around the language inability
139:     // to expand T... into a fixed parameter list of an alias template
140:
141:     template<class... T> using fn = typename mp_defer<F, T...>::type;
142: };

The test application looks like this:

#include <boost/mp11.hpp>
int main() {}

It is compiled wiht "-std=c++11"; I also tried c++14, but it doesn't make any difference.

"nvcc -v --keep" shows, that the error happens while compiling the host code:

gcc -std=c++11 -D__CUDA_ARCH__=300 -c -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -I"/home/alexander/develop/work/boost_1_66_0/install/include" -m64 -o "test.o" "test.cudafe1.cpp"

When I try to explicitly call this command by hand the same error happens, but "mp_quote" now looks like this in test.cudafe1.cpp:

# 136
template< template< class ...>  class F> struct mp_quote {
# 141
template< class ...T> using fn = typename detail::mp_if_c_impl< static_cast< bool>(detail::mp_valid_impl< F, T...> ::type::value), detail::mp_defer_impl< F, T...> , detail::mp_no_type...> ::type::type;
# 142
};

which explains the error as "detail::mp_no_type..." does not make much sense.

With Cuda 8 the code compiles without failes and "mp_quote" looks like this:

# 136
template< template< class ...>  class F> struct mp_quote {
# 141
template< class ...T> using fn = typename detail::mp_if_c_impl< static_cast< bool>(detail::mp_valid_impl< F, T...> ::type::value), detail::mp_defer_impl< F, T...> , detail::mp_no_type> ::type::type;
# 142
};

So the "..." for "detail::mp_no_type" is missing (as it should be). The rest of "mp_quote" looks the same.

I am using "Cuda compilation tools, release 9.1, V9.1.85" shipped with Debian testing for Cuda 9 tests and "Cuda compilation tools, release 8.0, V8.0.44" of our in house GPU cluster's module system.

[ax3l]

Can you please add your nvidia bug report number for documentation?

[ax3l]

Please also check how compiles with clang -x cuda instead of nvcc and CUDA toolkit 8/9.0/9.1 behave.

[ax3l]

cc @sbastrakov

[theZiz]

I directly wrote a mail as I got Ajax errors while reporting the bug :roll_eyes:
However they did not react yet so I opened a meta ticket to have the bug submission bug fixed before. :grin:

I did check clang -x cuda: The same error happens – but even with cuda 8. :open_mouth:

[ax3l]

I directly wrote a mail as I got Ajax errors while reporting the bug
[...] However they did not react

I understand your frustration. Nevertheless, you might want to re-submit it via the official channel they have, maybe from a different computer. To my experience, that's to most reliable way to get feedback/improvement while other communications can get lost (and did a lot in the past).

[ax3l]

I did check clang -x cuda: The same error happens – but even with cuda 8.

That's weird, they don't use the same splitter: https://llvm.org/docs/CompileCudaWithLLVM.html#compilation-models

[theZiz]

Le'me just retest, maybe I did something wrong.

[psychocoderHPC]

I also tried to compile the mini example with clang7 and got the same errors

$ clang --version
clang version 7.0.0 (https://git.llvm.org/git/clang.git/ 780143bbd516aa39d754e5cab50955321b71fba0) (https://git.llvm.org/git/llvm.git/ bef5f248b85fb47702bd94a11276728a4cb4689f)

$ clang++ -std=c++11 -x cuda main.cpp  --cuda-path=$CUDA_ROOT -I$BOOST_ROOT/include
In file included from main.cpp:1:
In file included from boost_1_66_0_clang7/include/boost/mp11.hpp:11:
In file included from boost_1_66_0_clang7/include/boost/mp11/list.hpp:13:
boost_1_66_0_clang7/include/boost/mp11/detail/mp_append.hpp:136:167: error: cannot
      refer to class template 'append_11_impl' without a template argument list
  ...mp_if_c<(sizeof...(L) > 11), mp_quote<append_111_impl>, mp_quote<append_11_impl>>>::template fn<...
                                                                      ^
boost_1_66_0_clang7/include/boost/mp11/detail/mp_append.hpp:47:261: note: template is
      declared here
  ...class L9 = mp_list<>, class L10 = mp_list<>, class L11 = mp_list<>> struct append_11_impl;
                                                                                ^
boost_1_66_0_clang7/include/boost/mp11/detail/mp_append.hpp:137:1: error: expected a
      type
{
^
boost_1_66_0_clang7/include/boost/mp11/detail/mp_append.hpp:138:2: error: expected a
      type
};

[ax3l]

Interesting, huh... maybe it's just an actual mp11 bug.

@theZiz wherever the root might be, please also report (& link) the issue in mp11 in Boost's trac issue tracker.

[theZiz]

@ax3l Will do so (mp11 trac)

I understand your frustration. Nevertheless, you might want to re-submit it via the official channel they have, maybe from a different computer. To my experience, that's to most reliable way to get feedback/improvement while other communications can get lost (and did a lot in the past).

Well, I just tried with different browsers and slightly different details, but it failed again. I will maybe try at home, maybe our network is blacklisted? But a TU Dresden VPN connection does not work, too... :confused:

[ax3l]

Let's see how you meta-issue on the tracker works out for this one ;-) Just wanted to warn you that this is the most reliable channel to get things changed :)

[theZiz]

If I don't get a respone until next week, I will ask you or @psychocoderHPC to submit the bug for me if it is working for you. :roll_eyes:

[ax3l]

There's also a >>> in rene's last comment... maybe the tokenizer gets into trouble there (although it's valid C++11).

[theZiz]

The problem is, as mentioned in my bug report, that the nvidia preprocessor generates invalid code, so it's not the tokenizer of the compiler itself. I guess it's some hacky half perl script half lex and yacc failing here. :laughing: However I will try to add spaces, would be great, if this would solve the problem...

[ax3l]

We tested if offline: if you refactor the >>> to > > > clang -x cuda (CUDA 8.0) works. Occurs on multiple occasions and only with -x cuda, @psychocoderHPC and I will report it upstream with a minimal reproducer (without mp11 if we find one).

Unfortunately, that's just yet another issue ;)

[ax3l]

I could imagine nvcc is suffering "similar root issues" even if it's located in the splitter, simply because it's in the end the same LLVM codebase... -> does not seem to fix it though...

[theZiz]

Nope, I tested replacing >>> with nvcc 9.1 and it still fails with the same error.

[ax3l]

Clang issue reported upstream and unrelated to what we see in nvcc, at least for CUDA 8. Untested yet if clang 7 + CUDA 9 work when the >>> is work-arounded.

[theZiz]

Okay, I guess I found a work around for boost::mp11.

The problem occurs on code like this:

template<template<class...> class F, class... T>
using mp_defer = mp_if<mp_valid<F, T...>, detail::mp_defer_impl<F, T...>, detail::mp_no_type>;

Everytime mp_if is used, the nvcc preprocessor tries to be "smart" and starts recursively replacing stuff so that in the end detail::mp_no_type...is given to the compiler.

I figured out I can prevent nvcc from doing this with invoking another struct inbetween like this:

template<template<class...> class F, class... T>
struct cuda_workaround
{
    using type = mp_if<mp_valid<F, T...>, detail::mp_defer_impl<F, T...>, detail::mp_no_type>;
};

template<template<class...> class F, class... T>
using mp_defer = typename cuda_workaround< F, T... >::type;

This occurs on 5 places in the boost mp11 code. I added an intermediate struct for every one and now my code compiles with gcc-5 and nvcc 9.

But tbh, I still need to check whether the code is also correct. :rofl: But it looks promising! :+1:

[ax3l]

for the >>> issue with clang: I also opened a PR to mp11 in case they accept it as a valid work-around: https://github.com/boostorg/mp11/pull/20

[theZiz]

I created an issue in the boost trac bug tracker but mp11 is not even listed as part of boost there :open_mouth: , so I guess I will just create a PR for my work around, too. :wink:
https://svn.boost.org/trac10/ticket/13513

[ax3l]

Cool, just open a PR on github as well, link the trac in it and link the PR in trac :)

[theZiz]

It's the plan. :)

[ax3l]

The >>> issue with tokenizing CUDA vs. template code in clang is now fixed (potentially in clang 9+?):
https://bugs.llvm.org/show_bug.cgi?id=37013
https://reviews.llvm.org/D61396

[BenjaminW3]

I will close this ticket.
More and more meta programming constructs get integrated into the c++ standard and our goal is to get rid of dependencies (especially boost) instead of adding new dependencies.