Picongpu: failed to build the Bremsstrahlung example

[cbontoiu]

I tried to build the Bremsstrahlung example as it comes with the 0.4.3 version; no change at all but my installation fails to build it. My commands from the terminal were:

spack load picongpu +adios %[email protected]
pic-create $PIC_EXAMPLES/Bremsstrahlung $HOME/PIC_INPUT/myBremsstrahlung
cd $HOME/PIC_BUILD
pic-configure -b "cuda:75" $HOME/PIC_INPUT/myBremsstrahlung
cd $HOME/PIC_INPUT/myBremsstrahlung
pic-build

The error messages are in the file attached:
complete-build-error.zip

[PrometheusPi]

@cbontoiu Thank you for posting this problem. We will have a look at it.

Just a side remark: with pic-configure -b "cuda:75" $HOME/PIC_INPUT/myBremsstrahlung you create a build directory in $HOME/PIC_BUILD where you could run make install so that the code from $HOME/PIC_INPUT/myBremsstrahlung is compiled and put into $HOME/PIC_INPUT/myBremsstrahlung/bin/. But you never run make install. With the later called pic-build in $HOME/PIC_INPUT/myBremsstrahlung you create a build directory in $HOME/PIC_INPUT/myBremsstrahlung/.build/, where you automatically run make install. So you could skip the two lines:

cd $HOME/PIC_BUILD
pic-configure -b "cuda:75" $HOME/PIC_INPUT/myBremsstrahlung

[PrometheusPi]

The main error message from the log file is:

[ 95%] Built target cuda_memtest
In file included from /home/.../src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/picongpu-0.4.3-gsy5h4idvpfvsvlf2e2qj6xbaptdvg7h/include/pmacc/../picongpu/_defaultParam.loader:53:0,
                 from /home/.../src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/picongpu-0.4.3-gsy5h4idvpfvsvlf2e2qj6xbaptdvg7h/include/pmacc/../picongpu/simulation_defines.hpp:40,
                 from /home/.../src/spack/opt/spack/linux-linuxmint19-westmere/gcc-7.4.0/picongpu-0.4.3-gsy5h4idvpfvsvlf2e2qj6xbaptdvg7h/include/picongpu/main.cpp:24:
/home/.../PIC_INPUT/myBremsstrahlung/include/picongpu/param/speciesDefinition.param:26:10: fatal error: pmacc/meta/conversion/MakeSeq.hpp: No such file or directory
 #include <pmacc/meta/conversion/MakeSeq.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at picongpu_generated_main.cpp.o.Release.cmake:219 (message):
  Error generating
  /home/.../PIC_INPUT/myBremsstrahlung/.build/CMakeFiles/picongpu.dir//./picongpu_generated_main.cpp.o


CMakeFiles/picongpu.dir/build.make:63: recipe for target 'CMakeFiles/picongpu.dir/picongpu_generated_main.cpp.o' failed
make[2]: *** [CMakeFiles/picongpu.dir/picongpu_generated_main.cpp.o] Error 1

[cbontoiu]

ok, but in this case how can I elevate the performance to compute capability 7.5 of CUDA as -b "cuda:75"?

pic-build -b "cuda:75" does not work.

[PrometheusPi]

@cbontoiu I understand your intention. There are two options to do that:

1. using pic-configure and make install inside $HOME/PIC_BUILD

spack load picongpu +adios %[email protected]
pic-create $PIC_EXAMPLES/Bremsstrahlung $HOME/PIC_INPUT/myBremsstrahlung
cd $HOME/PIC_BUILD
pic-configure -b "cuda:75" $HOME/PIC_INPUT/myBremsstrahlung
# plus the following line 
make install
# ignore pic-build 

1. using pic-build inside $HOME/PIC_INPUT/myBremsstrahlung

spack load picongpu +adios %[email protected]
pic-create $PIC_EXAMPLES/Bremsstrahlung $HOME/PIC_INPUT/myBremsstrahlung
# ignore two lines using PIC_BUILD
cd $HOME/PIC_INPUT/myBremsstrahlung
# add backend information to current shell:
export PIC_BACKEND="cuda:75"
pic-build

If you have not set PIC_BACKEND="cuda:75" before, you did not compile for that architecture when using pic-build.

I would recommend option two.

[cbontoiu]

OK, it works. I use the second option but I am forced to use either "cuda:60" or "cuda:70". My card is RTX2060 and should support 7.5 capability
https://en.wikipedia.org/wiki/CUDA
but I couldn't build with neither 6.5 nor 7.5. This makes me think that the code is restricted somehow.

[PrometheusPi]

Interesting! And thank you for your feedback - will will need to have a deeper look into that.

@psychocoderHPC and @sbastrakov are you aware of any restrictions regarding the CUDA compute capability in 0.4.3?

Side remark: there is no compute capability 6.5 - the Pascal architecture ends at 6.2.

Just out of curiosity, does 6.2 and 7.2 compile?

[cbontoiu]

RTX2060 is on Turing architecture.
I tried and neither 6.2 not 7.2 work. Here is the log in the latter case:

[ 12%] Built target cupla
nvcc fatal   : Unsupported gpu architecture 'compute_7.2'
nvcc fatal   : Unsupported gpu architecture 'compute_7.2'
CMake Error at cuda_memtest_generated_cuda_memtest.cu.o.Release.cmake:219 (message):
  Error generating
  /home/cristi/PIC_INPUT/myLaserWakefield/.build/build_cuda_memtest/CMakeFiles/cuda_memtest.dir//./cuda_memtest_generated_cuda_memtest.cu.o
CMake Error at cuda_memtest_generated_tests.cu.o.Release.cmake:219 (message):
  Error generating
  /home/cristi/PIC_INPUT/myLaserWakefield/.build/build_cuda_memtest/CMakeFiles/cuda_memtest.dir//./cuda_memtest_generated_tests.cu.o
build_cuda_memtest/CMakeFiles/cuda_memtest.dir/build.make:645: recipe for target 'build_cuda_memtest/CMakeFiles/cuda_memtest.dir/cuda_memtest_generated_cuda_memtest.cu.o' failed
make[2]: *** [build_cuda_memtest/CMakeFiles/cuda_memtest.dir/cuda_memtest_generated_cuda_memtest.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
build_cuda_memtest/CMakeFiles/cuda_memtest.dir/build.make:339: recipe for target 'build_cuda_memtest/CMakeFiles/cuda_memtest.dir/cuda_memtest_generated_tests.cu.o' failed
make[2]: *** [build_cuda_memtest/CMakeFiles/cuda_memtest.dir/cuda_memtest_generated_tests.cu.o] Error 1
CMakeFiles/Makefile2:221: recipe for target 'build_cuda_memtest/CMakeFiles/cuda_memtest.dir/all' failed
make[1]: *** [build_cuda_memtest/CMakeFiles/cuda_memtest.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 20%] Built target mpiInfo
[ 54%] Built target picongpu-hostonly
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

ERROR: Could not successfully run make install in build directory:
       .build

I also have an additional question.
Setting export PIC_BACKEND="cuda:70" or not, results in the same computation time for the LaserWakefield example, that is 1 min and 23 seconds. Either the compute capability is automatically set anyway or the performance gain is somewhere else but not in speed.

[PrometheusPi]

@cbontoiu did you get the error nvcc fatal : Unsupported gpu architecture 'compute_7.2' also with 7.5? What cuda version do you use (nvcc --version)?

[PrometheusPi]

If I remember correctly not setting a specific compute capability (used to) build a variate of ptx codes that is than executable by various GPUs. This would result in a slightly longer compile and/or start-up phase - not necessarily a longer run time.
But I am not an expert an that - @psychocoderHPC or @sbastrakov might provide more information on that.

[ax3l]

Re: the actual error

The proper workflow above should read way shorter and stay in the example directory for building:

spack load picongpu +adios %[email protected]

pic-create $PIC_EXAMPLES/Bremsstrahlung $HOME/picInputs/myBremsstrahlung
cd $HOME/picInputs/myBremsstrahlung
pic-build -b "cuda:70"

# now execute via: tbg ...

Reference: three-step-workflow

CUDA capability: CUDA 9.2 only builds up to SM 7.2. Since those SMs are forward compatible, there is no reason to worry too much here, just take -b "cuda:70" or -b "cuda:72" (careful, NO .! The error nvcc fatal : Unsupported gpu architecture 'compute_7.2' suggests you did a typo.)

[cbontoiu]

The compute capability error above was obtained for the Laser Wakefield Acceleration example and here is my CUDA compiler
(base) cristi@T7500:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

Yes it fails with 7.5 as well and I don't mix versions here. I have only the 0.4.3 files, examples and executables. Please check the log attached here.

compute-75-error.zip

[ax3l]

See above, just run

pic-build -b "cuda:70"

or

pic-build -b "cuda:72"

if you are using CUDA 9.2. PTX targets for compute capabilities are forward compatible, so your SM 7.5 generation card will work well with CUDA 9.2 and SM 7.0 or 7.2 binaries, too (as long as you have the necessary CUDA drivers installed).

Note: please always post the commands you typed with the error messages, so we can spot typos in usage earlier during support :)

[sbastrakov]

@cbontoiu as of the original build error (unrelated to CUDA versions discussion). It seems like you are using the example setup from dev and this causes the errors when building with the last release. I think so because of the file name being part of the error message.

Here is the dev version of the Bremsstrahlung example and it includes <pmacc/meta/conversion/MakeSeq.hpp>. This file is what your error message complains about, it was named differently in the release.

I am not sure about your setup. In case your $PIC_EXAMPLES is your local git repository, please make sure your current branch is 0.4.3 release (or master).

[sbastrakov]

Edit: fixed the links and file names in my previous message, sorry.

[cbontoiu]

Trying again the master 0.4.3 version of the Bremstrahlung example gives me this error at runtime:

terminate called after throwing an instance of 'std::runtime_error'
  what():  ParticleCalorimeter: parameter fileName and period are not used the same number of times

I did not change anything in the input folder (generated with pic-create) apart from making a 1.cfg file instead of an 8.cfg one and accordingly setting all TBG_devices = 1.

these were all my commands at the terminal:

spack load picongpu +adios %[email protected]
pic-create $PIC_EXAMPLES/Bremsstrahlung $HOME/PIC_INPUT/myBremsstrahlung
cd $HOME/PIC_INPUT/myBremsstrahlung
export PIC_BACKEND="cuda:70"
pic-build
tbg -s bash -c etc/picongpu/1.cfg -t etc/picongpu/bash/mpiexec.tpl ~/PIC_RUN/Bremsstrahlung_02

Otherwise the model it compiles fine. The error is attached here:

error.txt

[sbastrakov]

Sorry we missed it earlier @cbontoiu , thanks for providing more details. It looks like the 8.cfg in the 0.4.3 release is indeed bugged in the linked lines. The missing parameter is at least file, this has been fixed in dev already by #3097, sorry for a sloppy example setup.