Picongpu: how to append -g -lineinfo to nvcc flags

[FindHao]

I tried to modify picongpu/include/picongpu/CMakeLists.txt and picongpu/include/pmacc/CMakeLists.txt with the following lines.

set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -gencode arch=compute_75,code=sm_75 --ptxas-options=-v --debug --generate-line-info ")
set(CMAKE_NVCC_FLAGS "${CMAKE_CUDA_FLAGS} -gencode arch=compute_75,code=sm_75 --ptxas-options=-v --debug --generate-line-info ")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g")

But, when I use nvdisasm to check the cubin files, there are no line information. Could you tell me the correct way to add those flags?

[psychocoderHPC]

Hey @FindHao,

welcome to PIConGPU, and thanks for your question!

There should be no need to manipulate the CMake files.

pic-build -c "-DCMAKE_CXX_FLAGS=-g -DALPAKA_CUDA_ARCH=75 -DALPAKA_CUDA_SHOW_CODELINES=ON -DALPAKA_CUDA_KEEP_FILES=ON"

You should find the cubin files in .build/CMakeFiles/picongpu.dir

• -DCMAKE_CXX_FLAGS=-g will activate debug symbols for host code, not required for cubin files
• -DALPAKA_CUDA_ARCH =75 is defining the architecture, in that case support for NVIDIA Turing
• -DALPAKA_CUDA_SHOW_CODELINES=ON will enable c++ code as comments in the ptx files
• -DALPAKA_CUDA_KEEP_FILES=ON will keep the temporary files in .build/CMakeFiles/picongpu.dir

I tested it with the architecture 70 with nvdisasm main.sm_70.cubin --print-code --print-line-info-ptx --print-line-info.
In my tests I also miss the C++ annotations., the reason could be that alpaka is not setting --debug.

The reason that your direct manipulations of the CMake files not work could be that we flush CUDA_NVCC_FLAGS in alpaka.
https://github.com/ComputationalRadiationPhysics/picongpu/blob/fa864412e591870d04641d48d5ff58289112f35c/thirdParty/cupla/alpaka/cmake/alpakaCommon.cmake#L432
As a workaround, you could add your additional parameter after the flush directly into alpaka.

PS: If you already use PIConGPU, or plan to use it in the near future, please consider adding yourself to our community map! (https://github.com/ComputationalRadiationPhysics/picongpu-communitymap)

[FindHao]

Thanks for your reply. It works!
I'm trying to do some optimizations by our tool. If it goes well, I'll send pull requests.
Thanks.