Picongpu: St9bad_alloc Error

[cbontoiu]

I got an error on perfectly compiled and valid model just because I increased the mesh. The error is obtained with the PIConGPU: 0.5.0-dev and is about and unhandled exception

Unhandled exception of type 'St9bad_alloc' with message 'std::bad_alloc', terminating

The full terminal transcript is attached in the file below:

error.txt

but in principle refers to /pmacc/memory/buffers/HostBufferIntern.hpp and pmacc/memory/buffers/Buffer.hpp

I will try to allocate more memory in th meantime, instead of

static constexpr uint32_t BYTES_EXCHANGE_X = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Y = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Z = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EDGES = 128 * 1024; // 128 kiB
static constexpr uint32_t BYTES_CORNER = 32 * 1024; // 32 kiB

but maybe you want to investigate
Thank you

[psychocoderHPC]

The error looks like you was running out of memory.
Could you please post your used cfg file, I expect your number of cells was high or you used too much memory for the cell borders.

I will try to allocate more memory in th meantime, instead of

static constexpr uint32_t BYTES_EXCHANGE_X = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Y = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EXCHANGE_Z = 3 * 1024 * 1024;
static constexpr uint32_t BYTES_EDGES = 128 * 1024; // 128 kiB
static constexpr uint32_t BYTES_CORNER = 32 * 1024; // 32 kiB

What values do you used instead of these?

[cbontoiu]

I tried the following combinations for the bytes exchange:

X = 3*   3*   3*    6*
Y = 3*   6*  12*   6*
Z = 3*   3*   3*    6*

but all ended with the same CUDA error.

Below is the cfg file. The number of cells is 7.763e+06 which was easily handled in the past by the GPU. Maybe I am loading the simulation with too many macroparticles Random16ppc instead of Random6ppc but I changed this because of the low weight, 0.0693.
No, the error appears even with Random6ppc.

TBG_wallTime="1000:00:00"
TBG_devices_x=1
TBG_devices_y=1
TBG_devices_z=1
TBG_gridSize="152 336 152"
TBG_steps="40864"
TBG_periodic="--periodic 1 0 1"
TBG_hdf5="--hdf5.period 45 --hdf5.file simData --hdf5.source 'species_all,fields_all'"
TBG_e_pngYX="--e_png.period 45 --e_png.axis yx --e_png.slicePoint 0.5 --e_png.folder pngElectronsYX"
TBG_e_pngXZ="--e_png.period 45 --e_png.axis xz --e_png.slicePoint 0.5 --e_png.folder pngElectronsXZ"
TBG_e_pngYZ="--e_png.period 45 --e_png.axis yz --e_png.slicePoint 0.5 --e_png.folder pngElectronsYZ"
TBG_sumEnergy="--fields_energy.period 45 --e_energy.period 45 --e_energy.filter all"
TBG_e_macroCount="--e_macroParticlesCount.period 45"  
TBG_C_macroCount="--C_macroParticlesCount.period 45" 
TBG_e_countPerSuper="--e_macroParticlesPerSuperCell.period 45"
TBG_C_countPerSuper="--C_macroParticlesPerSuperCell.period 45"
TBG_plugins="!TBG_hdf5 !TBG_sumEnergy !TBG_e_pngYX !TBG_e_pngXZ !TBG_e_pngYZ \
             !TBG_e_macroCount !TBG_C_macroCount !TBG_e_countPerSuper !TBG_C_countPerSuper"
TBG_deviceDist="!TBG_devices_x !TBG_devices_y !TBG_devices_z"
TBG_programParams="-d !TBG_deviceDist \
                   -g !TBG_gridSize   \
                   -s !TBG_steps      \
                   !TBG_periodic      \
                   !TBG_plugins       \
                   --versionOnce"

TBG_tasks="$(( TBG_devices_x * TBG_devices_y * TBG_devices_z ))"
"$TBG_cfgPath"/submitAction.sh

[sbastrakov]

Hello @cbontoiu , the reason does not seem clear to me. So with 7.763e+06 cells, even using 16 macroparticles per cell in each cell (assuming worst case) and 2 species (judging from your .cfg fragment), it should give about 250 million macroparticles. Normally each macroparticle is 32 bytes, so that should be about 8 GB and still fit, and 6 macroparticles per cell should fit for sure. So either particles take way more than I think (or the configuration is actually different), or there is something else consuming lots of memory.

To investigate, is it possible that you reduce the number of macroparticles per cell even more, say to 1, and in case it runs then look at the output for the number of macroparticles that you already have enabled, to see that it matches your expectations?

[cbontoiu]

I confirm that it works with Random1ppc, TBG_gridSize="360 416 360" and 970 mil. macroparticles. It is therefore a problem of memory. Thank you.