Picongpu: problems with superCellsize

[cbontoiu]

Hello,

I prepared a 3D simulation based on using SuperCellSize = mCT::Int< 8, 8, 8 >;. In the .cfg file I know that my numbers of cells must be multiple of 8 in this case. So I need to use for example TBG_gridSize="360 296 416" but this fails with the message

what():  expression (gridSizeLocal[i] % MappingDesc::SuperCellSize::toRT()[i] == 0) failed in file (/home/quasar/src/spack/opt/spack/linux-linuxmint19-skylake/gcc-7.4.0/picongpu-0.4.3-2pwefzezdfmjduyxyl77kgyt4h2yj3wf/include/pmacc/../picongpu/simulationControl/MySimulation.hpp:768) : 

which leads to

private:

    template<uint32_t DIM>
    void checkGridConfiguration(DataSpace<DIM> globalGridSize, GridLayout<DIM>)
    {

        for(uint32_t i=0;i<simDim;++i)
        {
            // global size must be a devisor of supercell size
            // note: this is redundant, while using the local condition below
            PMACC_VERIFY(globalGridSize[i] % MappingDesc::SuperCellSize::toRT()[i] == 0);
            // local size must be a devisor of supercell size
            PMACC_VERIFY(gridSizeLocal[i] % MappingDesc::SuperCellSize::toRT()[i] == 0);
            // local size must be at least 2 supercells (1x core + 2x border)
            // note: size of border = guard_size (in supercells)
            PMACC_VERIFY(gridSizeLocal[i] / MappingDesc::SuperCellSize::toRT()[i] >= 2 * GuardSize::toRT()[i] + 1);
        }
    }

I checked other combinations such as TBG_gridSize="80 80 80", TBG_gridSize="160 160 160", TBG_gridSize="320 320 320" and all work, but not my particular combination.

I think this is something you want to look into as a potential bug, or maybe please help me to understand what happens there.
Thank you.

Cristian

[steindev]

Hey @cbontoiu, you are right, you need to specify your grid size such that an integer number of supercells fits into it.
But this is not all.
You actually have to fit an integer number of supercells per gpu and the minimum number of supercells per gpu is three.
Would you please post your -d option to see if this is the source of the error?
I assume you use 2 gpus in the y direction, since 296 cells on two gpus means 148 cells per gpu which is not divisible by 8.

Would you also please post the PIConGPU version you are using? Since an automatic grid resizing that avoids this error has been implemented 2 years ago with #2840.

[sbastrakov]

I think @steindev is right.

From the code piece you posted and the output it seems the check on the global grid size passed so globally grid has an integer number of supercells. But for local grid size (for one GPU) it didn't. Note that you can still make this global grid size work by manually distributing cells slightly unevenly, using --gridDist parameter that I mentioned yesterday while anwering another post of yours.

[cbontoiu]

Hello and thanks for your swift replies. It is true that I assign the y-cells to two GPUS and I use a moving window as

TBG_wallTime="100:00:00"

TBG_devices_x=1
TBG_devices_y=2
TBG_devices_z=1

TBG_gridSize="320 320 320"
TBG_steps="61315"
TBG_periodic="--periodic 1 0 1"
TBG_movingWindow="-m --windowMovePoint 0.9"

the version of picongpu is 0.4.3 I don't know how to handle the -d option and I would need some reference material or examples in order to use ---gridDist as I never tried it, please.

[sbastrakov]

What @steindev meant by -d are just values of your TBG_devices_... variable.

So I think from your original configuration of TBG_gridSize="360 296 416" that didn't work you could either make a change to

• TBG_gridSize="360 304 416" as then it will make 152 cells in Y per GPU, which is a multiple of 8
• Or try keeping the original gridSize and distribute it in Y as 144 vs. 152 by having TBG_gridSize="360 296 416 --gridDist '360' '144,152' '416'"

[cbontoiu]

OK, thank you. It seems that my new setup can hold TBG_gridSize="480 480 480" so it is even better. I might get problems with the memory allocation as ionization progresses and there are more particles to hold. For now

struct DefaultExchangeMemCfg
    {
        // memory used for a direction
        static constexpr uint32_t BYTES_EXCHANGE_X = 2 * 1024 * 1024; 
        static constexpr uint32_t BYTES_EXCHANGE_Y = 4 * 1024 * 1024; 
        static constexpr uint32_t BYTES_EXCHANGE_Z = 2 * 1024 * 1024; 
        static constexpr uint32_t BYTES_EDGES = 128 * 1024; 
        static constexpr uint32_t BYTES_CORNER = 32 * 1024; 
    };

[PrometheusPi]

@cbontoiu Just as a warning, the --gridDist parameters suggested by @sbastrakov will not work for a moving window. For a Moving window, y-grid distribution needs to be equally spaced. Thus combining --gridDist '360' '144,152' '416'" with -m will fail.

[sbastrakov]

Ah, sorry, I forgot it does not stack with the moving window.