Picongpu: Send/receive buffer for species

[berceanu]

I am testing an LWFA run with Nitrogen 5+ and get this output:

Running program...
PIConGPU: 0.4.0
  Build-Type: Release

Third party:
  OS:         Linux-4.15.0-38-generic
  arch:       x86_64
  CXX:        GNU (7.3.0)
  CMake:      3.12.3
  CUDA:       9.2.88
  mallocMC:   2.3.0
  Boost:      1.65.1
  MPI:
    standard: 3.1
    flavor:   OpenMPI (3.1.2)
  PNGwriter:  0.7.0
  libSplash:  1.7.0 (Format 4.0)
  ADIOS:      1.13.1
PIConGPUVerbose PHYSICS(1) | Sliding Window is ON
PIConGPUVerbose PHYSICS(1) | used Random Number Generator: RNGProvider3XorMin seed: 42
PIConGPUVerbose PHYSICS(1) | Courant c*dt <= 1.00229 ? 1
PIConGPUVerbose PHYSICS(1) | species e: omega_p * dt <= 0.1 ? 0.0209094
PIConGPUVerbose PHYSICS(1) | species e_nitro: omega_p * dt <= 0.1 ? 0.0209094
PIConGPUVerbose PHYSICS(1) | y-cells per wavelength: 18.0587
PIConGPUVerbose PHYSICS(1) | macro particles per device: 268435456
PIConGPUVerbose PHYSICS(1) | typical macro particle weighting: 4945.05
PIConGPUVerbose PHYSICS(1) | UNIT_SPEED 2.99792e+08
PIConGPUVerbose PHYSICS(1) | UNIT_TIME 1.39e-16
PIConGPUVerbose PHYSICS(1) | UNIT_LENGTH 4.16712e-08
PIConGPUVerbose PHYSICS(1) | UNIT_MASS 4.50464e-27
PIConGPUVerbose PHYSICS(1) | UNIT_CHARGE 7.92284e-16
PIConGPUVerbose PHYSICS(1) | UNIT_EFIELD 1.22627e+13
PIConGPUVerbose PHYSICS(1) | UNIT_BFIELD 40903.8
PIConGPUVerbose PHYSICS(1) | UNIT_ENERGY 4.04856e-10
initialization time:  5min 20sec 214msec = 320 sec
  0 % =        0 | time elapsed:       2min  1sec 995msec | avg time per step:   0msec
Send/receive buffer for species e is to small (max: 16384, direction: 2, retries: 1)
Send/receive buffer for species e is to small (max: 16384, direction: 2, retries: 1)
Send/receive buffer for species e is to small (max: 16384, direction: 2, retries: 1)
[...]
Send/receive buffer for species e is to small (max: 16384, direction: 2, retries: 1)
  5 % =     3500 | time elapsed:      20min 59sec 772msec | avg time per step: 295msec
 10 % =     7000 | time elapsed:      41min 24sec 106msec | avg time per step: 319msec
 15 % =    10500 | time elapsed:   1h  3min 41sec  96msec | avg time per step: 322msec
 20 % =    14000 | time elapsed:   1h 24min 21sec 506msec | avg time per step: 324msec

It seems the simulation is still running, but do you have any idea of what might be causing these messages to appear?

[ax3l]

The option is raised to inform you, that "exchange buffers" that move particles from one device to another are not large enough to do it in one go but with a resend, chunking them and potentially introducing some latency (but that's all; no wrong results or anything caused by this).

You can carefully increase the size of these buffers in memory.param: https://github.com/ComputationalRadiationPhysics/picongpu/blob/0.4.1/include/picongpu/param/memory.param#L84-L92 to fit your needs. One sometimes needs this, e.g. if you added significantly more PPC than the example you pic-created from or have a domain decomposition with significant particles going over one surface, etc.

[berceanu]

I see, thanks :) What I find strange is that this message is shown over 300 times as the simulation progresses from 0% to 5% and then another 30 times from 90% to 95%.

[ax3l]

Yes, probably because only during that time it's triggered by heavy local particle movement. Maybe your injection is pushing over the longitudinal GPU domain decomposition borders during those intervals (see our moving window implementation).

Not that this would be important, but if you are super curious, we have two plugins to diagnose macro particle memory:

• https://picongpu.readthedocs.io/en/0.4.1/usage/plugins/countPerSupercell.html
• https://picongpu.readthedocs.io/en/0.4.1/usage/plugins/resourceLog.html

[psychocoderHPC]

Info: if you have multiple species and only one species create this warning you can also adjust the buffer sizes only for one species instead for all.

https://github.com/ComputationalRadiationPhysics/picongpu/blob/f6f8ebd7a8771a1ed7d07e6f14d03011668235c0/include/picongpu/param/speciesAttributes.param#L237-L255

struct ElectronsExchangeMemCfg
{
    static constexpr uint32_t BYTES_EXCHANGE_X = 5 * 1024 * 1024;
    static constexpr uint32_t BYTES_EXCHANGE_Y = 5 * 1024 * 1024;
    static constexpr uint32_t BYTES_EXCHANGE_Z = 5 * 1024 * 1024;
    static constexpr uint32_t BYTES_CORNER = 16 * 1024;
    static constexpr uint32_t BYTES_EDGES = 16 * 1024;
};

// in speciesDefinition.param
using ParticleFlagsElectrons = MakeSeq_t<
    particlePusher< UsedParticlePusher >,
    shape< UsedParticleShape >,
    interpolation< UsedField2Particle >,
    current< UsedParticleCurrentSolver >,
    massRatio< MassRatioElectrons >,
    chargeRatio< ChargeRatioElectrons >,
    exchangeMemCfg<ElectronsExchangeMemCfg> // this adds the electron specific exchange memory config to the species
>;