Picongpu: check numerics - KHI growth rates

[PrometheusPi]

As suggested in the tests of #3227, we should test KHI growth rates in PIConGPU again. Currently it looks like the current dev produces slightly too low growth rates.
This could be caused by either mismatches in the box sizes and the KHI wave length in the default setup, the use of noisy particle shapes, etc.
I will have a look into that issue.

[PrometheusPi]

I just tested the KHI in a large 3D box and got too small growth rates for the eKHI regime.

Dashed lines represent the theoretically predicted growth rate of the magnetic field energy.
@ax3l Do you remember when you tested this the last time?

[PrometheusPi]

@psychocoderHPC This test used the TSC particle shape - the next one will use PCS shape.

[PrometheusPi]

@KseniaBastrakova In contrast to your analysis in #3227, I did not get any negative growth rates (so far). How did you compute the growth rate?

[PrometheusPi]

@KseniaBastrakova Sorry my fault, I get the same initial negative growth rates for B_x. But otherwise the growth rate looks similarly bad.

[PrometheusPi]

In contrast to @KseniaBastrakova growth rate data, which at least are closer to theoretical predictions.

[psychocoderHPC]

[deleted my last comment] so ignore it in your mails.

[PrometheusPi]

The 3D data for PCS particles shows a vary similar behavior.

[PrometheusPi]

@psychocoderHPC offline comment might be correct - with 512 cells in propagation directions, we only resolve 1.56 periods of the KHI mode. Thus growth might be highly suppressed. I am (again) surprised how huge these waves are - didn't expect such a small fraction of the wave inside of the simulation box after I tripled the box length - we thus do not resolve a single KHI wave in the default setup.

I will test if having a integer number of waves in the box will lead to the correct growth rate.

[PrometheusPi]

I checked with 3200 cells in x direction - and a cell-size and time-step adjustment so that the 10 waves fit perfectly into the simulation box. The result is not much better:

Growth rate B_z^2:

Growth rate B_x^2:

Either I am doing something really stupid (like a bug in my analysis code) or we have a problem.

[sbastrakov]

Do you think it makes sense to try on the last release version?

[PrometheusPi]

@sbastrakov definitely - I will test older versions as soon as I get some more compute resources. Currently I occupy all k20 resources on hemera myself with a different project. (And the other resources are/should be used by other projects/people.)

[psychocoderHPC]

btw: I am currently validate the AMD results and re-run KHI with positrons/electrons with CUDA.
I compared the heating against old documented runs and it looks different. I used this time 4gpus instead of 16 as normal.

new dev branch: 14d31d80e3d3b44293e3af1794fa7cbf88f735e7

old results: https://github.com/ComputationalRadiationPhysics/picongpu/pull/2275#issuecomment-331383352

Note: There was no larger difference over years and now it looks completely different. I will rerun my tests with 16GPUs as soon as we have 16K80 available.

[update] With 16GPUs it is also compleat different compared to my 2017 runs.

• PIConGPU 0.4.3 shows nearly the same plot than the current dev.
• Switching the current dev branch back to the current deposition strategy currentSolver::strategy::StridedCachedSupercells is also not changing the heating much.
• test of PIConGPU 0.3.2 fails: Boost incompatibilities, the environment on HZDR hemera is to new.

[psychocoderHPC]

The heating is nearly equal to 2017 runs. The problem is that the numerical heating plot tool is broken #3324.
Heating with corrected tool: 16gpus currentSolver::strategy::StridedCachedSupercells

[PrometheusPi]

@psychocoderHPC Thank you for that input - this points to a real issue.

[sbastrakov]

What is the status of this issue? #3324 was merged, seems it should have fixed it?

[PrometheusPi]

@sbastrakov Sorry - there has been no progress with this issue - it had nothing to do with the heating tool and thus not solved by #3324

[sbastrakov]

Ah, I misunderstood the conversation, sorry.

[sbastrakov]

@PrometheusPi while investigating another issue, we suspect some setups set the initial temperature improperly. Could you check if this is the case for KHI? Maybe that explains the growth rate not matching the theory.

[PrometheusPi]

@sbastrakov Will do.

[psychocoderHPC]

We have now collisions in PIConGPU. We never resolved the debay length in our KHI setup, maybe activating collisions is required. Add collisons.pram to KHI and change the collision pipe to

using CollisionPipeline = bmpl::vector<Collider<
   binary::RelativisticBinaryCollision,
   MakeSeq_t<Pair<PIC_Electrons, PIC_Ions>, Pair<PIC_Ions, PIC_Ions>, Pair<PIC_Electrons, PIC_Electrons>>,
   Params>>;

With collisions enabled the instability is starting later, therefore you should set the simulation iterations to 4000.

[bussmann]

I want to repeat this as it pops up again and again. Adding collisions to correct wrong heating results is not right.

[psychocoderHPC]

I want to repeat this as it pops up again and again. Adding collisions to correct wrong heating results is not right.

What will be the right way?

[bussmann]

First, one has to check if PIC is valid. If si, one has rigorously check what causes the heating. For KHI I would suspect too uniform initialization and not enough particles per cell. The solution of the equation of motion at strong magnetic fields also plays a roll IMHO. So, check initialization, pusher, field solver, sim size