Climatemachine.jl: Tracers performance improvement

[smarras79]

I see that the tracers are added with a whole additional set of loops over i,j,k after closing the i,j,k loops for the states.
Unless there is a specific reason for that choice, I think that the tracers should be moved within the loops where the states are calculated.

[jkozdon]

We were told that there could be 10s to 100s of tracers. This significantly increase register pressure on the GPU, so it would not be feasible to do all the variables at the same time. We also need to compute the derivatives in shared memory on the GPU, so we are inherently limited by the size of GPU shared memory.

We are likely moving away from these cubed kernels where we launch one GPU thread per DOF. Instead we will launch a 2-D block of threads and loop up the element. Some work suggests that this is the preferred approach for new GPU hardware.

What likely does need to happen is to handle blocks of tracers together, namely have a loop over the loops and block up the tracer variables into groups.

[jkozdon]

Why do you think it would be performance improvement to move them in?

[glwagner]

To what lines in the code does this issue refer? This discussion is also potentially relevant to Oceananigans.jl development. @ali-ramadhan

[smarras79]

Why do you think it would be performance improvement to move them in?

We were told that there could be 10s to 100s of tracers. This significantly increase register pressure on the GPU, so it would not be feasible to do all the variables at the same time. We also need to compute the derivatives in shared memory on the GPU, so we are inherently limited by the size of GPU shared memory.

We are likely moving away from these cubed kernels where we launch one GPU thread per DOF. Instead we will launch a 2-D block of threads and loop up the element. Some work suggests that this is the preferred approach for new GPU hardware.

What likely does need to happen is to handle blocks of tracers together, namely have a loop over the loops and block up the tracer variables into groups.

In that case then I'd suggest to keep the 4 water tracers alone (I think that it won't be more than ql, qv, qi, and possibly). I will send you a proposed solution via slack in a bit; that solution should allow us to calculate the gas constant and Pressure as a function of water content regardless of how the passive tracers are treated.

[smarras79]

Why do you think it would be performance improvement to move them in?

I was thinking about reducing the number of redundant loops to keep the number of operations smaller.

[jkozdon]

@glwagner

VanillaAtmosDiscretizations_cuda

VanillaAtmosDiscretizations_kernels

[jkozdon]

I was thinking about reducing the number of redundant loops to keep the number of operations smaller.

It's all about looping through memory, so if we're looping through memory in a coalesced fashion I think it will be OK. The blocking of tracers will hopefully get us some instruction level parallelism as well to improve performance. But we store i,j,k as the fastest index, not state so there is no reason that we need to load these earlier unless the other state updates depend on them.

[jkozdon]

will send you a proposed solution via slack in a bit;

Why not just code it up and submit a pull request? The GPU side will also need to be updated.

[smarras79]

will send you a proposed solution via slack in a bit;

Why not just code it up and submit a pull request? The GPU side will also need to be updated.

Will do that. thanks

[charleskawczynski]

We were told that there could be 10s to 100s of tracers. This significantly increase register pressure on the GPU, so it would not be feasible to do all the variables at the same time. We also need to compute the derivatives in shared memory on the GPU, so we are inherently limited by the size of GPU shared memory.

We are likely moving away from these cubed kernels where we launch one GPU thread per DOF. Instead we will launch a 2-D block of threads and loop up the element. Some work suggests that this is the preferred approach for new GPU hardware.

What likely does need to happen is to handle blocks of tracers together, namely have a loop over the loops and block up the tracer variables into groups.

Just to note, they will be 10s to 100s of 1D fields and fortunately not 3D fields.

[lcw]

@charleskawczynski, do you mean 100s of 1D fields per column? If so then there is still a potential for a large memory requirement. If we do it right, the compute kernels should be as expensive as a memcpy for the data they touch.

[charleskawczynski]

Yes, 100s of 1D fields per column.. That's a good point, there will be many instances of these single column models.

[lcw]

How many 1D elements (aka elements in the vertical) are you expecting?

[charleskawczynski]

Higher resolution compared to the dycore. I'm not 100% sure but I think ~100 - ~150 elements? It'd be nice for @tapios to confirm this though.

[tapios]

Number of elements in the vertical should be flexible. O(100) for now sounds right, but we want to be able to run at coarser and finer resolution.

For global tracers, we need less than 10 for now (most importantly, 3 water tracers). More may come later (once chemistry is added) but is not important now.

[jkozdon]

old dycore is going away...