E3sm: High-res G case failed with 'ERROR: lon lat cosz sizes disagree' on Summit

[minxu74]

When I ran a high-res G-case on Summit using 48600 cores, the model failed at the first time step with the error:
"ERROR: lon lat cosz sizes disagree" in the cime/src/share/streams/shr_tInterp_mod.F90.

But it ran successfully with the same configuration but use 16000 cores.

Compset: GMPAS-IAF, res: T62_oRRS18to6v3

[rljacob]

I think that is from one of the data models. Which log files was that error in?

[worleyph]

Now I remember ... you are overdecomposing the T62-based components. You need to fix the decompostion to something like 9600 and only increase OCN, ICE, and CPL for the G case.

[minxu74]

Thanks.
@rljacob the error is logged in the e3sm.log.
and @worleyph I will follow your instruction and update the results here asap.

[minxu74]

@worleyph Yes. After reducing the PES from 48000 to 16000 for all components except for OCN, ICE, and CPL, the model was run successfully. Before I close the issue, I wonder if you have an estimation of maximum MPI tasks for each component in the usual grid settings.

@rljacob Is it possible to give more useful information about this kind of over-decomposition?
Thanks.

[rljacob]

There is an issue open for this. https://github.com/ESMCI/cime/issues/2949