E3sm: ne30_oQU120 Resolution Input Data Incomplete on Cori

[jasonltorchinsky]

Hello,

I am currently attempting to run the latest version of the E3SM on Cori at NERSC using the quickstart script provided by the E3SM quickstart tutorial. The original configuration of the script (5-day simulation with resolution ne4_oQU240, with the proper machine and project) worked as intended. As did a longer (30-year) simulation.

I am now attempting to recreate some of the figures from this paper, which uses a "global cubed-sphere grid [...] set at ne30np4" (see Section 2.3), and so I would like to use a grid that includes that resolution. I figured that ne30_oQU120 would be a similar enough case to run, so I made the necessary changes to the submit script (attached as .txt), and received the error:

ERROR: Could not find all inputdata on any server

Most of the output of ./run_e3sm.jlturner.csh is attached as error_log.txt. The rest of the output is pasted below:

/global/cscratch1/sd/jlturner/E3SM_simulation/cori-knl.default.A_WCYCL1850_5Y.ne30_oQU120/case_scripts/env_mach_specific.xml already exists, delete to replace

env_batch.xml appears to have changed, regenerating batch scripts
manual edits to these file will be lost!

Model clm no file specified for finidat
Model clm no file specified for finidat
wget failed with output:  and errput /global/cfs/cdirs/e3sm/inputdata/ice/mpas-cice/oQU120/mpas-cice.graph.info.151209.part.5120: Permission denied

wget failed with output:  and errput /global/cfs/cdirs/e3sm/inputdata/ocn/mpas-o/oQU120/mpas-o.graph.info.160318.part.640: Permission denied

Model clm no file specified for finidat
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ice/mpas-cice/oQU120/mpas-cice.graph.info.151209.part.5120' doesn't exist

svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ocn/mpas-o/oQU120/mpas-o.graph.info.160318.part.640' doesn't exist

ERROR: Could not find all inputdata on any server

run_e3sm.jlturner.csh.txt
error_log.txt

[whannah1]

@jasonltorchinsky While we don't really have "supported grids" that would definitely not be one of them if we did.
So are you actually trying to run the MMF? Since I wrote that paper I'd be happy to help you set up some experiments.

[jonbob]

Hi Jason - those error messages are indicative of missing ocn and ice partition files. The oQU120 ocn/ice mesh is relatively coarse, with ~28K surface points and your case is specifying 5120 seaice processors and 640 ocean processors. In general we do not expect performance over ~100 surface points/processor, so the partition files available for that ocn/ice grid are more like 24-256 processors

[whannah1]

@jasonltorchinsky depending on which figures you're looking to reproduce I have some recent results that we haven't published yet that might influence what you are trying to do. Specifically, the grid imprinting we reported in that paper is a bit more complicated than what I first though. Luckily, we recently implemented a fix for all that stuff, but we won't get it published for a bit. Either way, I'm happy to work with you on reproducing our results.

[jasonltorchinsky]

@whannah1 Thank you for the prompt response! I'm not trying to run the MMF (at least, I don't think I am). I'm trying to learn more about the E3SM (understanding the output files and common plots that people create), and I thought it would be a nice exercise to recreate the E3SM parts of some figures from a recent paper. I recreated part of Figure 2 in your paper (the 5-year climatology of precipitation), and noticed the grid resolution was vastly different with the original grid I used (ne4_oQU240)
clim_10Y.pdf
and wanted to use (what I assume to be) a higher-resolution grid. I have some experience used E3SM-diags for generating figures (which look much better than my mock-up), but some of the more interesting figures from your paper (e.g., Figures 8 and 9) don't seem to be built-in to E3SM-diags. My "end goal" at the moment is to recreate the E3SM parts of Figures 2, 4, 5, 6, 7, 8, and 9. Any insight you could offer on doing that (e.g., what compset, resolution, etc. you used, plotting scripts) would be greatly appreciated!

@jonbob Ah, I see! I'm still learning about what exactly the different resolutions are and what all of the options in the script mean, so I really appreciate the insight. If I'm understanding you correctly, the issue is that I am requesting too large of a processor configuration for the given resolution and that I should request fewer? Would a different resolution be better to use (such as ne30_g16)?

[jonbob]

@jasonltorchinsky - you have a couple of choices, either request fewer processors for the ice and ocn in env_mach_pes.xml or use a different ice/ocn grid. For what it's worth, g16 is not an MPAS grid. You could use oEC60to30v3, which along with ne30 is our v1 "low-res" configuration. That one does have decomposition files more in line with what you had specified.

[mt5555]

from the "quick start guide":

./create_newcase --case cmip6-picontrol --compset A_WCYCL1850S_CMIP6 --res ne30_oECv3_ICG --mach cori-knl --user-mods-dir

With the 100km atmosphere grid (ne30), we usually run the "oECv3" ocean grid, which is 60-30km. So that combination will be much better supported. It will require many more nodes than ne4_oQU120 though.

Not sure about oQU120, but in the atmosphere, ne4 is extremly coarse and is mostly used for testing. It's too coarse to capture baroclinic instability, for example.

[whannah1]

@jasonltorchinsky in that case don't worry about Figures 8 and 9, they are pretty specialized for looking at the MJO.

We are moving away from running on the ne30 grid that we used to use. The dynamics will still run on that grid, but we run the physics on a slightly coarser grid, which we refer to as "ne30pg2". The history data files will output on that grid too. There's a paper I submitted that explains this new grid, but it's in review so I can't point you to it. If you're interested in readin about it I'm happy to send you the manuscript.

Since you don't seem to need a coupled ocean, I would suggest running with the "ne30pg2_ne30pg2" grid using the FC5AV1C-L compset.

If you want to plot the E3SM data on it's native grid, then I recommend using python with the PyNGL software package developed at NCAR. Alternative packages like MatPlotLib are not good for the native grid plots. I have a lot of example code for that type of plot if you're interested.

I'm gonna close this issue, but if you have more questions just shoot me an email
[email protected]

[jasonltorchinsky]

@jonbob Thank you for the information! I will follow @mt5555 's suggestion of the ne30_oECv3_ICG grid, which seems to have submitted successfully to the queue.

@whannah1 Thank you for the suggestion! My research work has to do with looking at the MJO (trying to improve how climate models capture the MJO), which is what drew my (advisor's) eye to your paper. Figures 8 and 9 are very illustrative of the MJO, and I'd love to have figures like that in my future work. I will certainly look into the PyNGL software package, and will follow up with you if questions arise!

I truly appreciate all of the insight and help I received, thanks!

[whannah1]

@jasonltorchinsky in that case I should mention that Figure 8 (equatorial wave spectra) was created with NCL using their "canned" routines. These routines have not been ported to python so they are not available in PyNGL yet. The diurnal cycle phase figure (i.e. "Evans plot") was similarly created with NCL.

[whannah1]

@jasonltorchinsky one last comment, I would strongly encourage you to switch to the ne30pg2 grid because it will run faster and not affect the MJO.

[jasonltorchinsky]

@whannah1 Thanks for the suggestion! Would that be the ne30pg2_ne30pg2 grid with the FC5AV1C-L compset that you mentioned before, the A_WCYCL1850S_CMIP6 compset that @mt5555 mentioned, or the A_WCYCL1850 that I was originally using?

[whannah1]

@jasonltorchinsky the compsets that start with "A_WCYCL" use a fully coupled ocean, so if that's what you want then you should use the ne30pg2_oECv3_ICG grid, otherwise I'd recommend the FC5AV1C-L compset with the ne30pg2_ne30pg2 grid for a case with prescribed SSTs that represent the present day, similar to the F2000 compset in CESM. If you want pre-industrial prescribed SST then the F1850C5AV1C-L compset should work.

If you're not specifically looking to investigate how the MJO interacts with a coupled ocean then I would receommend not bothering with it because it will be more expensive and you'll have to run it much longer to deal with drift and interannual variability. Whereas if you're just looking to experiment with aspects of the atmosphere model / convection then I would stick with the prescribed SST compsets. The MMF results in the paper you pointed to are good evidence that the MJO can be improved quite a bit while still using prescribed SST.