E3sm: Various tests failing unless we set finidat='' (that were not failing before)

[ndkeen]

For example, the following tests fail on cori-knl (I doubt it is machine dependent), but will run if I add finidat='' to user_nl_clm

SMS_PT_Ld2.ne120_ne120.FC5AV1C-H01A
SMS_Ld5.ne11_ne11.FC5AV1C-L
SMS_PT.ne30_oECv3_ICG.A_WCYCL1950S_CMIP6_LRtunedHR

The actual error is:

   0:  pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1288 : 
   0:  NetCDF: Invalid dimension ID or name
   0: Image              PC                Routine            Line        Source             
   0: e3sm.exe           0000000004A43146  Unknown               Unknown  Unknown
   0: e3sm.exe           0000000002E4A7CA  pio_support_mp_pi         118  pio_support.F90
   0: e3sm.exe           0000000002E48937  pio_utils_mp_chec          59  pio_utils.F90
   0: e3sm.exe           0000000002E2C1C0  nf_mod_mp_pio_inq        1288  nf_mod.F90
   0: e3sm.exe           000000000238C32F  ncdio_pio_mp_chec         354  ncdio_pio.F90.in
   0: e3sm.exe           00000000023D0B08  restfilemod_mp_re        1072  restFileMod.F90
   0: e3sm.exe           00000000023CEDAE  restfilemod_mp_re         486  restFileMod.F90
   0: e3sm.exe           0000000002278114  clm_initializemod         722  clm_initializeMod.F90
   0: e3sm.exe           000000000224ADF5  lnd_comp_mct_mp_l         256  lnd_comp_mct.F90
   0: e3sm.exe           000000000042B1EF  component_mod_mp_         246  component_mod.F90
   0: e3sm.exe           000000000041A337  cime_comp_mod_mp_        1284  cime_comp_mod.F90
   0: e3sm.exe           0000000000427F14  MAIN__                    122  cime_driver.F90

Note these tests are using a repo with some updated modules on cori, but I've also seen the same behavior with default repo. I can verify if needed. Here is an example case location:
/global/cscratch1/sd/ndk/acme_scratch/cori-knl/mfmu2019/SMS_Ld5.ne11_ne11.FC5AV1C-L.cori-knl_intel.r01

[rljacob]

@thorntonpe and @bishtgautam , 2 of our high-res test cases are failing because of the finidat change. I don't think those are cases where we want to do a cold start so can you, Gautum, please remake the finidat files.

ne120_ne120.FC5AV1C-H01A
ne120_oRRS18v3_ICG.A_WCYCL2000_H01AS

[bishtgautam]

Hi @cameronsmith1, I'm generated new land initial condition files for few failing tests and compsets. I noticed that for the A_WCYCL1950S_CMIP6_LRtunedHR, I had generated the land IC file (clmi.A_WCYCL1950S_CMIP6_LRtunedHR.ne30_oECv3_ICG_simyr1950_c20180316.nc) by remapping a restart file that you had provided via:

interpinic \
-i /global/cscratch1/sd/pjc/for_Gautam/sample_lnd_ne120_restart/20171208.beta3rc10_H1.ne30_oECv3_ICG.edison.clm2.r.1950-01-01-00000.nc \
-o ne30_oECv3_ICG.A_WCYCL1950S_CMIP6_LRtunedHR.c706dd419.clm2.r.0001-01-02-00000.nc

Where can I find 20171208.beta3rc10_H1.ne30_oECv3_ICG.edison.clm2.r.1950-01-01-00000.nc?

[PeterCaldwell]

We used the beta3rc10 restart because the CMIP DECK wasn't available yet. I think for this new version we should use the restart for 1950 from a DECK historical run: https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/537985236/DECKv1+historical+simulations . I'll try to figure out where exactly the needed file sits.

[PeterCaldwell]

Hi @bishtgautam - I extracted the clm2.r file for 1950 from the CMIP DECK historical simulation ensemble member 5 (the one started after the most PI spinup) and stuck it here for you to grab:
/global/cscratch1/sd/petercal/junk/ForGautam/archive/rest/1950-01-01-00000/20180307.DECKv1b_H5.ne30_oEC.edison.clm2.r.1950-01-01-00000.nc

Note that chmod and chgrp -R commands are still processing, so you may not be able to see the file right away. Thanks for your help on this!

[PeterCaldwell]

@bishtgautam - it took all day, but you should now have permissions to grab that file. ;-) Let me know if you have any trouble.

[bishtgautam]

@ndkeen, @PeterCaldwell, When you get a chance, could you test out PR-#2741? Thanks.

[ndkeen]

Using the branch noted above, I tried the following two tests and verified they completed.

SMS_PT_Ld2.ne120_oRRS18v3_ICG.A_WCYCL2000_H01AS.cori-knl_intel.cam-cosplite 
SMS_PT.ne30_oECv3_ICG.A_WCYCL1950S_CMIP6_LRtunedHR.cori-knl_intel.cam-cosplite

Should I expect that the other 2 tests noted will also work now?

SMS_PT_Ld2.ne120_ne120.FC5AV1C-H01A.cori-knl_intel
SMS_Ld5.ne11_ne11.FC5AV1C-L.cori-knl_intel

[PeterCaldwell]

Thanks Noel! Yeah, I think the other two tests should work now too, but I don't know anything about those compsets so I was ignoring them. It would be good if you could test them too...

[ndkeen]

I ran the following 2 tests which both completed (not looking at any other metrics)

SMS_Ld5.ne11_ne11.FC5AV1C-L.cori-knl_intel
SMS_PT_Ld2.ne120_ne120.FC5AV1C-H01A.cori-knl_intel

[bishtgautam]

Thanks, @ndkeen for testing the PR.

[worleyph]

FYI, I just now tried

 -compset A_WCYCL1950S_CMIP6_LRtunedHR -res ne30_oECv3_ICG

and

  -compset A_WCYCL1950S_CMIP6_LR -res ne30_oECv3_ICG

and they both failed with, for example,

  704:  pio_support::pio_die:: myrank=          -1 : ERROR: nf_mod.F90:        1288 :
  704:  NetCDF: Invalid dimension ID or name

[tangq]

I tried yesterday on cori-knl
--compset F1950C5-CMIP6-HR --res ne120_oRRS18v3
and failed with the same errors as @worleyph
pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1288 :
NetCDF: Invalid dimension ID or name

[rljacob]

@tangq and @worleyph try that case on the branch bishtgautam/lnd/updated-ic-at-ne30-and-ne120 in PR #2741