E3sm: Run failed on compy

[xuezhengllnl]

I'm trying to run a piControl case on Compy with the latest master code (using run script version 3.0.18). The simulation failed at the initialization stage. Here are the error messages in e3sm.log.XXX:
3160: NC_FORMAT_NETCDF4 is not yet supported
4400: NC_FORMAT_NETCDF4 is not yet supported
0: pio_support::pio_die:: myrank= -1 : ERROR: ionf_mod.F90: 235 :
0: Feature is not yet supported.
2400: pio_support::pio_die:: myrank= -1 : ERROR: ionf_mod.F90: 235 :
2400: Feature is not yet supported.

I loaded netcdf/4.6.3 mvapich2/2.3.1 gcc/4.8.5 in .bash_profile.

[singhbalwinder]

As a temporary solution you can change PIO_TYPENAME in env_run.xml to netcdf for all the components. @rljacob is working on converting all input files to netcdf 3 format.

 <entry id="PIO_TYPENAME">
      <type>char</type>
      <valid_values>netcdf,pnetcdf,nothing</valid_values>
      <desc>pio io type</desc>
      <values>
        <value compclass="ATM">netcdf</value>
        <value compclass="CPL">netcdf</value>
        <value compclass="OCN">netcdf</value>
        <value compclass="WAV">netcdf</value>
        <value compclass="GLC">netcdf</value>
        <value compclass="ICE">netcdf</value>
        <value compclass="ROF">netcdf</value>
        <value compclass="LND">netcdf</value>
        <value compclass="ESP">netcdf</value>
        <value compclass="IAC">netcdf</value>
      </values>
    </entry>

[xuezhengllnl]

As a temporary solution you can change PIO_TYPENAME in env_run.xml to netcdf for all the components. @rljacob is working on converting all input files to netcdf 3 format.

Thank you for your solution. It worked! However, I still cannot reproduce the DECKv1b piControl run with master code, because the land initial file has changed. The error message in e3sm.log.XX:
0: pio_support::pio_die:: myrank= -1 : ERROR: nf_mod.F90: 1288 :
0: NetCDF: Invalid dimension ID or name
0: e3sm.exe 00000000034682FA Unknown Unknown Unknown
0: e3sm.exe 0000000003247071 pio_support_mp_pi 118 pio_support.F90
0: e3sm.exe 0000000003245092 pio_utils_mp_chec 74 pio_utils.F90
0: e3sm.exe 000000000322C8A2 nf_mod_mp_pio_inq 1288 nf_mod.F90
0: e3sm.exe 00000000027CA964 ncdio_pio_mp_chec 354 ncdio_pio.F90.in
0: e3sm.exe 0000000001FCE971 restfilemod_mp_re 1221 restFileMod.F90
0: e3sm.exe 0000000001EAA6FB clm_initializemod 722 clm_initializeMod.F90
0: e3sm.exe 0000000001E947D8 lnd_comp_mct_mp_l 281 lnd_comp_mct.F90
0: e3sm.exe 000000000043C074 component_mod_mp_ 257 component_mod.F90
0: e3sm.exe 000000000042A784 cime_comp_mod_mp_ 1338 cime_comp_mod.F90
0: e3sm.exe 0000000000438FB9 MAIN__ 122 cime_driver.F90
0: e3sm.exe 00000000004183E2 Unknown Unknown Unknown
0: libc-2.17.so 00002B75B963D3D5 __libc_start_main Unknown Unknown
0: e3sm.exe 00000000004182E9 Unknown Unknown Unknown

[singhbalwinder]

Tagging @bishtgautam as he might know if there exists a new file to go with the recent version of master.

[rljacob]

What is your create_newcase command?

[xuezhengllnl]

What is your create_newcase command?

/qfs/people/zhen797/E3SM_code//20190815_master/cime/scripts/create_newcase --case 20190815.DECKv1b_piControl_test.ne30_oECv3_ICG.compy --compset A_WCYCL1850S_CMIP6 --script-root /compyfs/zhen797/E3SM_simulations//20190815.DECKv1b_piControl_test.ne30_oECv3_ICG.compy/case_scripts --res ne30_oECv3_ICG --pecount XL --handle-preexisting-dirs u --mach compy --output-root /compyfs/zhen797/E3SM_simulations/20190815.DECKv1b_piControl_test.ne30_oECv3_ICG.compy --project e3sm

I used a run script based on https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/DECKv1/ne30_oEC/run_acme.20180129.DECKv1b_piControl.ne30_oEC.edison.csh

It is a hybrid case with restart files from previous DECKv1b simulations. I'm trying an initial case now.

Thanks,
-Xue

[rljacob]

Right the hybrid run restart files have the old land format. They won't work on master without modification.

We should change that case so it is not by default a hybrid run on master. That was useful for maint-1.0.

[rljacob]

@golaz should cases with A_WCYCL1850S_CMIP6 ne30_oECv3_ICG still start from those restart files or should we do initial cases until a new restart set is available?

[rljacob]

Well now I'm not sure. The test SMS_Ld2.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.anvil_intel.allactive-v1cmip6 works with latest next so master should also be able to read that file.

[rljacob]

Ok I see. The hybrid setting was already removed from master so that test and your initial case run on master will work. But if you try to use those old restart files, it won't work.

[xuezhengllnl]

Ok I see. The hybrid setting was already removed from master so that test and your initial case run on master will work. But if you try to use those old restart files, it won't work.

Thank you for your explanation. I tried an initial run with the latest master but it still failed with different error messages: (I used the same run script, set model_start_type = initial; NO old restart files from 20180129.DECKv1b_piControl.ne30_oEC.edison)
srun: error: n0156: task 3025: Segmentation fault (core dumped)
srun: Terminating job step 5516.0
0: slurmstepd: error: * STEP 5516.0 ON n0001 CANCELLED AT 2019-08-16T10:14:30 *
1: forrtl: error (78): process killed (SIGTERM)
1: Image PC Routine Line Source
1: e3sm.exe 0000000003476754 Unknown Unknown Unknown
1: libpthread-2.17.s 00002B18F56C95D0 Unknown Unknown Unknown
1: libpthread-2.17.s 00002B18F56C86DE read Unknown Unknown
1: libmpi.so.12.0.0 00002B18F4BFD76F Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F4BFA4CB Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F493C23C Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F4CFABB9 Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F471B32E Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F4712818 Unknown Unknown Unknown
1: libmpi.so.12.0.0 00002B18F4707D1C MPI_Init Unknown Unknown
1: libmpifort.so.12. 00002B18F3E58CFB MPI_INIT Unknown Unknown
1: e3sm.exe 0000000000437273 cime_comp_mod_mp_ 671 cime_comp_mod.F90
1: e3sm.exe 0000000000438DE5 MAIN__ 61 cime_driver.F90
1: e3sm.exe 00000000004183E2 Unknown Unknown Unknown
1: libc-2.17.so 00002B18F5BFA3D5 __libc_start_main Unknown Unknown
1: e3sm.exe 00000000004182E9 Unknown Unknown Unknown
14: forrtl: error (78): process killed (SIGTERM)

[bbye]

I think that node (n0156) is bad on compy. I had a failure yesterday with it for no known reason; resubmit was fine. @jonbob also had some trouble on that node (see slack compy-users).

[xuezhengllnl]

I think that node (n0156) is bad on compy. I had a failure yesterday with it for no known reason; resubmit was fine. @jonbob also had some trouble on that node (see slack compy-users).

Thanks! It is running now. (not on n0156)

[singhbalwinder]

I think that node (n0156) is bad on compy. I had a failure yesterday with it for no known reason; resubmit was fine. @jonbob also had some trouble on that node (see slack compy-users).

Has it been reported to rc-support?

[singhbalwinder]

I just noticed that 156th node is down, so a resubmit should work.

[bbye]

I have not reported it. The first time I saw anything was on August 2nd on Slack. Yesterday was the first time I got that node and it crashed. I also had one job crash earlier today on that node. If it's down now, perhaps someone has taken notice?

[rljacob]

Closing. Main problem was using incompatible restart files.

[golaz]

Reopening this. There are cases when we need to use restart files from E3SMv1 DECK simulations as initial conditions for new simulations (with master). For example, @kaizhangpnl needs to run an F AMIP simulation starting from restart files in 1980. What is the procedure to modify the land files to make them "compatible"?

[bishtgautam]

@golaz, @kaizhangpnl, I can create the new land restart file. Can you tell me the full ./create_newcase command that you are using? Also, what is the commit/branch for which the F AMIP compset works and for which one it fails?

[bishtgautam]

Btw, is the compset in question F1850SC5-CMIP6 as pointed out by @rljacob in this comment?

[kaizhangpnl]

@bishtgautam I was trying to perform a branch run using restart files from the DECKv1 simulations. The run failed, but it seems due to a problem in the atm model initiation. Nevertheless, we think the old land restart file from the DECKv1 AMIP simulation might not work in the branch run, since the input data dimension is different from that in the current model.

see this page for details: https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/1027834380/compy.E3SM+master+201908.F20TRC5-CMIP6+A01+rest.ne30+ne30

BTW, the startup run also fails if we don't specify the new initial data for clm. I think the namelist use_case file for F20TRC5-CMIP6 needs to changed:
https://github.com/E3SM-Project/E3SM/blob/master/components/clm/bld/namelist_files/use_cases/20thC_CMIP6_transient.xml

see https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/1020199348/AMIP+simulation+based+on+a+recent+master

[bishtgautam]

@kaizhangpnl, I have created the following two new land initial conditions:

clmi.I1850CLM45.ne30_ne30.cac8d284e.clm2.r.0001-01-06-00000_c20190826.nc
clmi.I1850CLM45.ne30_oECv3.cac8d284e.clm2.r.0001-01-06-00000_c20190826.nc

These files are currently available on Compy at /compyfs/inputdata/lnd/clm2/initdata_map. Could test them out? I will upload these new initial conditions to the e3sm inputdata server soon.

[kaizhangpnl]

@bishtgautam Thank you for the help and sorry for the delay. I missed the automatic email notification and didn't notice the last comment until yesterday. The new restart file works well for an initial run (now I use the ne30_oECv3 grid for my simulations).

Would it be possible to modify the land restart files generated from a previous model version (maint-1.0 in our case), so that we (@kaizhangpnl @xuezhengllnl ) can use them to perform some branch runs? There are some additional variables (compared to the old restart file) in the new restart file. Will they be used in the model simulation (or read but not used?)? Thanks!

[kaizhangpnl]

The missing values (in the old restart file) are:

double FSD24H_VALUE(topounit) ;
double FSD240H_VALUE(topounit) ;
double FSI24H_VALUE(topounit) ;
double FSI240H_VALUE(topounit) ;
double nlim_m_VALUE(pft) ;
double plim_m_VALUE(pft) ;
double eflx_dynbal_amt_to_dribble(gridcell) ;
double qflx_liq_dynbal_amt_to_dribble(gridcell) ;
double qflx_ice_dynbal_amt_to_dribble(gridcell) ;
int FSD24H_PERIOD ;
int FSD240H_PERIOD ;
int FSI24H_PERIOD ;
int FSI240H_PERIOD ;
int nlim_m_PERIOD ;
int plim_m_PERIOD ;

[bishtgautam]

Would it be possible to modify the land restart files generated from a previous model version (maint-1.0 in our case), so that we (@kaizhangpnl @xuezhengllnl ) can use them to perform some branch runs?

@kaizhangpnl, I can certainly generate new land initial conditions. Tell me the full ./create_case that you are using for various cases with maint-1.0 and I will generate the new files

There are some additional variables (compared to the old restart file) in the new restart file. Will they be used in the model simulation (or read but not used?)? Thanks!

The new variables in the land restart file that you listed in comment will not impact the satellite-phenology (SP) mode of the land model, which is the ELM mode you are using.

[kaizhangpnl]

@bishtgautam The create_newcase command is

./create_newcase --case $CASEROOT --project e3sm --mach compy --res ne30_oECv3 --compset F20TRC5-CMIP6

Maybe I didn't explain our need clearly. We wanted to use restart files generated from a previous AMIP run (using maint-1.0 version) to perform a branch run using current master. The land restart file is:

/compyfs/zhan524/restart/20180316.DECKv1b_A1.ne30_oEC.edison.1980-01-01/20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc

Based on your comment, the new variables in the land restart file (for current master) will not affect the simulation. Would it be possible to bypass them during the restart reading process?

[bishtgautam]

@kaizhangpnl, I will generate the new restart file today by remapping the following file:

/compyfs/zhan524/restart/20180316.DECKv1b_A1.ne30_oEC.edison.1980-01-01/20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc

Based on your comment, the new variables in the land restart file (for current master) will not affect the simulation. Would it be possible to bypass them during the restart reading process?

I don't believe it would be possible to bypass the reading of those variables. Even if those variables are read they won't impact the simulation. The *24H* and *240H* are 24 and 240-hr aveg of variables and these variables are not used in SP-mode. Similarly, *dynbal* are used in ELM when dynamic LULC is active, which is not the case in SP mode.

[golaz]

@bishtgautam: thanks for doing this. Can you document the steps needed?

@xuezhengllnl has also used the finidat_interp_source namelist option to read older files, but we are not sure exactly what it does.

[bishtgautam]

@bishtgautam: thanks for doing this. Can you document the steps needed?

Sure I will document the steps.

[bishtgautam]

I don't have experience with finidat_interp_source, but I can look into it later this week.

[bishtgautam]

@golaz, I have documented steps on https://gist.github.com/bishtgautam/bbc765571fbd9ce0b2a77356e01384a1. If you prefer I save this steps on a confluence page, let me under which space should I create such a page and will copy/paste the steps.

[kaizhangpnl]

@bishtgautam Thanks for your help. The new land restart file is read with no problem, but the start date is not correctly set (the restart date should be 1980-01-01).

(mylib) bash-4.2$ ncdump -v timemgr_rst_start_ymd, timemgr_rst_ref_ymd, timemgr_rst_curr_ymd 20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc

data:
timemgr_rst_start_ymd = 18500101 ;
timemgr_rst_ref_ymd = 18500101 ;
timemgr_rst_curr_ymd = 18500106 ;
}

As a result, elm (only) ran from 1850, but other model components didn't run:

-rw-r--r-- 1 zhan524 users 5.2K Sep 3 15:48 ocn.log.21727.190903-154652
-rw-r--r-- 1 zhan524 users 453K Sep 3 15:48 atm.log.21727.190903-154652
-rw-r--r-- 1 zhan524 users 13K Sep 3 15:48 ice.log.21727.190903-154652
-rw-r--r-- 1 zhan524 users 71K Sep 3 15:48 cpl.log.21727.190903-154652
-rw-r--r-- 1 zhan524 users 77M Sep 3 15:48 compy.E3SM_master_201908.F20TRC5-CMIP6_A01_rest.ne30_oECv3.clm2.h0.1850-01.nc
-rw-r--r-- 1 zhan524 users 57M Sep 3 15:48 compy.E3SM_master_201908.F20TRC5-CMIP6_A01_rest.ne30_oECv3.clm2.h0.1850-02.nc
-rw-r--r-- 1 zhan524 users 57M Sep 3 15:48 compy.E3SM_master_201908.F20TRC5-CMIP6_A01_rest.ne30_oECv3.clm2.h0.1850-03.nc
-rw-r--r-- 1 zhan524 users 57M Sep 3 15:48 compy.E3SM_master_201908.F20TRC5-CMIP6_A01_rest.ne30_oECv3.clm2.h0.1850-04.nc
-rw-r--r-- 1 zhan524 users 57M Sep 3 15:48 compy.E3SM_master_201908.F20TRC5-CMIP6_A01_rest.ne30_oECv3.clm2.h0.1850-05.nc

[bishtgautam]

When using finidat, I don't believe timemgr_rst_start_ymd is read. For example your 1980 initial condition file has timemgr_rst_start_ymd = 18700101

>ncdump -v timemgr_rst_start_ymd /compyfs/zhan524/restart/20180316.DECKv1b_A1.ne30_oEC.edison.1980-01-01/20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc

data:

 timemgr_rst_start_ymd = 18700101 ;
}

Things might be different for a branch run. Are you running a branch run? If yes, does your older simulation using 20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc produce ELM output that were *1980-01.nc? Do you need to do a xmlchange for RUN_STARTDATE?

[kaizhangpnl]

Yes, it is a branch run (An initial run should run well). I wasn't able to do a simulation using 20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc with the current master, so I can't say yes or no. For branch runs, we don't need to change the run start date. They are similar to restart runs and will read the restart date from the restart file.

The original land restart file (/compyfs/zhan524/restart/20180316.DECKv1b_A1.ne30_oEC.edison.1980-01-01/20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.nc) has:

timemgr_rst_start_ymd = 18700101 ;
timemgr_rst_ref_ymd = 18700101 ;
timemgr_rst_curr_ymd = 19800101 ;

I guess timemgr_rst_curr_ymd is the one that matters. Could you reset these values in the new restart file to those in the original restart file (listed above)?

[kaizhangpnl]

I temporarily modified these values in the new restart file using NCL. Now the model runs 7 days without problem.

[bishtgautam]

@kaizhangpnl, Let me try to create a new initial condition today that has timemgr_rst_curr_ymd = 19800101.

[kaizhangpnl]

@bishtgautam Thanks! It's great to include the fix in your workflow (shown in the gist), so we can follow the procedure next time.

[bishtgautam]

@kaizhangpnl, Please test out the following file on Compy:

/compyfs/inputdata/lnd/clm2/initdata_map/20180316.DECKv1b_A1.ne30_oEC.edison.clm2.r.1980-01-01-00000.8575c3f_c20190904.nc

If this file works, I will remove the older file from Compy inputdata dir and the E3SM public inputdata server.

Additionally, I have modified my gist notes.

[kaizhangpnl]

@bishtgautam Thanks! The new restart file worked well.