E3sm: netCDF variable attributes change/broken after EAM restarts

[brhillman]

Something strange is happening with the writing of output netCDF files after an EAM restart. First, it seems like the long_name variable attribute in the netCDF files is being truncated to a single word. Second, a new mixing_ratio attribute seems to be added to the new output files after restart as well. To illustrate with a reproducing example, create an ERS test without building:

./create_test ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu --no-build

cd to the test directory, and then edit user_nl_eam to output frequently enough for us to catch the bug:

cat > user_nl_eam <<EOF
nhtfrq = 1    
mfilt = 1
EOF

Build and run:

./case.build && ./case.submit

and then inspect the outputs:

[bhillma@mappy ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames]$ ncdump -h run/ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames.eam.h0.0001-01-01-18000.nc.base | grep TREFHT
        float TREFHT(time, ncol) ;
                TREFHT:units = "K" ;
                TREFHT:long_name = "Reference height temperature" ;
                TREFHT:cell_methods = "time: mean" ;
[bhillma@mappy ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames]$ ncdump -h run/ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames.eam.h0.0001-01-01-18000.nc.rest | grep TREFHT
        float TREFHT(time, ncol) ;
                TREFHT:units = "K" ;
                TREFHT:mixing_ratio = "" ;
                TREFHT:long_name = "Reference" ;
                TREFHT:cell_methods = "time: mean" ;

Notice that "Reference height temperature" has been truncated to "Reference", and the new attribute mixing_ratio is added to the attributes of TREFHT. Closer inspection reveals that mixing_ratio has been added to all data variables!

[oksanaguba]

I see it in my runs.

[rljacob]

Is the truncation happening to attributes in other model's output?
Which hash of master shows this?

[brhillman]

Current master.

[rljacob]

@brhillman and @oksanaguba can you point me to a directory with output that has this problem? You could just move the history output you have to compy or anvil.

[oksanaguba]

/lcrc/group/acme/ac.onguba/acme_scratch/anvil/theta-gw-pr1-baseline-nov24/run

compare
ncdump -h theta-gw-pr1-baseline-nov24.eam.h0.0001-01.nc
and
ncdump -h theta-gw-pr1-baseline-nov24.eam.h0.0019-01.nc

[dqwu]

@brhillman Is this issue reproducible with scorpio classic (xmlchange PIO_VERSION=1)?

[brhillman]

@dqwu with PIO_VERSION=1, the "mixing_ratio" attribute is still erroneously appended, but the long_names are not truncated.

[bhillma@mappy ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames]$ ncdump -h run/ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames.eam.h0.0001-01-01-18000.nc.base | grep TREFHT
        float TREFHT(time, ncol) ;
                TREFHT:units = "K" ;
                TREFHT:long_name = "Reference height temperature" ;
                TREFHT:cell_methods = "time: mean" ;
[bhillma@mappy ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames]$ ncdump -h run/ERS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L.mappy_gnu.test-restart-longnames.eam.h0.0001-01-01-18000.nc.rest | grep TREFHT
        float TREFHT(time, ncol) ;
                TREFHT:units = "K" ;
                TREFHT:mixing_ratio = "> �" ;
                TREFHT:long_name = "Reference height temperature" ;
                TREFHT:cell_methods = "time: mean" ;

[rljacob]

Looking at the files pointed to by @oksanaguba, the truncation happens to long_name on every eam variable and its truncated to the first word, not to the first n characters. This does not happen to elm files which are fine.

[rljacob]

Also doesn't happen to cice history files (this is an Fcase). DOES happen the eam.h1 files. This case had one h1 variable "U" and the long name went from "Zonal wind" to "Zonal".

[rljacob]

Try master hash df999cca819 and PIO_VERSION=2. That was before the recent SCORPIO update.

[brhillman]

Looks like that hash is from Nov 26. I'm going to guess it's happening well before that, since we first noticed this on our SCREAM dyamond2 branch, which I do not think has seen an upstream merge since well before Nov 26. But, I'll run the test anyways.

[brhillman]

Confirmed, still present with that hash.

[brhillman]

Looks like this started with #3831, master hash a3b0ba5. If I go back to 0c69f38 I do not see the truncated long_names. I do, however, still see the "mixing_ratio" attribute being added to every data variable. So perhaps the long_name truncation is a SCORPIO issue, and the "mixing_ratio" attribute is unrelated.

[brhillman]

Pinging @jayeshkrishna who appears to be the author of that PR.

[jayeshkrishna]

The PR #3831 did make some changes related to writing out text in Scorpio.
I agree that the "mixing_ratio" attribute being added is unrelated to Scorpio updates.

[jayeshkrishna]

Hmmm, this issue looks like a memory corruption (or initialization) issue to me. Scorpio ensures that only the valid part of the string (long_name) is written out. The appearance of mixing_ratio in the output is also due to this issue.

Note that the same information is written out correctly in other (earlier sim time steps) files.

e.g. *eam.h0.0001-01-01-00000.nc has the correct values for long_name (and does not have mixing_ratio in the list of attributes for the variable) while *eam.h0.0001-01-01-14400.nc has the truncated value (and the mixing ratio). In the first case EAM does not write out the mixing ratio (cam_history.F90: tape(t)%hlist(f)%field%mixing_ratio) because the string is empty (Scorpio is not called to write out the variable) while in the second case EAM calls Scorpio to write out an invalid (the contents of the string are not valid) string. Also in the first case EAM writes out a valid/complete string (cam_history.F90: tape(t)%hlist(f)%field%long_name) for the long name of the variable and in the second case passes a truncated string to Scorpio (and its written out)

[jayeshkrishna]

In the case where the correct long_names of the variables are printed out, the mixing ratio is defined as wet, dry or empty (empty string). In the case where the output (variable attributes) is truncated/corrupted the value of mixing_ratio in a lot of calls is invalid.

[rljacob]

Its only after a restart so some interaction between the restart read and the history write must be causing the memory overwrite.

[jayeshkrishna]

This issue was due to a bug in Scorpio and I am currently testing the fix for the issue. The issue affects writing strings (char variables) that have spaces in the string. The model writes out the long_name attributes for all the variables in the restart file in an array of strings (In *eam.r.0001-01-01-10800.nc, variable "long_name"). The contents of this array was not correct and resulted in truncated strings while reading it back during model restart. Once the data read back was written to the history files the issue showed up as truncated long_name attributes for the variables.
The mixing_ratio issue was not related to Scorpio and requires a minor fix in E3SM.
I will issue a PR soon with fixes for both the issues