Esmvaltool: CMOR fix applied twice

[axel-lauer]

Fixes are erroneously applied twice if

• derived: true in the recipe
• variable is available with no derivation needed for a data set with an existing fix file

This is typically the case if a variable needs to be derived for models but is readily available from observations and a fix file for the observations exists, e.g. (from version2_lauer17rse):

  fig19_a:
    variables:
      xco2:
        derive: true
        mip: Amon
        field: T2Ms
        start_year: 2003
        end_year: 2003
        exp: [esmHistorical, esmrcp85]
        frequency: mon
        reference_dataset: ghgcci
        additional_datasets:
          - {dataset: MPI-ESM-LR, project: CMIP5, ensemble: r1i1p1}
          - {dataset: ghgcci, project: obs4mips, level: l3, version: v100, tier: 1}

@jvegasbsc applied some changes to my branch (see PR #755), but those seem to have been overwritten by merging the latest version2_development branch into my branch. @jvegasbsc what were those changes? Can we also implement them into version2_development? Help please!

[valeriupredoi]

@axel-lauer why are you saying they are applied erroneously twice? As in, the fix is applied once on the CMOR variable and then it's applied again on the derived variable?

[valeriupredoi]

ah I see now: xco2 from obs4mips gets fixed twice = once at the very beginning (together with all the other CMOR variables that enter the derivation) and one more time after derivation completed

[valeriupredoi]

and the explanation for those two checks is that the second and last one is done on the actual derived variable and the first one is done on the input obs4mips variable that is considered as secondary variable, see the output structure:

(esmvaltool_conda468) [valeriu@jasmin-sci2 esmvaltool_var_test]$ ls -la fig19/recipe_dask_20190408_111840/preproc/fig19_a/
total 0
drwxr-xr-x 1 valeriu users 0 Apr  8 12:29 .
drwxr-xr-x 1 valeriu users 0 Apr  8 12:18 ..
drwxr-xr-x 1 valeriu users 0 Apr  8 12:30 xco2
drwxr-xr-x 1 valeriu users 0 Apr  8 12:19 xco2_derive_input_co2
drwxr-xr-x 1 valeriu users 0 Apr  8 12:20 xco2_derive_input_hus
drwxr-xr-x 1 valeriu users 0 Apr  8 12:25 xco2_derive_input_ps
drwxr-xr-x 1 valeriu users 0 Apr  8 12:29 xco2_derive_input_xco2
drwxr-xr-x 1 valeriu users 0 Apr  8 12:21 xco2_derive_input_zg

so I think this is all good, man, there is the need to do both these checks

See in debug log:

2019-04-08 11:29:42,759 UTC [10115] DEBUG   esmvaltool.preprocessor:186 Running fix_data(dry_atmosphere_mole_fraction_of_carbon_dioxide / (1.0e-6) (time: 12; latitude: 36; longitude: 72)
...
2019-04-08 11:29:42,767 UTC [10115] DEBUG   esmvaltool.preprocessor:196 Running preprocessor step save...to /home/users/valeriu/esmvaltool_var_test/fig19/recipe_dask_20190408_111840/preproc/fig19_a/xco2_derive_input_xco2/obs4mips_ghgcci_l3_v100_xco2_2003-2003.nc

and

019-04-08 11:29:43,520 UTC [10115] DEBUG   esmvaltool.preprocessor:186 Running fix_data(dry_atmosphere_mole_fraction_of_carbon_dioxide / (1.0e-6) (time: 12; latitude: 36; longitude: 72)
...
2019-04-08 11:29:43,531 UTC [10115] DEBUG   esmvaltool.preprocessor:196 Running preprocessor step save
2019-04-08 11:29:43,532 UTC [10115] DEBUG   esmvaltool.preprocessor:186 Running save...
to /home/users/valeriu/esmvaltool_var_test/fig19/recipe_dask_20190408_111840/preproc/fig19_a/xco2/obs4mips_ghgcci_l3_v100_xco2_2003-2003.nc

In this case the xco2 didnt need derivation for the obs4mips dataset but the code is not that smart to realize that, it assumes xco2 was needed to derive xco2 and that's why it applied the fix twice; this can defo be repaired but I suspect we are opening a can of worms since all this is done serially and generally for all cases; I'll have a look at the code anyways

[axel-lauer]

Actually that's strange, because xco2 is not an input variable for the derivation of xco2. Input variables for the variable derivation are only co2, hus, zg, and ps. The problem with the fix being applied twice is that the unit conversion (multiplication with 1e6) is also done twice resulting in totally wrong values.

[valeriupredoi]

that is happening because the code first looks for xco2 files, for each dataset eg:

Looking for input files for variable xco2 of dataset MPI-ESM-LR
...
Looking for input files for variable xco2 of dataset ghgcci

and in the obs4mips case it actually finds them and uses them as input CMOR variable files. There has to be a hook in whereas the actual xco2 files should not be considered as eaw CMOR varable files and treated separately. One suggestion would be to put the obs4mips dataset in a separate preprocessor, that will allow for a separate treatment of its case; then fetch the file for diagnostic. I had the same issue with vars that were used both for derivation AND the same vars would be used for the diagnostic and fixes would be applied twice on them; @bouweandela suggested to put them in a separate preprocessor and that did the trick

[axel-lauer]

Not sure how putting things in two separate preprocessors would look like, so I couldn't try this trick. In any case, I think this would be a recipe for producing wrong results because this would be very difficult for users of the ESMValTool to understand (requires users to know about variable derivation, what exactly is provided by observations, fixes available for observations, type of fix, etc.).
Isn't there a possibility to prevent this from happening, for example by always excluding the target variable (here: xco2) from the list of variables needed for variable derivation? Having the target variable in the list of variables needed for derivation doesn't make any sense to me anyways.

[valeriupredoi]

Yes you are right, not sure what I was thinking when I was suggesting
putting the dataset in a different preproc :grin: We'll have to look
exactly where the input variable (which is also the output one in the
obs4mips case) is treated as derive member

Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom

On Tue, 9 Apr 2019, 07:26 Axel Lauer, [e-mail] wrote:

Not sure how putting things in two separate preprocessors would look like,
so I couldn't try this trick. In any case, I think this would be a recipe
for producing wrong results because this would be very difficult for users
of the ESMValTool to understand (requires users to know about variable
derivation, what exactly is provided by observations, fixes available for
observations, type of fix, etc.).
Isn't there a possibility to prevent this from happening, for example by
always excluding the target variable (here: xco2) from the list of
variables needed for variable derivation? Having the target variable in the
list of variables needed for derivation doesn't make any sense to me
anyways.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/issues/1013#issuecomment-481119420,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCo0w-fvqqJzL0xl93QSZ7wbD_k4JOks5vfDKwgaJpZM4ch0Wi
.

[valeriupredoi]

I will be on annual/Easter leave from tomorrow until Tuesday 23rd but will look into this asap after I get back; by the way @axel-lauer your poster at EGU was well liked, @bouweandela and myself we stood by it since @hb326 had to be by her own poster in the same session :beer:

[bouweandela]

It looks like I made a mistake with the default preprocessor settings in esmvaltool/_recipe.py, I'll have a look at this on Friday.

[axel-lauer]

I tested the fix with my branch version2_lauer17rse and it works! Thanks! I would be fine closing this issue then if there are no vetoes...