Esmvaltool: Custom variables: CMOR table

[axel-lauer]

Currently, the backend only reads custom CMOR tables if the custom variable is a derived variable. If not, the custom CMOR table will be ignored ultimately causing the preprocessor to stop. This makes it impossible to read non-derived variables that are not included in the CMIP5/6 CMOR tables. An example for such a variable would be "standard error of precipitation flux" (prStderr), which is a variable provided by obs4mips. A fix for this would be urgently needed. This problem is in my opinion also relevant to issue #718.

[bouweandela]

Maybe we can use obs4mips CMOR tables instead of the CMIP5 CMOR table for obs4mips data? That seems much more appropriate and probably solves this issue.
https://github.com/PCMDI/obs4MIPs-cmor-tables

[schlunma]

I think it's possible to put derive: True in the recipe even if the variable is not derived. If the variable you asked for matches the one in the input file, the derivation step is skipped:

https://github.com/ESMValGroup/ESMValTool/blob/b8e4e555a4f78646135ab5f7157afda44d3a2a66/esmvaltool/preprocessor/_derive/__init__.py#L65-L67

I think I used that before to read standard_names which were not in the CMIP5 CMOR tables, but I can't remember where... :confused:

[axel-lauer]

I think custom CMOR tables / extensions should always be possible as there will always be non-derived variables that are not in the official tables. This is particularly the case for observational products such as ESA-CCI to name just one example. For the case described above, using the obs4mips tables would not help as prStderr is not included even though it is in the obs4mips database.

[axel-lauer]

Setting "derive : True" did not help: "Don't know how to derive variable 'prStderr'"

[schlunma]

I'm pretty sure that worked before my PR with the new variable derivation. I think these lines are the problem:
https://github.com/ESMValGroup/ESMValTool/blob/b8e4e555a4f78646135ab5f7157afda44d3a2a66/esmvaltool/_recipe.py#L505-L516

Could you remove all of them and try again please?

[axel-lauer]

When deleting these lines, the preprocessor task is successfully created but eventually crashes with Iris refusing to load the input file: "Exception: Can not load cubes from ('...')". I would guess that the constraints when loading the cube are too strict or that the custom table is not evaluated properly or that the obs4mips file is not formatted properly or that I did something wrong. In case that the obs4mips file is not formatted properly, the constraints when loading the cube would have to be reduced as we claim that the ESMValTool can read obs4mips files without reprocessing and I would assume that a similar formatting has been applied to all observational uncertainties included in obs4mips ("Stderr").

[schlunma]

Ok, I will open a PR for this. The Cannot load cube exception... is probably caused by the constraints...

[valeriupredoi]

yup, that's because standard_name is not in the Cube

[mattiarighi]

Related to #606 and #552

[bouweandela]

I believe @jvegasbsc is working on this: https://github.com/ESMValGroup/ESMValTool/pull/638#issuecomment-438217781

We can probably just replace the standard_name constraint in the load function by no constraint, but keeping only cubes that have the correct short_name or standard_name or long_name, any of those will do. The correct standard_name etc would then need to be fixed in the fix_metadata function, or it can be directly applied in the load function.

[valeriupredoi]

also need to set the correct standard_name for derived variables (they currently have an incorrect_standard_name name), the problem is that needs to be CMOR compliant for 'iris' which is not, so... :confused:

[bouweandela]

also need to set the correct standard_name for derived variables (they currently have an incorrect_standard_name name), the problem is that needs to be CMOR compliant for 'iris' which is not, so... confused

Please make a separate issue if you have a problem with derived variables.

[bouweandela]

For the case described above, using the obs4mips tables would not help as prStderr is not included even though it is in the obs4mips database.

@axel-lauer Maybe you can use the method suggested in #718 to get them implemented.

I think custom CMOR tables / extensions should always be possible

The best solution for this would be to implement the option to specify custom additional table entries per cmor project. That would mean that the current implementation should be changed in such a way that the entries in esmvaltool/cmor/tables/custom are always loaded when a table from esmvaltool/cmor/tables/cmip5/Tables is loaded (and esmvaltool/cmor/tables/custom should be renamed to esmvaltool/cmor/tables/cmip5_custom for clarity, because it doesn't work with the cmip6 tables).

If at all feasible, getting variables in the official tables should always be the preferred way of getting a variable supported, because this makes the effort available to a wider community and also requires less tampering with libraries, e.g. we need to change the cf conventions used by iris to load the custom variables into cubes.

[axel-lauer]

We have a bunch of such variables. Getting them in the official table is not an option. Typically, these are variables from observational products, so moving them to cmip5_custom does not make sense to me. Cf conventions used by iris do not need to be changed if we lower the constraints when loading the variable in such cases as discussed in #606.

[valeriupredoi]

at the lowest level I suggest a generic load constrained on variable name, from my experience even the worst files still have a variable name:

    vn_constraint = iris.Constraint(
        cube_func=(lambda c: c.var_name == 'UM_0_fc8_vn405')
    )
vn_cube = iris.load(hadslp_file, constraints=vn_constraint)[0]

[axel-lauer]

@valeriupredoi that sounds good. That would give us quite some flexibility when processing observational data or when analyzing non-ipcc output from models, similar to v1.1. We always say that the ESMValTool can be used during model development comparing different runs from the same model, which would be extremely limited if we restrict ourselves to variables that are in the official cmor tables.

[BenMGeo]

I need to emphasize the needs for this functionality for our contract in c3s_511. We need to include custom variables by end of this year, ideally asap, like xch4.
I tried to include them and found the V1 custom CMOR tables (*.dat) in the backend, but just copying them there and using the derive option does not help. It looks like the internal table object/variable is not initialized at all, as it always stucks at table.frequency = value with table being None.

[valeriupredoi]

@bouweandela @jvegasbsc you fellas doing anything about this at the moment? If not I'll get something hacked today so Ben can get his work done. Ben, you need to populate esmvaltool/preprocessor/_derive/ dir with the needed new variable setup code, see esmvaltool/preprocessor/_derive/rtnt.py for an example of a non-CMOR derived variable

[BenMGeo]

Thank you, I'll have a look at these derive files.

[BenMGeo]

Hi Val,

I still get the error:

  File "/media/bmueller/Work/GIT/ESMValTool_V2_c3s511/esmvaltool/cmor/table.py", line 453, in _read_table_file
    table[value] = self._read_variable(value, table.frequency)
AttributeError: 'NoneType' object has no attribute 'frequency'

I think the TableInfo() is not correctly initialized as the V1 tables do not have a table_id (see lines 428-454).

https://github.com/ESMValGroup/ESMValTool/blob/88cc56d609eee9bd3cc1a564b2667848a56d1e5f/esmvaltool/cmor/table.py#L428-L454

[valeriupredoi]

hi @BenMGeo how does the entry for your variable look like - see eg esmvaltool/preprocessor/_derive/swcre.py - this one has a required variables dictionary - the variables used in the computation of swcre:

    _required_variables = {
        'vars': [{
            'short_name': 'rsut',
            'field': 'T2{frequency}s'
        }, {
            'short_name': 'rsutcs',
            'field': 'T2{frequency}s'
        }]
    }

• are these variables that you are using CMOR variables? If not, you will have to create entries for them as well in esmvaltool/cmor/tables/custom/ (see example CMOR_* tables that are there; also probably add frequency in the file as well after modelling_realm). What's the name of your branch? I may be talking out of my arse if I don't get to see your implementation per se

[BenMGeo]

Hi @valeriupredoi ,

I had a discussion with @schlunma and we found out after a long line of inserted loggers, that it was a dumb user error (I had amip: Amip instead of amip: Amon in the recipe, though there is no "no table available" error).

In the end it leads to the same issues that @axel-lauer reported: the standard_name not being available.

[valeriupredoi]

excellent! ok, in that case we can just use the load with options, depending what name there is in the file

[BenMGeo]

Do we already have this?

[valeriupredoi]

now we do: https://github.com/ESMValGroup/ESMValTool/pull/733 - can you pls checkout the branch and test? :beer:

[jvegasbsc]

Going back about the custom CMOR files: I will implement a fall back mechanism for searching in the custom if the project does not contain a variable. The only question here is if that mechanism should always be available or we just want to switch to a 'strict' mode that can make sense for CMIP5, CMIP6 data and such, as we shouldn't be able to find variables outside the data request