Esmvaltool: Request to include reformatting for new obs data - Transcom basis function map

[rswamina]

Hello,

I would like to request including the Transcom basis function map as a new obs data set. Here is a brief description of the file that describes the Transcom regions:

The file 'smoothmap.fix.2.dat' contains a single real, binary
array dimensioned 360 x 180. The array contains the numbers 1
through 22, denoting each of the 22 basis functions in the
TransCom 3 experiment. This file was written on an SGI Origin

2000 hosting UNIX.

The Transcom experiment website is http://transcom.project.asu.edu/transcom03_protocol_basisMap.php

This is a data set used extensively in Land Surface metrics to categorize regions based on vegetation. Will it be possible to include CMOR-reformatizing steps for this data?
I wasn't sure if this request had to be part of an existing issue such as #232, hope this is ok.

Thanks!
-Ranjini

[mattiarighi]

Yes, this is possible.
Just create a cmorizing script in NCL or in python (see example in esmvaltool/utils/cmorizers/obs) and open a pull request once ready.

[rswamina]

Ok. Thanks!

[rjp23]

I came here looking to see if this was already possible i.e. I want to extract data in the preprocessor onto specified geographical (e.g. TransCom) regions.

Is the correct way to do this to have these different geographic regions be defined as datasets themselves? It seems like being able to subset datatsets into different pre-defined regions should be a bigger and consistent piece of functionality (e.g. you might want to select "South America Tropical" from TransCom, or "Japan" from NaturalEarth, etc).

[mattiarighi]

There is a preprocessor functionality to extract specific regions.
@ledm can give you hints on that.

[ledm]

There is a function area_slice, loaded as extract_region in the preproccesor from here , but this only extracts a rectangular shaped region (using start_longitude, end_longitude, start_latitude, end_latitude).

There is a extract_named_regions preprocessor which can be used when the iris cube has an region coordinate, see for instance the mfo files. This is unlikely to be what you want, as very few datasets have a region coordinate.

At the UKESM meeting Reading recently, @valeriupredoi talked about using Natural Earth datasets to extract specific regions, but I don't know how this would work. It's fairly crucial to have this functionality though, and I'm very interested in getting it working too! V, any ideas?

[ledm]

There's a make with shape file function in _mask.py.

That might be a good starting point, but I think it is only called to mask out land or sea, so far. Does anyone (@valeriupredoi, @mattiarighi , @schlunma ?) know a way to use it to extract specific named regions from the Natural Earth Shapefiles?

[valeriupredoi]

yesh, Natural Earth masks are applied when the desired fx files are not available - see https://github.com/ESMValGroup/ESMValTool/blob/version2_development/esmvaltool/preprocessor/_mask.py specifically _mask_with_shp() function. I would suggest putting such functionality in the diagnostic rather than preprocessor since these things can be very secific to the user's needs and best suited for diagnostic

[valeriupredoi]

note that you can still use that function to mask a (diagnostic) cube instance with a shapefile defined in the diagnostic, no need to copy the function in the diagnostic

[ledm]

Admittedly, I've never used the shapefiles. Naively, I would have thought that it would be extremely useful to use them to create masks in the preprocessor?

[valeriupredoi]

yes but they come in all sorts of shapes (pun intended) and parameters and we'd need a hefty set of case-by-case treatment functions; if you want to use the most generic land/sea files use the mask module; on top of it all we'd have to either download the files on the fly or store them internally permanently, either scenario is not nice. Lee if you can think of a few masking cases that are widely used then we can defo put them in the preprocessor, but I would stay away from a generalized shapefile-masking

[ledm]

Okay, I'll have a think about this.

@rjp23, you want to use Natural Earth files to extract specific regions. Is there a specific use case you have in mind, ie do you have a diagnostic which requires the Japan region or the "South America Tropical" TransCom region. It might not be possible to make a completely generic tool to extract any region from any file, but some specific common shape files might be possible. (No promises!)

[rjp23]

@ledm At the minute I'm purely using ESMValTool as a preprocessor to get all the data onto common grids and then I'm doing the analysis offline. Being able to do some of that analysis as part of a recipe would be really useful.

For example, I would want to extract all data over "Bangladesh" (as a named NaturalEarth Country). If I could do that either in the preprocessor or as part of the diagnostic it would enable much more "science" to be done with ESMValTool directly.

Likewise, I would like to subset model data onto different TransCom regions, calculate averages, etc in the preprocessor.

They're the two most obvious use cases but I assume there are other regions that people regularly select that aren't TransCom.

[valeriupredoi]

@mattiarighi see comment above - would this be something we'd support in the preprocessor? ie I can write a mask wrapper function and call it something like mask_region / region: SOME_COUNTRY for the preprocessor; it would most probably use _mask_with_shp() function but am not 100% sure just yet since I need to see the shp files per se; also, if we decide to have it in the preprocessor someone tell me which NE files to get

[mattiarighi]

I would see this better at the diagnostic level.

[bouweandela]

We also have a diagnostic script (currently in PR #700), that contains a recipe and Python diagnostic script to select regions based on shapefiles.

[mattiarighi]

Reopen if necessary.