Esmvaltool: MAGIC WP4 SMPI

[bjoernbroetz]

Tracking issue for the spi-metric. Developed in MAGIC_SPMI.

[hb326]

Documentation for the SMPI v2 added to the branch.

[bouweandela]

How is the progress on this, considering the MAGIC deadline? Can a pull request already be made?

[hb326]

@bouweandela: We are working on it. I have to check with @bjoernbroetz how far we are, and what is still needed. I will get back to you on Monday.

[hb326]

@bouweandela: I checked with Björn and have a better idea now what is still missing. Doesn't look like much, so I will test everything tomorrow when MISTRAL from DKRZ is available again, and will keep you posted.

[hb326]

@bouweandela: we had a version of the SMPI recipe running a while back, and had some trouble now to adjust the code to be compliant with the changes to the backend that happened since then. We got the SMPI to run again with a reducet set of input data and reduced time period. I am doing some more tests now to make sure all the needed variables are read correctly. I hope we can do a pull request early next week.
(FYI: @bjoernbroetz)

[hb326]

We tested the recipe with 9 out of 12 variables. These work fine. Other three are on irregular grids (e.g. sic) which are still not possible to be processed in v2. We are adding the provenance right now.

[bouweandela]

irregular grids (e.g. sic) which are still not possible to be processed in v2

Is there an open issue for this? Because there is some support for regridding irregular grids.

[hb326]

@bouweandela: I double-checked with @mattiarighi, and it seems I did not have the latest info about the irregular grids. They should work. So I will check why they did not in my diagnostics.

[hb326]

@bouweandela: I opened a pull request (#896). The three variables that cannot be analyzed yet have a problem in the pre-processing. @mattiarighi is planning on opening an issue about this (#897).

[bouweandela]

The recipe currently merged into the version_development branch does not work with the CP4CDS dataset. This is because:

• variable hfds is not part of the CP4CDS dataset
• not all combinations of variables and models are available in the CP4CDS dataset

For the MAGIC project it is required that diagnostics work with the CP4CDS dataset.

[hb326]

The recipe currently merged into the version_development branch does not work with the CP4CDS dataset. This is because:

* variable `hfds` is not part of the CP4CDS dataset

* not all combinations of variables and models are available in the CP4CDS dataset

For the MAGIC project it is required that diagnostics work with the CP4CDS dataset.

@bjoernbroetz and @axel-lauer, could you check that please? I will be on vacation next week, and I don't want that dragging on.
Basically, it does not matter which variables are used for the SMPI, it is just a matter that the same variables are used for all models. So taking hfds out is not a problem. However, it might be a little tricker to figure our which variables are all available for all the models.

[axel-lauer]

Where is the CP4CDS dataset to be used located?

[bouweandela]

On Jasmin in /group_workspaces/jasmin2/cp4cds1/data/c3s-cmip5/output1/ and the observational data is in /group_workspaces/jasmin2/cp4cds1/data/dkrz_pool

[axel-lauer]

A recipe_smpi_4cds.yml has been added and updated with the CP4CDS datasets available on Jasmin in branch magic_smpi_cp4cds: https://github.com/ESMValGroup/ESMValTool/blob/magic_smpi_cp4cds/esmvaltool/recipes/recipe_smpi_4cds.yml
The recipe is currently being tested on Jasmin, a pull request will be created once the new recipe was run successfully.

[bjoernbroetz]

Recipe tested on Jasmin. See: PR https://github.com/ESMValGroup/ESMValTool/pull/1069

[hb326]

PR #1069 was approved and the branch was merged. So can we close this issue here?

[bouweandela]

There are a few remaining issues with the SMPI recipe.

• The provenance for the collect script is not implemented correctly: ancestor files should be the files created by the tasks listed under ancestors in the recipe, not the preprocessor files used as input to those tasks.
• Some diagnostic script options either don't work or are not correctly documented. For example, I tried changing plot_type to cycle, but this leads to a crash. According to the documentation, time_avg can be monthlyclim, seasonalclim, annualclim, but the actual recipe uses yearly. Did you test this?
• In the documentation, it says that sic (ocean-ice, monthly mean, longitude latitude time) is not implemented yet, but actually it is implemented and seems to work fine.
• start_year and end_year is specified per dataset in the recipe, but should be the same for all datasets according to the documentation, so I think it would make much more sense to specify this per variable (and use yaml anchors to avoid duplication in the variable sections to make sure it is the same for all variables).
• there are a lot of commented out datasets in recipe_smpi.yml. I think those should either be uncommented or removed.
• the recipes start with
  ############################################################################### # recipe_smpi.xml
  I think it should be .yml instead and I believe ESMValTool should be mentioned somewhere there, look at the other recipes.

[hb326]

* there are a lot of commented out datasets in recipe_smpi.yml. I think those should either be uncommented or removed.

We have two recipies for the SMPI. The one for Jasmin is called "recipe_smpi_4cds.yml", similar to the recipe for the perfmetrics. In this recipe only the diagnostic for "hfds" is commented out since this variable is not available for the models on Jasmin. However, we think we should leave the commented out section in the recipe so that we can keep track of possible diagnostics to add to the recipe if the variable would become available at some point.

[hb326]

* the recipes start with
  ```
  ###############################################################################
  # recipe_smpi.xml
  ```


  I think it should be .yml instead and I believe ESMValTool should be mentioned somewhere there, look at the other recipes.

We changed the header of the recipe to be more consistent with the other recipies.

[bouweandela]

https://github.com/ESMValGroup/ESMValTool/issues/767#issuecomment-495117470 was about recipe_smpi.yml, not about recipe_smpi_4cds.yml, as it says in the comment. Why are these commented out?
https://github.com/ESMValGroup/ESMValTool/blob/9d68644bfa54e8040bc251d05e54e02ed5eb2c19/esmvaltool/recipes/recipe_smpi.yml#L29-L37

We changed the header of the recipe to be more consistent with the other recipies.

It looks like it was only changed for recipe_smpi_4cds.yml, not for recipe_smpi.yml?
https://github.com/ESMValGroup/ESMValTool/blob/9d68644bfa54e8040bc251d05e54e02ed5eb2c19/esmvaltool/recipes/recipe_smpi.yml#L1-L2

[hb326]

* The provenance for the collect script is not implemented correctly: ancestor files should be the files created by the tasks listed under `ancestors` in the recipe, not the preprocessor files used as input to those tasks.

We followed with our description the example of the perfmetrics. SMPI is basically a very specific case of the perfmetrics with one additional step added. Therefore we would like to keep the provenance for now the same as for the perfmetrics. However, we might want to rethink our strategy what we think should be included in the provenance at the workshop in June.

[hb326]

• Some diagnostic script options either don't work or are not correctly documented. For example, I tried changing plot_type to cycle, but this leads to a crash. According to the documentation, time_avg can be monthlyclim, seasonalclim, annualclim, but the actual recipe uses yearly. Did you test this?

• In the documentation, it says that sic (ocean-ice, monthly mean, longitude latitude time) is not implemented yet, but actually it is implemented and seems to work fine.

I updated the documentation for the recipe. I hope it does make more sense now. See pull request #1088

[hb326]

* start_year and end_year is specified per dataset in the recipe, but should be the same for all datasets according to the documentation, so I think it would make much more sense to specify this per variable (and use [yaml anchors](https://blog.daemonl.com/2016/02/yaml.html) to avoid duplication in the variable sections to make sure it is the same for all variables).

I added anchors for the start_year and end_year of the variables. See pull request #1088.

[hb326]

#767 (comment) was about recipe_smpi.yml, not about recipe_smpi_4cds.yml, as it says in the comment. Why are these commented out?

https://github.com/ESMValGroup/ESMValTool/blob/9d68644bfa54e8040bc251d05e54e02ed5eb2c19/esmvaltool/recipes/recipe_smpi.yml#L29-L37

We changed the header of the recipe to be more consistent with the other recipies.

It looks like it was only changed for recipe_smpi_4cds.yml, not for recipe_smpi.yml?

https://github.com/ESMValGroup/ESMValTool/blob/9d68644bfa54e8040bc251d05e54e02ed5eb2c19/esmvaltool/recipes/recipe_smpi.yml#L1-L2

The lines with the model specifications were commented out because we tested the recipe with the first two models after minor code adjustments. I fixed this now. I also adjusted the header of the recipe, and added the anchors for the start_year and end_year definition for the variables.

[bouweandela]

We followed with our description the example of the perfmetrics. SMPI is basically a very specific case of the perfmetrics with one additional step added. Therefore we would like to keep the provenance for now the same as for the perfmetrics.

Indeed provenance was also not correctly implemented for perfmetrics. If you look at the resulting provenance file or plot, you can see that all information is missing. Does the collect script use the preprocessor files directly in any way? Or only the files from the work_dir of the ancestors as defined in the recipe?

[mattiarighi]

--> only the files from the work_dir of the ancestors as defined in the recipe

[bouweandela]

Ok, in that case it really makes sense to record those in the provenance instead of the preprocessor files.

[bouweandela]

The lines with the model specifications were commented out because we tested the recipe with the first two models after minor code adjustments.

There is a command line option to select the number of models, see esmvaltool --help, so there is no need to change the recipe for this kind of test.

[mattiarighi]

Solved in #1090.