Esmvaltool: Test before ESMValTool v2.1.0 release

[valeriupredoi]

Hello @ESMValGroup/esmvaltool-developmentteam - it's me the release manager asking you a big favour - can peeps grab their favorite recipe(s) and run it with the release branch and if so tick the box(es) below, please? We will be releasing ESMValTool v2.1.0 on next Monday (26 October) and it would be great to have tested all these recipes to be sure all works well. @mattiarighi and @hb326 it would also be fantastic if you guys had a bit of time to test some of the cmorizers, since I have no access to any of the raw data. Your efforts shall be forever appreciated and we promise we'll fix a robot to do this disrt work before the next release :beer: :beer:

• [x] recipe_albedolandcover.yml
• [ ] recipe_anav13jclim.yml
• [x] recipe_arctic_ocean.yml
• [x] recipe_autoassess_radiation_rms_Amon_all.yml
• [x] recipe_autoassess_radiation_rms_Amon_obs.yml
• [x] recipe_autoassess_radiation_rms_cfMon_all.yml
• [x] recipe_autoassess_stratosphere.yml
• [x] recipe_capacity_factor.yml
• [x] recipe_carvalhais14nat.yml
• [x] recipe_climwip.yml
• [x] recipe_clouds_bias.yml
• [x] recipe_clouds_ipcc.yml
• [ ] recipe_collins13ipcc.yml
• [x] recipe_combined_indices.yml
• [ ] recipe_consecdrydays.yml
• [ ] recipe_cox18nature.yml
• [x] recipe_cvdp.yml
• [ ] recipe_deangelis15nat.yml
• [ ] recipe_diurnal_temperature_index.yml
• [x] recipe_ecs_constraints.yml
• [x] recipe_ecs_scatter.yml
• [x] recipe_ecs.yml
• [ ] recipe_ensclus.yml
• [ ] recipe_extreme_events.yml
• [ ] recipe_extreme_index.yml
• [x] recipe_eyring06jgr.yml
• [ ] recipe_flato13ipcc.yml
• [ ] recipe_heatwaves_coldwaves.yml
• [x] recipe_hyint_extreme_events.yml
• [x] recipe_hyint.yml
• [ ] recipe_kcs.yml
• [x] recipe_landcover.yml
• [ ] recipe_lauer13jclim.yml
• [x] recipe_li17natcc.yml
• [x] recipe_martin18grl.yml
• [ ] recipe_miles_block.yml
• [ ] recipe_miles_eof.yml
• [ ] recipe_miles_regimes.yml
• [x] recipe_modes_of_variability.yml
• [x] recipe_multimodel_products.yml
• [x] recipe_ocean_amoc.yml
• [x] recipe_ocean_bgc.yml
• [x] recipe_ocean_example.yml
• [x] recipe_ocean_ice_extent.yml
• [x] recipe_ocean_Landschuetzer2016.yml
• [x] recipe_ocean_multimap.yml
• [x] recipe_ocean_quadmap.yml
• [x] recipe_ocean_scalar_fields.yml
• [ ] recipe_perfmetrics_CMIP5_4cds.yml
• [ ] recipe_perfmetrics_CMIP5.yml
• [ ] recipe_perfmetrics_land_CMIP5.yml
• [x] recipe_quantilebias.yml
• [x] recipe_rainfarm.yml
• [x] recipe_runoff_et.yml
• [ ] recipe_russell18jgr.yml
• [ ] recipe_seaice_drift.yml
• [x] recipe_seaice_feedback.yml
• [ ] recipe_seaice.yml
• [ ] recipe_shapeselect.yml
• [ ] recipe_smpi_4cds.yml
• [ ] recipe_smpi.yml
• [ ] recipe_snowalbedo.yml
• [x] recipe_spei.yml
• [x] recipe_tcr.yml
• [x] recipe_thermodyn_diagtool.yml
• [x] recipe_toymodel.yml
• [x] recipe_validation_CMIP6.yml
• [x] recipe_validation.yml
• [ ] recipe_wenzel14jgr.yml
• [ ] recipe_wenzel16jclim.yml
• [ ] recipe_wenzel16nat.yml
• [ ] recipe_williams09climdyn_CREM.yml
• [x] recipe_zmnam.yml

cmorizers:

• [ ] recipe_era5.yml

examples:

• [x] recipe_check_obs.yml
• [x] recipe_concatenate_exps.yml
• [x] recipe_correlation.yml
• [ ] recipe_extract_shape.yml
• [x] recipe_julia.yml
• [x] recipe_my_personal_diagnostic.yml
• [x] recipe_ncl.yml
• [ ] recipe_preprocessor_derive_test.yml
• [x] recipe_preprocessor_test.yml
• [x] recipe_python_object_oriented.yml
• [x] recipe_python.yml
• [x] recipe_r.yml
• [x] recipe_variable_groups.yml

hydrology:

• [ ] recipe_hype.yml
• [x] recipe_lisflood.yml
• [x] recipe_marrmot.yml
• [x] recipe_pcrglobwb.yml
• [x] recipe_wflow.yml

schlund20jgr:

• [ ] recipe_schlund20jgr_gpp_abs_rcp85.yml
• [ ] recipe_schlund20jgr_gpp_change_1pct.yml
• [ ] recipe_schlund20jgr_gpp_change_rcp85.yml

[valeriupredoi]

hey guys @ESMValGroup/esmvaltool-developmentteam we'd really appreciate it if you got your favorite recipe run and ticked the box - just a couple days til the release and if these things don't get tested before the day of the release we'll exclude your recipe that wasn't tested (just kidding...or maybe not :grin: ) Cheers muchly! :beer:

[Peter9192]

@jeromaerts would you have time to run some simple hydrology recipes with this branch?

[remi-kazeroni]

I successfully ran all the recipes in examples except the following ones:

• recipe_extract_shape.yml (auxiliary data missing at DKRZ: Elbe.shp)
• recipe_julia.yml (most likely due to my installation)
• recipe_check_obs.yml (preprocessor issue)
• recipe_preprocessor_derive_test.yml ((same?) preprocessor issue)

Could someone doublecheck for these?

[ruthlorenz]

recipe_extract_shape.yml does not work for me either.
recipe header says "The shapefile can be copied from
esmvaltool/diag_scripts/shapeselect/testdata/Elbe.shp and
placed in the auxiliary_data_dir defined in config-user.yml."

I did that, but then it crashes with "fiona.errors.DriverError: Unable to open /ESMVal/auxiliary_data/Elbe.shx or /ESMVal/auxiliary_data/Elbe.SHX. Set SHAPE_RESTORE_SHX config option to YES to restore or create it."

recipe_julia.yml works fine for me (ticked it above).

The other two I cannot check because of missing data.

[ruthlorenz]

I also run most of recipe_collins13ipcc.yml
I cannot check one diagnostic in it because of missing obs data (the second last, fig12-31ad).
Rest runs fine just takes forever (my bad so I should not complain ).

[bouweandela]

@ruthlorenz You need to copy all the Elbe.* files, not just the one with the .shp extension.

[valeriupredoi]

@ruthlorenz @remi-kazeroni cheers for the help guys! Ruth, as Bouwe says - it needs all the metadata files for the Elbe region, shapefiles come as a collection of files for each of the region they describe. @bouweandela you got time to check the other non-obvious fails, Jasmin is belly up today, great timing for it to go

[bouweandela]

recipe_check_obs.yml (preprocessor issue)
recipe_preprocessor_derive_test.yml ((same?) preprocessor issue)

@remi-kazeroni Could you attach the file run/main_log_debug.txt so we can see what's going wrong?

[remi-kazeroni]

recipe_check_obs.yml (preprocessor issue)
recipe_preprocessor_derive_test.yml ((same?) preprocessor issue)

@remi-kazeroni Could you attach the file run/main_log_debug.txt so we can see what's going wrong?

main_log_debug_recipe_check_obs.txt
main_log_debug_recipe_preprocessor_derive_test.txt

and the recipes I used (one or two missing datasets commented out)

recipe_check_obs.yml.txt
recipe_preprocessor_derive_test.yml.txt

[ruthlorenz]

My bad, copying all files helps but recipe still fails.

main_log_debug.txt
log.txt

[SarahAlidoost]

I am working on "cmorizers: recipe_era5.yml", after that I can test some of the hydrology recipes. @Peter9192 and @jeromaerts FYI.

[valeriupredoi]

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

[valeriupredoi]

@ruthlorenz cheers for your efforts - I have reproduced your issue with recipe_extract_shape.yml and will look into it :beer:

[valeriupredoi]

OK @bouweandela - as recipe admin - you have a bug, sir :grin: https://github.com/ESMValGroup/ESMValTool/issues/1878

[remi-kazeroni]

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

@valeriupredoi you are right, I didn't have the lastest master branch, sorry... I have retested all the example recipes and I still could not run the same two ones, namely: recipe_check_obs.yml and recipe_preprocessor_derive_test.yml. These stop at the same point as before, including the PIOMAS task for recipe_check_obs.yml. Could it be due to mssing data on DKRZ?

[valeriupredoi]

@remi-kazeroni I have run the PIOMAS task OK with esmvalcore installed both as a dependency package from ESMValTool (off conda-forge) and source-installed from the latest master and had no issues, can you please make sure you have the latest master installed and not some dev branch?

@valeriupredoi you are right, I didn't have the lastest master branch, sorry... I have retested all the example recipes and I still could not run the same two ones, namely: recipe_check_obs.yml and recipe_preprocessor_derive_test.yml. These stop at the same point as before, including the PIOMAS task for recipe_check_obs.yml. Could it be due to mssing data on DKRZ?

I too don't have the full data on Jasmin, but mines stop at Missing data, as it should be the case when there's no data. I did manage to run the PIOMAS though since there is data on Jasmin (albeit not correctly named in file names); I am bit confused how @mattiarighi ran them last time (last release) - Mattia, any chance you can try them yourself again, mate? Sorry, I know you busy :beer:

[katjaweigel]

I found an issue with recipe_deangelis15nat.yml: The recipe is running, but several figures are not displayed correctly (deangelisf3f4/deangelisf3f4/fig3a_.png, deangelisf3f4/deangelisf3f4/fig3b.png, and deangelisf3f4/deangelisf3f4/fig4.png). Looks like an issue with the x axis limits. The recipe is unchanged since the last test on 7 October. I'll try to find the reason tomorrow.

[valeriupredoi]

cheers @katjaweigel - it could be a different version of whatever package is used to plot the plots? :beer:

[bouweandela]

@remi-kazeroni The recipes stop because there is not a single dataset defined for the variable you're trying to process, because you commented the one dataset that was there out. The error message is not very informative unfortunately, I've created an issue to report it https://github.com/ESMValGroup/ESMValCore/issues/843.

[koldunovn]

recipe_arctic_ocean.yml runs, but with one caveat :) After environment update I get matplotlib 3.3.1, that has a bug, that affects the plotting (https://github.com/mwaskom/seaborn/issues/2194). This was fixed in matplotlib 3.3.2, so with this version everything runs smoothly. You might consider to put matplotlib!=3.3.1 in setup.py.

[koir-su]

recipe_carvalhais14nat.yml runs.

[katjaweigel]

I had a closer look at the figures and it is not the axis but the value which changed (factor 100.0) for the plots from recipe_deangelis15nat.yml. Has any unit changed from 1 to % ? (It's in a variable calculated from several other ones, including prw and radiance units but maybe also others. I'll check which of them could have changed.)

[bouweandela]

Has any unit changed from 1 to % ?

@katjaweigel I suspect this might be related to https://github.com/ESMValGroup/ESMValCore/pull/754

[axel-lauer]

I just successfully tested the full examples/recipe_check_obs.yml on Mistral (DKRZ).

[katjaweigel]

@bouweandela thanks, yes it looks like there is a change in the derived variable rsnstcsnorm (which is in %) it is already too high by the factor 100 in the preprocessed data. In _derive/rsnstcsnorm.py I calculated manually times 100.0, I guess this is now done automatically in addition? I'll remove the manual factor and try if it changes the plots.

[bouweandela]

I guess this is now done automatically in addition?

@schlunma Could you answer that question?

[valeriupredoi]

@schlunma is on hols, he ain't gonna reply until the week after, looking through the PR it looks like the % units are now updated automatically :+1:

[katjaweigel]

@schlunma and @bouweandela Adding rsnstcsnorm_cube.units = Unit('percent') in ESMValCore/esmvalcore/preprocessor/_derive/rsnstcsnorm.py solves the issue with the factor 100.0 (or leaving the factor 100.0 out, but I think the other one is more "backward compatible"). But this is in the Core - I guess it should nevertheless be fixed now instead of putting some work around into the diagnostic?

[valeriupredoi]

cheers @koldunovn for both running the test and for pointing to us the Matplotlib bug - I don't think there's a need to restrict the version since 3.3.2 has been out and about for more than a month now (see release table ) and any newer environment will pick up 3.3.2 no problemo :+1:

[valeriupredoi]

@katjaweigel the units should be set correctly in Core from PR ESMValGroup/ESMValCore#754 so technically no need to set them - is this a case that's missed by the said PR?

[katjaweigel]

@valeriupredoi They are set correctly, but to calculate percentage I always had the factor 100.0 in there (but I forgot to set the unit manually to percentage). So the old version was:
rsnstcsnorm_cube = (((rsdt_cube - rsutcs_cube) - (rsdscs_cube - rsuscs_cube)) / rsdt_cube) * 100.0
(all cubes despite the result with the same radiance unit). As long as it did not care about the unit this worked well for my further calculations. Now it tries to be clever and somehow decided that dividing cubes with the same unit through each other should result in percentage. This puts the unit right but destroyed the value, because it is not clever enough to see that the factor 100.0 was already there. So there are two possible solution:
1) remove the factor 100.0 and let it correct everything itself
2) set the unit and the factor 100.0 manually, so the new cube is correct from the beginning and does not need the automatic fix anymore.
(I tested it, both works). Because it could be possible that somebody has a strange combination of new Core and old Tool version I'd prefer the solution (2), because it works both with and without the new automatic fix.

[koldunovn]

cheers @koldunovn for both running the test and for pointing to us the Matplotlib bug - I don't think there's a need to restrict the version since 3.3.2 has been out and about for more than a month now (see release table ) and any newer environment will pick up 3.3.2 no problemo 👍

Well, I did an update with conda env update --name esmvaltool --file environment.yml and I got the 3.3.1, so maybe some dependency is more conservative on the matplotlib site. But of course in a few months the chances to hit this problem will be fewer :)

[valeriupredoi]

@koldunovn - good point! conda update works in mysterious ways, conda env create picked up 3.3.2 for me so that's why I'm not too concerned
@katjaweigel cheers! I would not change the current Core - but is the resulting cube unit from the derivation script wrong? If it's wrong then it needs to be fixed in there and not by the external fix :beer:

[katjaweigel]

The resulting value is now wrong (with the factor 100.0 because it is used twice). Therefore I'd like to add
rsnstcsnorm_cube.units = Unit('percent')
in ESMValCore/esmvalcore/preprocessor/_derive/rsnstcsnorm.py then the values for rsnstcsnorm_cube are correct again

[valeriupredoi]

good stuff @katjaweigel - could you pls open a PR for that in le Core :beer:

[katjaweigel]

ok

[valeriupredoi]

ok

Danke schoen! :beer:

[katjaweigel]

@valeriupredoi Should I use master or release_2.1 as base?

[valeriupredoi]

for Core, use master please, we've made the release and we should delete that branch :beer:

[SarahAlidoost]

I am working on "cmorizers: recipe_era5.yml", after that I can test some of the hydrology recipes. @Peter9192 and @jeromaerts FYI.

For these recipes from hydrology folder run was successful:

• recipe_lisflood.yml
• recipe_marrmot.yml
• recipe_pcrglobwb.yml
• recipe_wflow.yml

I didn't run recipe_hype.yml, because I am looking for the shapefile needed by the recipe.

For cmorizer: recipe_era5.yml, some files are still downloading. Also, I'm working on setting the correct default drs, see this issue.

[soufianekar]

successfully tested recipe_cvdp.yml

[Peter9192]

I wonder why the test of examples/recipe_python crashed in this bot test: https://github.com/ESMValGroup/ESMValTool/pull/1882

[bouweandela]

Successfully tested examples/recipe_julia.yml

[earnone]

We have an issue with the plotting part of recipe_extreme_events.yml. @jhardenberg is looking into that.

[valeriupredoi]

@Peter9192 - am running that myself; @earnone will run it too. BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions :+1:

[valeriupredoi]

OK examples/recipe_python.yml tested successfully :+1:

[Peter9192]

BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions

There is an issue about this in the bot repo.

I managed to reproduce the examples/recipe_python.yml errors on a fresh install from the release branch. Then, upgrading cartopy to 0.18.0 fixed the issue for me.

[bouweandela]

BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions

This information would be so useful for debugging, that we could also consider including it in the debug log: https://github.com/ESMValGroup/ESMValCore/issues/847

[Peter9192]

https://github.com/SciTools/cartopy/issues/1615 btw this was the cartopy issue that we had in the example recipe

[valeriupredoi]

OK so this seems to be a bit of an endemic (not pandemic :grin: ) issue - reported above by @koldunovn (Matplotlib in that case) - either a conda update or just a simple fresh install will not cut it, recreating the env from scratch will do (as in my case, I built my env a few days ago, so fairly new) - it looks like a few dependencies have now become very sensitive to versioning and we must rebuild the env

[valeriupredoi]

BTW it would be useful if the Bot could post the environment in which it works, most of the issues stem from dependencies with different versions

This information would be so useful for debugging, that we could also consider including it in the debug log: ESMValGroup/ESMValCore#847

good point! how about we just write a file current_environment.yml to disk anyway, so that people can easily use it for a conda create -f current_environment.yml -n blast-why-did-it-crash

[bouweandela]

how about we just write a file current_environment.yml to disk anyway

Good idea, but let's take that discussion to ESMValGroup/ESMValCore#847

[sloosvel]

Regarding PRIMAVERA recipes, recipe_seaice_feedback.yml runs fine but recipe_seaice_drift.yml crashes due to an error with pyproj CRS. I tried running it with an old environment and the recipe finishes fine, whereas with the new environment it crashes.

[valeriupredoi]

hey @ESMValGroup/esmvaltool-developmentteam thank you much and much for helping us by running a LOT of recipes! We have now released version 2.1.0 and appreciate your help. Am gonna close this issue, but can I please ask you the ones that have had issues with recipes to open issues so we fix those before the next release? It would be very useful if you could include a copy of the environment when you do that - a lot of the issues are most probably due to dependencies having changed versions - like @koldunovn reported the bug with Matplotlib 3.3.1 (BTW Nikolay, the new version picks up 3.3.2 when newly installed, I have just checked!). Cheers much again :beer:

[valeriupredoi]

one more thing: I have just removed the release_2.1 branch so all's clean now - master is your go-to as usual :+1: