Esmvaltool: Running a batch of (many) recipes

[valeriupredoi]

Just had an issue raised by @alistairsellar and previously asked by @ledm about running more than one recipe - Alistair asked since there is the case when a model changes and there is the need to run a bunch of the recipes that use that model but with the new model version - it would be inconvenient to run each of them individually. Is there anything in the pipes for this or we should think about approaching such an issue? Also, @mattiarighi could you please add @alistairsellar to the github contributors list (I don't know if I can do it myself) - Alistair is the chief scientific coordinator of the UKESM project and one of the Autoassess originals

[mattiarighi]

I've just send him an invitation to join the development team.
Could you please e-mail me his full affiliation?
Thanks. 👍

[bouweandela]

Maybe you could write a small script that uses job array submission on Jasmin (see the section 'Job array submission' in the Jasmin documentation) and add it to the esmvaltool/utils/batch-jobs directory? The array would then be a list of recipes.

[jvegasbsc]

I can see two other options here:

• Consolidating all recipes into one (with the problem that we are now providing authorship and metadata by recipe and not metric)

• Use a workflow manager to manage the runs. In Jasmin you can use Cylc. This is just a more advanced form of Bouwe's suggestion

I prefer the second approach because will be easier to rerun just parts of what you need if something goes wrong and also will give you more parallelism

[bouweandela]

I think the first solution by @jvegasbsc would be an awesome feature to have in the long term because it prevents preprocessing the same data over and over again if it is used in multiple recipes, but it would require a considerable amount of work to implement that. For now I think just starting one esmvaltool + recipe instance multiple times is the way to go.

[jvegasbsc]

Just a disclaimer: I was thinking about consolidating them by hand, sort of a custom perfmetrics recipe merging diagnostics from multiple metrics.

[bouweandela]

Ok, I was thinking automatic ;-) Just run esmvaltool recipe_1.yml recipe_2.yml recipe_3.yml and it will automatically parse and make the smallest possible recipe and run it. Something to consider for the future.

[valeriupredoi]

@mattiarighi Alistair Sellar: Met Office https://www.metoffice.gov.uk/research/people/alistair-sellar

@bouweandela does the esmvaltool -c config-user.yml recipe1.yml recipe2.yml ... work now? Have tried it running like this a couple weeks ago in it took a dump on me :grimacing:

[bouweandela]

@valeriupredoi No it doesn't, it's a lot of work to implement.

[valeriupredoi]

running esmvaltool as a rose/cylc suite on LOTUS on Jasmin should work fine as @jvegasbsc suggested then

[alistairsellar]

Thanks for the invite Mattia - now accepted.

I would definitely recommend the suggestion of a workflow manager. One of the worst things about our auto-assess tool is that (because I wasn't aware of any off-the-shelf workflow managers 10 years ago) it does its own workflow management, running about 15 "areas" of metrics in parallel and polling them itself. A proper workflow manager is much better for users, particularly if any part of the system has an error and they need to debug / rerun.

Just to expand a little on the V's nice summary of the issue... The main purpose of an evaluation suite for our (UK) model development is to compare two model versions (a reference and a new test) over a large number of metrics. At the moment we have a top-level command which is given two run IDs and then calculates the metrics and displays them on a set of web pages. Ultimately our goal is to use ESMValTool in a similar way, to replace our existing tools like auto-assess.

[valeriupredoi]

@alistairsellar welcome to the cool kids' group :laughing:
Just as a note to you and the others - note that now, in ESMValTool, each autoassess area runs in its own recipe (the metrics are called and executed from autoassess_area_base.py via any of the 'recipe_autoassess_*` recipes) so you can run just one, or a subset, or all of your recipes ie assessment areas. These are independent and each produce metrics and a NAC plot

[valeriupredoi]

this is solved (at least for the case esmvaltool is installed on a machine with Rose/Cylc; Jasmin is a good example) by https://github.com/ESMValGroup/ESMValTool/pull/702

[valeriupredoi]

closing this since it was sorted out via rose/cylc in #702

[valeriupredoi]

note to self I should write documentation