Esmvaltool: Using var for derivation and same var for diagnostic resets the file preprocessing on that var

[valeriupredoi]

@bouweandela @mattiarighi here's a relatively nasty issue: I am using rlut for deriving a var called rtnt and also using rlut in my diagnostic as a diag variable, together with rtnt and a bunch of other variables; my preprocessing is only a regrid on a 1x1 grid. rlut gets regridded fine when the work is done on it but when deriving rtnt, the files for rlut are replaced with native-grid rlut ones (which is correct since the derivation needs native grids) BUT they remain on the native grid and my diagnostic fails when analyzing rlut since it needs a regridded variable. This is probably averted if all the derived variables are on top of the variable list (and using a single thread) but this should not happen and the order in which the variables are in the recipe should not be a factor of success/failure. Any ideas where to fix this @bouweandela ? Cheerios, gents :grin:

[valeriupredoi]

guys, here is the recipe I am talkin about: https://github.com/ESMValGroup/ESMValTool/blob/version2_autoassess_v1_port/esmvaltool/recipes/recipe_autoassess_radiation_rms_Amon_all.yml

[bouweandela]

Hi V., you can probably work around this issue by creating a second preprocessor with a different name (same settings) and use that for rlut, just so the preprocessed rlut files are saved to a different directory and not overwritten by the files used for derivation. Can you test if that works?

[valeriupredoi]

cheers @bouweandela -- actually, by moving the derived vars at the top of the list it works fine as well (I am guessing if there is no multithreading on). I will put them all in a separate derived preprocessor instance, I see no reason why this shouldn't fix the issue properly -- but we need to specify this somewhere somehow so others are not gonna be running into the same issue, no?

[mattiarighi]

We could do it when addressing #505

[bouweandela]

Yes, that would work.

A more elegant solution, would be to slightly change the naming scheme for preprocessed files. Instead of using the diagnostic + preprocessor + short variable name for the directory, we could use a variable name as defined in the recipe. I believe this already works in the recipe, but the custom variable name is not used anywhere yet, e.g. you can write

diagnostics:
  diagnostic_name:
    variables:
      variable_name:
        short_name: ta
        preprocessor: preprocessor_1
        field: T3M

instead of

diagnostics:
  diagnostic_name:
    variables:
      ta:
        preprocessor: preprocessor_1
        field: T3M

if you want.

[mattiarighi]

That would be a bit less user-friendly, I think.

[valeriupredoi]

morning @bouweandela - I agree with @mattiarighi that this option may a bit confusing for the user, but it's good that it's there! It works great with setting up a dedicated preprocessor for the derived variables, so no bother -- let's just make sure these options (including the one above) get written in the documentation so peeps are aware of it, then I can close this

[bouweandela]

It's not required, just a possibility. It would also allow to for example have several groups of datasets for the same variable within one diagnostic (if you attach the datasets to the variable), something that @zklaus asked me about during the workshop.

[valeriupredoi]

good stuff, man @bouweandela

[bouweandela]

I think the current setup is confusing, where your data may be overwritten if you happen to use the wrong order (i.e. what this issue is about).

[mattiarighi]

@valeriupredoi is your question above answered?
Can we close this, given that we will adress it as part of #505 ?

[valeriupredoi]

yep yep, all good on my side

On Mon, Oct 22, 2018 at 2:01 PM Mattia Righi [e-mail]
wrote:

@valeriupredoi https://github.com/valeriupredoi is your question above
answered?
Can we close this, given that we will adress it as part of #505
https://github.com/ESMValGroup/ESMValTool/issues/505 ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/issues/519#issuecomment-431828802,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbpCoyT4y9ktDfOAP0IgPANiFFgeSiBNks5uncGxgaJpZM4VUsJA
.

--
Dr. Valeriu Predoi
Computational Scientist for UKESM Core Team
Department of Meteorology, University of Reading
Earley Gate, Office 1U08
READING, RG6 6BB
United Kingdom
Mobile number: 07847416092

"If one day you be questioning your ability to come up with professional
results, think of this: Noah's ark was built by farmers whereas the Titanic
was crafted by skilled engineers"

[bouweandela]

Note that there is no need for the workaround suggested here https://github.com/ESMValGroup/ESMValTool/issues/519#issuecomment-406232538 anymore, as this solution https://github.com/ESMValGroup/ESMValTool/issues/519#issuecomment-408777347 has been implemented in #649.