Esmvaltool: Allow different relative contributions of variable groups in ClimWIP

[lukasbrunner]

Description
A basic version of ClimWIP was recently implemented in ESMValTool via #1648. Here we would like to discuss the option of allowing different relative contributions of included metrics when calculating the weights (the default is an unweighted average over all metrics). This idea was introduced and used in Brunner et al 2020 and is needed for #1850.

The implementation of this functionality should be very straight forward. I basically requires an additional parameter in the recipe and an update of this lines to a weighted mean:
https://github.com/ESMValGroup/ESMValTool/blob/a7dfd0e8b4de15ac20b43c08d963786c3a8f4404/esmvaltool/diag_scripts/weighting/climwip.py#L356-L357

However I'm a bit unsure how to call the related parameter and how it should behave in order for it to be as clear as possible. Some thoughts:

• What is a good name of the parameter? I would like to avoid the term 'weight' in the parameter name to exclusively reserve this for model weights
• If the parameter is not set it should default to equal weighting (obviously) but if we want to set it for at least one metric there is a trade-off between flexibility and clearness.
  
  
  
  • should it be allowed to set this parameter only for one metric but not for the others? E.g., the default being 1, meaning when a metric is set to 2 it will contribute twice as much.
  
  
  • in case we demand it for all metrics: do we also demand that the sum of values is 1 (i.e., that the weights sum up to 1)? (this would me most clear I think but will also mean the largest effort when writing the recipe).
  
  

I think I'm tending towards the more flexible solution at the moment. Something like
performance_scaling_factor: 2
which just gives the metric in questing twice the weight (obliviously it could also be <1). But this is not a strong opinion and I'm happy for other suggestions or concerns with this approach. What do your think @ruthlorenz ?

[ruthlorenz]

do we really only need this for the performance metrics or potentially also for the independence ones (once we have them separated)? if for both I would avoid the "performance" in the name.
metric_scaling_factor?
I think default 1, 2 for twice the weight, 0.5 for half the weight etc. is intuitive.
I would not suggest to expect them to sum to 1 (3 metrics with equal importance would then need 0.3333333....), but probably to normalize them in the diagnostic?

[lukasbrunner]

I would keep it as flexible as possible and allow it for performance and independence. But that also means that we need the 'performance' in the name as we would want different weights for performance and independence, right?

About the summing up point: yes I agree I though of the same potential problem.

[ruthlorenz]

I guess this depends on where we want to set this parameter. I would have set it with the variable, but that might be a bad idea.
Something like that?

    scripts:
      climwip:
        script: weighting/climwip.py
        obs_data: native6
        sigma_performance: 0.43
        sigma_independence: 0.54
        performance_scaling_factor: [1, 2, 0.5] #order would be the same as metrics set above      
        independence_scaling_factor: [1,2]

and then yes, we should have the performance in the parameter name.

[lukasbrunner]

I guess this depends on where we want to set this parameter. I would have set it with the variable, but that might be a bad idea.
Something like that?

    scripts:
      climwip:
        script: weighting/climwip.py
        obs_data: native6
        sigma_performance: 0.43
        sigma_independence: 0.54
        performance_scaling_factor: [1, 2, 0.5] #order would be the same as metrics set above      
        independence_scaling_factor: [1,2]

and then yes, we should have the performance in the parameter name.

Setting it with the variable would be consisted with the parameters performance and independence that determine if a variable is use for either part of the weighting.

But I can also see the advantage of setting it directly under climwip, this seems to be a slightly better fit conceptually I guess?

In any case I think they need to be set both in the same place. So either something like this

pr_CLIM:
        <<: *common_settings
        short_name: pr
        independence: False
        performance: True  # optional, defaults to True
        performance_scaling_factor: 2

OR
what you suggested Ruth but then also moving the list of metrics to use to climwip (like in your original comment):

    scripts:
      climwip:
        script: weighting/climwip.py
        obs_data: native6
        sigma_performance: 0.43
        sigma_independence: 0.54
        performance_metrics: [tas_CLIM, pr_CLIM, psl_CLIM]
        performance_scaling_factor: [1, 2, 0.5] #order would be the same as metrics set above

@Peter9192 was there any specific reason for putting the independence and performance parameters under variable?

[lukasbrunner]

Additional comment: I noticed I started to use the term 'metric' for the combination of variable, time period, time aggregation, region, etc. (so what we call 'diagnostic' in our papers). I think this is a correct name for it conceptually, is there any collision with ESMValTool terminology? Otherwise we could consistently stick with it.

[Peter9192]

was there any specific reason for putting the independence and performance parameters under variable

I thought it made sense to have everything related to a single variable group together under the same header. That way, if you add or remove a variable group, you only change the variable group, and you don't have to worry that some other part of the recipe breaks.

But if you prefer passing them as parameters to the script, that's also fine. Then I'd suggest to make a dict instead of 2 lists:

scripts:
      climwip:
        script: weighting/climwip.py
        obs_data: native6
        sigma_performance: 0.43
        sigma_independence: 0.54
        performance_contributions: 
          tas_CLIM: 1 
          pr_CLIM: 2,
          psl_CLIM: 0.5
          pr_std: 0   # perhaps we could use this to exclude a certain variable group, instead of the performance=False flag?

[Peter9192]

Additional comment: I noticed I started to use the term 'metric' for the combination of variable, time period, time aggregation, region, etc. (so what we call 'diagnostic' in our papers). I think this is a correct name for it conceptually, is there any collision with ESMValTool terminology? Otherwise we could consistently stick with it.

I don't think there's a real clash, but I'm also not sure whether it is immediately obvious to other ESMValTool users/developers that you use the term 'metric' to refer to a 'variable group'. I first thought you wanted to support medians etc in addition to the mean.

At least this is way less confusing than 'diagnostic' ;-)

[lukasbrunner]

scripts:
      climwip:
        script: weighting/climwip.py
        obs_data: native6
        sigma_performance: 0.43
        sigma_independence: 0.54
        performance_contributions: 
          tas_CLIM: 1 
          pr_CLIM: 2,
          psl_CLIM: 0.5
          pr_std: 0   # perhaps we could use this to exclude a certain variable group, instead of the performance=False flag?

I like this! Maybe I would be in favour of setting it to 0 by default, this seems to be less prone to unexpected things happening?!

Should it be contributions_performance to be consistent with the sigma? (or performance_sigma maybe?)

[lukasbrunner]

I don't think there's a real clash, but I'm also not sure whether it is immediately obvious to other ESMValTool users/developers that you use the term 'metric' to refer to a 'variable group'. I first thought you wanted to support medians etc in addition to the mean.

At least this is way less confusing than 'diagnostic' ;-)

The meaning of variable groups seems less obvious to me but then again this is mainly used in the backend so it might be better to stay consistent with ESMValTool terminology (I think I did not realize this was 'official' terminology until just now). I will try to get myself to use that than. :)