Esmvaltool: fx files are not found for certain domains

[schlunma]

Analogous to #539, I have the following problem:

  diag_npb_main: &diag_main
    variables:
      nbp:
        ...
        mip: Lmon
        fx_files: [sftlf]
    additional_datasets:
      ...
    scripts:
      ...

Unfortunately, the tool cannot find the sftlf files because they are located in the atmos directory, not the land one. This seems to be a general problem. Somehow the tool needs to know the locations of the fx files, just searching for the given mip of the variable is not enough.

2018-08-16 12:32:12,737 UTC [30355] DEBUG   esmvaltool._data_finder:22 Looking for files matching sftlf_fx_IPSL-CM5A-LR_historical_r0i0p0.nc in .../DATA/IPSL/IPSL-CM5A-LR/historical/fx/land/fx/r0i0p0/v20110406/sftlf
2018-08-16 12:32:12,737 UTC [30355] INFO    esmvaltool._recipe:857 Using fx files for var nbp of dataset IPSL-CM5A-LR: {'sftlf': None}

[valeriupredoi]

hey @schlunma thx for raising this! I am a bit confusado here since sftlf is, to my knowledge, an atmos variable, and looking in the land bit I see sftgif as the one to grab: eg on BADC:

[valeriu@jasmin-sci1 ~]$ ls -la /badc/cmip5/data/cmip5/output1/IPSL/IPSL-CM5A-LR/historical/fx/land/fx/r0i0p0/latest/sftgif/
total 176
drwxr-x--- 2 badc open 4096 Oct 21  2011 .
drwxr-x--- 3 badc open 4096 Oct 21  2011 ..
lrwxrwxrwx 1 badc open   71 Oct 21  2011 sftgif_fx_IPSL-CM5A-LR_historical_r0i0p0.nc -> ../../files/sftgif_20110406/sftgif_fx_IPSL-CM5A-LR_historical_r0i0p0.nc

I will introduce support for sftgif right about now but if we need to crossover from one medium to another to retrieve fx files that's gonna be a bit of a headache since we need to indtroduce multiple paths in config-developer - sftgif is something @ruthlorenz needs as well

[valeriupredoi]

@schlunma I stand corrected - sftgif is only a land-ice mask, for some reason I was under the assumption that it covers land-sea as well; I see what you need and I am looking into fixing this right now :grin:

[schlunma]

Very nice, thank you! I double checked it as well after your comment :smiley:

[valeriupredoi]

ok guys, I have been thinking about how best to fix this - just before I start talking about the fix, @schlunma this should not be a major-major problem because the code will still apply a Natural earth mask rather than a native sftlf or sftof mask, so land/sea will still be masked.

In my opinion, the least destructive solution (which I tested) is to reassign locally the variable['mip'] when in need of sftlf/of files inside get_input_fx_dirname_template(variable, rootpath, drs) ie:

    # Set the drs
    _drs = drs.get(project, 'default')
    input_dir = cfg['fx_dir']
    for fx_ind in range(len(variable['fx_files'])):
        if isinstance(input_dir, six.string_types):
            if 'sftlf' in variable['fx_files'] or 'sftof' in variable['fx_files']:
                variable['mip'] = 'Amon'
                dir2 = replace_tags(input_dir, variable, i=fx_ind)
        elif _drs in input_dir:
            if 'sftlf' in variable['fx_files'] or 'sftof' in variable['fx_files']:
                variable['mip'] = 'Amon'
                dir2 = replace_tags(input_dir[_drs], variable, i=fx_ind)
        else:
            ....

This way we don't have to change any other function that is actually used for rather important data retrieval :grin: -- what do you think @bouweandela ?

[bouweandela]

Fine with me if it works, just make sure you copy variable, i.e. do a variable = dict(variable) before modifying it in the code above.

[schlunma]

If it works, fine for me as well 👍

[schlunma]

Fixed by #580.