Esmvaltool: Tag replacement not working

[tobstac]

My recipes (landveg and runoff_et) fail because input files are not found. Debug output states
Skipping non-existent /work/bm0986/m214103/ESGF_Data/Tier[tier]/[dataset]
even though the directory exists - albeit with correct values for [tier] and [dataset]. Digging deeper I had the impression, that my tags are not replaced correctly in the _replace_tags function as there
re.findall(r'{([^}]*)}', path) with path being Tier[tier]/[dataset] results in an empty list and therefore none of the tags are replace with the values provided in my recipe.

I assume it is an user error because this could hardly go unnoticed, but I don't see anything wrong with my recipe. Any idea what goes wrong here?

Tests are done with a fresh installation using the release version provided in #1875
main_log_debug.txt
recipe_landcover.txt

[valeriupredoi]

let me run these on Jasmin, not 100% what's going on, but I suspect config issues. I am running a very hefty recipe at the mo and as soon as it's done I'll run those and see what's the hap :beer:

[valeriupredoi]

so a correct finding is producing this output (as in the case of my run on Jasmin):

2020-10-21 13:18:15,025 UTC [23915] DEBUG   Retrieving OBS configuration
2020-10-21 13:18:15,025 UTC [23915] DEBUG   Retrieving OBS configuration
2020-10-21 13:18:15,028 UTC [23915] DEBUG   Found /group_workspaces/jasmin4/esmeval/obsdata-v2/Tier2/ESACCI-LANDCOVER
2020-10-21 13:18:15,028 UTC [23915] DEBUG   Retrieving OBS configuration
2020-10-21 13:18:15,029 UTC [23915] DEBUG   Looking for files matching ['OBS_ESACCI-LANDCOVER_sat_L4-LCCS-Map-300m-P5Y-aggregated-0.500000Deg_Lmon_baresoilFrac[_.]*nc'] in ['/group_workspaces/jasmin4/esmeval/obsdata-v2/Tier2/ESACCI-LANDCOVER']

what does the /work/bm0986/m214103/ESGF_Data/ dir contain and what is the entry for rootpath: OBS: say in your configuration file?

[valeriupredoi]

also could you make sure that your config developer file has this entry for OBS:

OBS:
  cmor_strict: false
  input_dir:
    default: 'Tier{tier}/{dataset}'
    BSC: '{type}/{institute.lower}/{dataset.lower}/{freq_folder}/{short_name}{freq_base}'
    RCAST: '{dataset}'
  input_file:
    default: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc'
    BSC: '{short_name}_*.nc'
    RCAST: '{short_name}_{mip}_{type}_{dataset}_*.nc'
  output_file: '{project}_{dataset}_{type}_{version}_{mip}_{short_name}'
  cmor_type: 'CMIP5'

you might have changed that and forgot about it?

[tobstac]

I replaced my OBS entry with yours and this already solved much of the problem - know my obs and cmip5 directories are found. My config developer file uses [ ] for all tags but obviously now { } are used. I replaced these throughout the file, but now get a complaint about:
KeyError: "Dataset key _. must be specified
which either means I replace a [ too much or maybe I am missing some changes done since I used the esmvaltool the last time.

Is there a default developer config file somewhere I could compare against?

[valeriupredoi]

ahaa therein lies the trouble! You have multiple options to update the config-developer, probably the easiest one is to run esmvaltool config get_config_developer - that'll grab the default config-developer file and put it in a $HOME/.esmvaltool dir. Your default config-developer may be out of date if the esmvalcore package you are using is old (e.g. you installed it in developer mode and custom-installed esmvalcore rather than rely on the stock conda install via esmvaltool, installed as a dependency of esmvaltool, that is). So it'd be good to check that first; in any case, the stock config-developer is in esmvalcore :beer:

[tobstac]

Thanks a lot, that solved my problem!