Esmvaltool: Tool fails because 'institute' key is not found

[schlunma]

After the latest PR (#632), the tool fails for every recipe (I tested all examples) with the following error:

Traceback (most recent call last):
  File "~/ESMValTool/esmvaltool/_main.py", line 215, in run
    conf = main(args)
  File "~/ESMValTool/esmvaltool/_main.py", line 143, in main
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "~/ESMValTool/esmvaltool/_main.py", line 189, in process_recipe
    recipe = read_recipe_file(recipe_file, config_user)
  File "~/ESMValTool/esmvaltool/_recipe.py", line 56, in read_recipe_file
    raw_recipe, config_user, initialize_tasks, recipe_file=filename)
  File "~/ESMValTool/esmvaltool/_recipe.py", line 803, in __init__
    raw_recipe['diagnostics'], raw_recipe.get('datasets', []))
  File "~/ESMValTool/esmvaltool/_recipe.py", line 834, in _initialize_diagnostics
    raw_diagnostic.get('additional_datasets', []))
  File "~/ESMValTool/esmvaltool/_recipe.py", line 915, in _initialize_preprocessor_output
    self._initialize_variables(raw_variable, raw_datasets)
  File "~/ESMValTool/esmvaltool/_recipe.py", line 893, in _initialize_variables
    drs=self._cfg['drs'])
  File "~/ESMValTool/esmvaltool/_data_finder.py", line 260, in get_input_fx_filelist
    files = _find_input_files(var, rootpath, drs, fx_var)
  File "~/ESMValTool/esmvaltool/_data_finder.py", line 235, in _find_input_files
    input_dirs = _find_input_dirs(variable, rootpath, drs, fx_var)
  File "~/ESMValTool/esmvaltool/_data_finder.py", line 209, in _find_input_dirs
    for dirname_template in _replace_tags(path_template, variable, fx_var):
  File "~/ESMValTool/esmvaltool/_data_finder.py", line 116, in _replace_tags
    "your recipe entry".format(tag, variable))
KeyError: "Dataset key institute must be specified for {'preprocessor': 'preprocessor_1', 'field': 'T3M', 'fx_files': ['sftlf'], 'short_name': 'ta', 'diagnostic': 'ta_diagnostics', 'dataset': 'bcc-csm1-1', 'project': 'CMIP5', 'mip': 'Amon', 'exp': 'historical', 'ensemble': 'r1i1p1', 'start_year': 2000, 'end_year': 2002, 'cmor_table': 'CMIP5', 'filename': '/scratch/b/b309141/work_v2/recipe_ncl_20181017_085921/preproc/ta_diagnostics_preprocessor_1_ta/CMIP5_bcc-csm1-1_Amon_historical_r1i1p1_T3M_ta_2000-2002.nc', 'frequency': 'mon', 'modeling_realm': ['atmos']}, check your recipe entry

I used my old config-user.yml and a new one with a list as rootpaths; both fail. The old version (8e39c7ce) still works.

[mattiarighi]

Funny. Where did you try this?
I've successfully tested this PR with and without multiple rootpaths on both DKRZ and our cluster.

[mattiarighi]

@valeriupredoi reported a similar issue in #656, could this be related to it?

[schlunma]

Weird...I ran it on DKRZ.

[schlunma]

Maybe an issue of the config-user.yml? I used a fresh one from the repo and just adapted the paths:

###############################################################################
# User's configuration file for the ESMValTool
###############################################################################
---

# Diagnostics create plots? [true]/false
write_plots: true
# Diagnositcs write NetCDF files? [true]/false
write_netcdf: true
# Set the console log level debug, [info], warning, error
log_level: info
# verbosity is deprecated and will be removed in the future
# verbosity: 1
# Exit on warning? true/[false]
exit_on_warning: false
# Plot file format? [ps]/pdf/png/eps/epsi
output_file_type: pdf
# Destination directory
output_dir: .../work_v2
# Use netCDF compression true/[false]
compress_netcdf: false
# Save intermediary cubes in the preprocessor true/[false]
save_intermediary_cubes: false
# Remove the preproc dir if all fine
remove_preproc_dir: true
# Run at most this many tasks in parallel null/[1]/2/3/4/..
# Set to null to use the number of available CPUs.
# Make sure your system has enough memory for the specified number of tasks.
max_parallel_tasks: 8
# Path to custom config-developer file, to customise project configurations.
# See config-developer.yml for an example. Set to None to use the default
config_developer_file: null
# Get profiling information for diagnostics
# Only available for Python diagnostics
profile_diagnostic: false

# Rootpaths to the data from different projects (lists are also possible)
rootpath:
  CMIP5: .../
  OBS: [.../, .../]
  obs4mips: [.../, .../]
  Ozone_CMIP5_ACC_SPARC: .../
  default: .../

# Directory structure for input data: [default]/BADC/DKRZ/ETHZ/etc
# See config-developer.yml for definitions.
drs:
  CMIP5: DKRZ

[katjaweigel]

Maybe I'm totally wrong, but I had something similar few days ago and for me the reason was that I used a custom config-developer.yml, where the "CMIP5:" had the point "institute" but when downloaded the status from version2_development last week it needed "institutes".

[schlunma]

I use the config-developer.yml of the repo. Exchanging institutes by institute in the CMIP5 section didn't solve the problem...

[valeriupredoi]

@schlunma can you make sure that _recipe.py is the same as the one in version2_development (specifically the institute key gets a value at line 880). My problem stemmed from exactly those two lines of code not being placed in there after a git merge version2_development into my branch (which was very odd, because I have been doing branch merging since neolithic times, and am fairly positive I didnt cock anything up)

[mattiarighi]

I can confirm this is a problem with the institute key, since it works perfectly with the ETHZ drs which does not require such key.

[valeriupredoi]

@mattiarighi and @schlunma can you guys pls confirm that _recipe.py is up to v2_dev standards - see my comment above; if that's true, then it is a problem with institute indeed and suggest reviving my PR that I closed on Monday, if not, that means you are hitting the same problem I hit on Friday - git is messing up

[schlunma]

@valeriupredoi I'm currently on the version2_development branch:

12:33:38 ~/ESMValTool > git pull
Already up to date.
12:33:43 ~/ESMValTool > git status
On branch pubv2
Your branch is up to date with 'public/version2_development'.

nothing to commit, working tree clean
12:33:48 ~/ESMValTool > git diff public/version2_development -- esmvaltool/_recipe.py
12:33:50 ~/ESMValTool >

So i guess we need to revive your PR.

[mattiarighi]

branch: version2_development
commit: c39788
md5sum: 4ac58a9aa6166e3c4bd9c0f71a3d53e3 _recipe.py
_recipe.py:880 institute = get_institutes(variable)

[schlunma]

I even downloaded _recipe.py from the browser interface and made a diff with the file in the repo (to make sure git diff is not fooling me), both files are equal.