ESMValTool file finder is NOT case sensitive

[earnone]

In our database we currently have data with model name "EC-EARTH" and data with model name "EC-Earth" because they have different provenance. I noticed ESMValTool is not case sensitive when handling model names - or it drops case sensitivity if the correct name is not found? This implies that it picks EC-EARTH data if the EC-Earth one is missing even if the model name in the namelist is EC-Earth, therefore processing the wrong dataset. I feel that the file finder should strictly stick to the model name (or other tags) requested in the namelist to avoid possible errors. If the user typed the wrong model name, then no data should be returned.

[valeriupredoi]

Great, maybe we can implement some AI features as well, maybe port code from Amazon Alexa ask you extra questions about models :grinning: I will look into this very soon, currently in the middle of a large overhaul of software on JASMIN, they moved our work spaces and all sorts of mayhem was triggered due to this :grimacing:

[valeriupredoi]

@earnone - Enrico, man, did you check out and install the latest version of ESMValTool (specifically the one that uses fnmatch.filter rather than the old fin in data_finder) -- I ran a quick test by renaming a CMIP5 MPI-ESM-LR to MPI-esm-lr and again by renaming the original to some bogus XR:
ta_Amon_MPI-ESM-XR_historical_r1i1p1_200001-200512.nc
ta_Amon_MPI-esm-lr_historical_r1i1p1_200001-200512.nc

If the correct MPI-ESM-LR file is present, it pickis it up and ONLY that one, not the esm-lr as well, if that's missing (renamed XR), the code fails and will not pick up the lower case one. Also fnmatch.filter() is very exact.

Could you pls point me to the esmvaltool/data_finder.py version you using?

[earnone]

@valeriupredoi - Thanks for the prompt interaction :-) I see no fnmatch.filter in the data_finder.py. We all run branch version2_knmi_sprint here.

[mattiarighi]

@earnone was your branch created from REFACTORING_backend?
If yes, I would recommend to keep your branch synchronized with REFACTORING_backend.

Otherwise we run the risk that your branch diverges too much from the original one and it will be hard to examine the pull request once you are finished.

[valeriupredoi]

@earnone , @mattiarighi is correct, that KNMI branch is pretty old (lots of new stuff since February). Pls get REFACTORING_backend and branch out/track it from there. I assume all your changes are in diagnostics and namelist and namelists so hopefully you'll be able to adapt your envronment quickly and benefit from the good stuff we have put in the backend in the meantime since the sprint

[earnone]

@mattiarighi, @valeriupredoi ok, we'll move to REFACTORING_backend and keep that sync. No big problem, indeed we just have diagnostics and namelists. Thanks.

[valeriupredoi]

awesome, cheers! let us know if you get in trouble :grinning:

[mattiarighi]

Given that @valeriupredoi tested this in the latest version and found no bug, I would close this.

@earnone feel free to reopen the issue if you spot anything.