Esmvaltool: GitHub Action tests fail

[schlunma]

Since the merge of #1707 the GitHub Action tests fail:

 =========================== short test summary info ============================
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg1] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg0] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg3] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg2] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg5] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg4] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg7] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg6] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg9] - ...
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg8] - ...
FAILED tests/integration/test_recipe_filler.py::test_not_adding_datasets[cfg0]
FAILED tests/integration/test_recipe_filler.py::test_adding_datasets[cfg10]
FAILED tests/integration/test_recipe_filler.py::test_bad_var - FileNotFoundEr...
FAILED tests/integration/test_recipe_filler.py::test_not_adding_datasets[cfg1]
ERROR esmvaltool/cmorizers/obs/cmorize_obs_cds_uerra.py - ImportError: cannot...
ERROR esmvaltool/cmorizers/obs/cmorize_obs_cds_uerra.py - ImportError: cannot...
= 14 failed, 1437 passed, 1 skipped, 233 warnings, 2 errors in 369.66s (0:06:09) =
Error: Process completed with exit code 1.

However, the corresponding Circle CI tests pass...

[valeriupredoi]

yep, this is because the recipe filler test is coded to use the released version of esmvalcore but the dev test on GA uses the development version which is ahead of the released one (the config module is already included in the dev version); nothing we can do for now until the next release this will keep failing; fixing it will introduce fails in Circle so we want to stay away from that :grin:

[schlunma]

Ah, makes sense, cheers :beer:

[valeriupredoi]

I mean it's not exactly elegant, and that might mask issues related to the dev version of esmvalcore, so I could mark those tests as xfail until the next release - what do you think @schlunma ?

[schlunma]

Yeah, I think that's a good idea! Would it maybe also be possible to use skipif in combination with esmvalcore.__version__, so that the tests are only skipped for GitHub actions and not for CircleCI?

[valeriupredoi]

hmm skipif won't work since both are 2.2.0

[schlunma]

Ahh, yes, of course. If you mark them as xfail and they don't fail then tthis will result in an error, right? So marking them as xfail will fix the test for GA but break them for CircleCI...

[valeriupredoi]

correct! Schnapps all round, I don't have a solution :laughing:

[valeriupredoi]

I'll just skip them altogether and reactivate them when we have made the release