Esmvaltool: Installation and tests on OSX

[valeriupredoi]

OSX is giving us headaches, as seen from the results of Github Actions here - issue https://github.com/ESMValGroup/ESMValTool/pull/1806 Here's a few remarks:

• OSX and Python 3.8.5 are not friends to say the least, in fact for ESMValCore the conda installation doesnt work and for ESMValTool the environment setup fails too https://github.com/ESMValGroup/ESMValTool/pull/1806/checks?check_run_id=1081622594
• R packages fail to install see eg https://github.com/ESMValGroup/ESMValTool/pull/1806/checks?check_run_id=1081622566 - @bouweandela @jvegasbsc @nielsdrost any of you guys tried to build the R packages on a MacOS?
• ncurses is a pain in the rear and needs to be installed from conda-forge only - at install stage conda will supersede the conda-forge installation with the version from main that is not working, so currently we need to force it again from conda-forge; massive annoyance that is

I propose we offer limited support for OSX - installing from source works fine with Python 3.7.3, see https://github.com/ESMValGroup/ESMValTool/pull/1806/checks?check_run_id=1081622576 - conda installation can be repaired if we fix the ncurses issue see https://github.com/ESMValGroup/ESMValTool/pull/1806/checks?check_run_id=1081622693 but I have absolutely no idea how to fix the R packages installation issue - any takers? :beer:

UPDATE

• installation for esmvaltool-python and esmvaltool-ncl work fine for OSX, so things aren't that bad overall;
• making sure ncurses is installed from conda-forge allows actual installation of R-packages but the actual packages don't fully install with ps showing the first issues;
• Python 3.8 and OSX is still a major problem

[valeriupredoi]

Can one of the tech devs look into the OSX issues please (I would do it myself but I don't have access to a Mac, and I am really phobic to them anyway :grin: ) - we need to sort these issues out albeit I am quite pessimistic about the Python 3.8.5 env solvability :beer:

[nielsdrost]

Thanks @valeriupredoi for this! Unfortunately I do not have a mac either :-(

[valeriupredoi]

@nielsdrost not a worry! I got me access to a remote OSX via a VM: good news - installation from source via a miniconda3 environment works all fine for an osx=10.14.6 - I'll test more stuffs, bad news - I don't know where to get an osx=10.15 but hey, a bit older but golden :grin:

EDIT crap! I forgot to update to Python 3.8 from 3.7.3 - so the GA tests don't lie :grin:

[valeriupredoi]

OK - progress (sort of): on an osx=10.14.6 with fresh miniconda3 install and python=3.8.5 I get the same exact (cryptic) error as the GA machine reports:

(base) swmc9001:ESMValTool valeriu$ conda install -c conda-forge -c esmvalgroup esmvaltool
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \ 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                            

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

But not even creating en empty environment and trying to install python=3.8.5 doesn't work, not even doing anything esmvaltool-y:

(base) swmc9001:ESMValTool valeriu$ conda env create -n esmvaltool-conda
Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate esmvaltool-conda
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) swmc9001:ESMValTool valeriu$ conda activate esmvaltool-conda
(esmvaltool-conda) swmc9001:ESMValTool valeriu$ python -V
Python 3.7.3
(esmvaltool-conda) swmc9001:ESMValTool valeriu$ conda install python=3.8.5
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining requests:  77%|██████████████████████████████████████████████████████████████████▊                    | 178/232 [00:30<00:07,  7.11it/\ failed

so that means there is something inherently wrong with conda and python 3.8.5 on OSX! testing now with Python=3.8.3

EDIT same story for Python=3.8.3

[valeriupredoi]

...and starting from a Python=3.8.3 or 3.8.5 from (base) esmvaltool env creating works but reverts Python to 3.7.3 and trying to shove 3.8 in results in condatears:

(base) swmc9001:ESMValTool valeriu$ python -V
Python 3.8.3
(base) swmc9001:ESMValTool valeriu$ conda env create -n esmvaltool-py383 -f environment.yml 
Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate esmvaltool-py383
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) swmc9001:ESMValTool valeriu$ conda activate esmvaltool-py383
(esmvaltool-py383) swmc9001:ESMValTool valeriu$ python -V
Python 3.7.3
(esmvaltool-py383) swmc9001:ESMValTool valeriu$ conda install python=3.8.3
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining fsspec:  58%|███████████████████████████████████████████████████▍                                     | 134/232 [00:24<00:20,  4.86it/s]failed

[valeriupredoi]

OK I give up - I created a test environment with just python=3.8 as dependency and tried both the esmvaltool conda installation and the installation from source with prior conda env update -n testpy38 -f environment.yml and I am still hitting the same deps conflicts and conda not being able to solve the environment; and the damn thing is not clearly telling me which is which one creating conflicts! @bouweandela do you mind rebuilding the conda package on esmvalgroup - I reckon since it was built on Aug 3rd and Pyhton 3.8.5 was available for OSX only a couple weeks before, dependencies were not really up to date...

[valeriupredoi]

found the bugger! conda is like a woman that says ALL'S FINE but inreality stuff's anything but fine, it should really be more verbose -> it's libunwind that is conflicting with other packages for python=3.8 - interestingly enough it doesn't complain for >=3.7 even though libunwind does not have support for OSX (!!!) see here

[bouweandela]

Some updates to this topic:

• @bvreede Managed to install the tool on OSX and run the python example recipe with Python 3.8, however, the unit tests fail #2000
• @stefsmeets Reproduced the failing unit tests on Linux by setting the multiprocessing start method to 'spawn'
• @bouweandela Reported that he ran the Python example recipe on Linux with the multiprocessing start method set to 'spawn' just fine

Conclusion from this is that it looks like the tool seems to be working on Mac OSX, but there is a problem with the tests that @bvreede is looking into, with the help of @stefsmeets.

[valeriupredoi]

Some updates to this topic:

* @bvreede Managed to install the tool on OSX and run the python example recipe with Python 3.8, however, the unit tests fail #2000

* @stefsmeets Reproduced the failing unit tests on Linux by setting the multiprocessing start method to 'spawn'

* @bouweandela Reported that he ran the Python example recipe on Linux with the multiprocessing start method set to 'spawn' just fine

Conclusion from this is that it looks like the tool seems to be working on Mac OSX, but there is a problem with the tests that @bvreede is looking into, with the help of @stefsmeets.

you forgot:

• @bouweandela is still very suspicious about the obvious issue with multiprocessing start method not being set in OSX for Python 3.8+
  

[bouweandela]

@bouweandela is still very suspicious about the obvious issue with multiprocessing start method not being set in OSX for Python 3.8+

Indeed, I'm always very suspicious about stuff I do not understand, think it is set (to spawn), but you say it's not, so there is something we don't understand and I haven't seen a good explanation for so far. Could you explain it better? And why exactly does this cause our tests to fail?