Esmvaltool: Installing ESMValTool on a MacOS OSX-64 Mojave 10.14.5 Darwin kernel

[valeriupredoi]

The installation on a Mac is problematic at the moment, as pointed out in #1215 (bottom) since esmvalcore does not sit in an osx-64 or noarch repo of the esmvalgroup channel; while this will be sorted out, here are the instructions on how to fully install ESMValTool on a (modern) MacOS:

• grab the latest installer Anaconda3-2019.07-MacOSX-x86_64.sh
• install it by: bash Anaconda3-2019.07-MacOSX-x86_64.sh; tell conda to initialize the (base) channel (it will add nice stuff to your .bash_profile:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/valeriu/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/valeriu/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/valeriu/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/valeriu/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

• once in the base channel, grab ESMValCore git ball and create its sudo-environment with this file:

name: esmvaltool
channels:
  - conda-forge

dependencies:
  - esmpy
  - iris>=2.2
  - graphviz
  - libunwind  # Needed for Python3.7+
  - python-stratify

note that python and gcc are gonzo; create the env as usual: conda env create -n esmvaltool -f environment.yml; activate it; then install esmvalcore as usual by pip install -e '.[develop]'; check the installed package in the environment: conda list esmvalcore;

• now grab the ESMValTool git ball and use this environment file

name: esmvaltool
channels:
  - conda-forge

dependencies:
  # Python packages that cannot be installed from PyPI:
  - xarray>=0.12.0  # Can be installed from PyPI, but here to get a consistent set of depencies with iris.
  # Non-Python dependencies
  - cdo
  - imagemagick
  - nco

  # Multi language support:
  - tiledb=1.6.0  # Needed by the new ncl=6.6.2; conda keeps it at 1.2.2
  - ncl
  - r-base
  - r-curl  # Dependency of lintr, but fails to compile because it cannot find libcurl installed from conda.
  - r-udunits2  # Fails to compile because it cannot find udunits2 installed from conda.

Note how many packages have now disappeared from the list since they are already installed from the initial creation of the environment from esmvalcore. Now update the environment with this file (update is the magic word):

conda env update -n esmvaltool -f environment.yml

• install ESMValTool as per usual via pip install -e '.[develop]'; run the tests and the --installation tests etc

Voila. Not pretty, but it works.

I belive this should probably sit somewhere in the documentation, I leave it to you @bouweandela @mattiarighi @jvegasbsc @bjoernbroetz to decode where to put it, I've done the engineering :grin:

[valeriupredoi]

also @ledm you were asking at some point

[ledm]

I'm not a mac person, but we had some trouble with mac users at tutorials (@koir-su and @ytakano3). This was back in January, but we created an issue here: #827

Its good news that ESMValTool devs are actively working to supporting these users though. A lot of people use macs!

[valeriupredoi]

cool, pls give it a shot with this method, I closed the other issue since that was obsolete since we've changed quite a bit in the installation procedure and code structure in the meantime :beer:

[ValerioLembo]

Thank you @valeriupredoi.

I don't have the opportunity to test it now. But I will give a try the next week.

I also believe that it should appear somewhere on the official documentation for the sake of the (many) MacOS users.

[valeriupredoi]

dox is on its way, man :beer:

[valeriupredoi]

https://github.com/ESMValGroup/ESMValCore/pull/208 and https://github.com/ESMValGroup/ESMValTool/pull/1257

[bouweandela]

Getting back to this very good comment by @jvegasbsc https://github.com/ESMValGroup/ESMValTool/issues/827#issuecomment-498230770

The question is, can we test the MacOS installation automatically? If we can, I think we can make the extra effort to support it

If we provide installation files, we also need to test them, or they will end up broken and provide a bad user experience. CircleCI supports MacOS, but the costs are about 40 euro per month for open source. Also, supporting this will take up already limited developer time. Maybe it would be easiest to just recommend MacOS users to install from source.

[valeriupredoi]

Good point! well, this approach I tested is somewhat (or fully) from
source. We can test it via CircleCI w/o paying for the OSX support simply
by generically testing the two changed environments (for -core and -tool)
that we know they work on OSX

Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom

On Fri, 23 Aug 2019, 08:50 Bouwe Andela, [e-mail] wrote:

Getting back to this very good comment by @jvegasbsc
https://github.com/jvegasbsc #827 (comment)
https://github.com/ESMValGroup/ESMValTool/issues/827#issuecomment-498230770

The question is, can we test the MacOS installation automatically? If we
can, I think we can make the extra effort to support it

If we provide installation files, we also need to test them, or they will
end up broken and provide a bad user experience. CircleCI supports MacOS,
but the costs are about 40 euro per month for open source. Also, supporting
this will take up already limited developer time. Maybe it would be easiest
to just recommend MacOS users to install from source.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ESMValGroup/ESMValTool/issues/1253?email_source=notifications&email_token=AG5EFI3FHQPTOPPSXPK6X5TQF6JDDA5CNFSM4INCL2E2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD47NQRQ#issuecomment-524212294,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG5EFI2UHD4KLPAH3REBF33QF6JDDANCNFSM4INCL2EQ
.

[ValerioLembo]

@valeriupredoi I tested the approach on my MacOS 10.12.6. Just as a double check. It is working perfectly fine.

[valeriupredoi]

excellent, cheers for testing @ValerioLembo :beer:

[valeriupredoi]

Getting back to this very good comment by @jvegasbsc #827 (comment)

The question is, can we test the MacOS installation automatically? If we can, I think we can make the extra effort to support it

If we provide installation files, we also need to test them, or they will end up broken and provide a bad user experience. CircleCI supports MacOS, but the costs are about 40 euro per month for open source. Also, supporting this will take up already limited developer time. Maybe it would be easiest to just recommend MacOS users to install from source.

@bouweandela so the most we can do without paying the stupid OSX fee to CircleCI is to verify the build of the environment-osx emulated on linux-64 -> we can test this in the CircleCI free Linux container as a first pass assuming that if the package releases that conda grabs for Linux are in harmony then it should be fine on OSX as well. Given that the environment-osx is actually a subset of environment for both -core and -tool I don't see a reason why we should actually even implement the build test in the CircleCI test. What say you?

[bouweandela]

What say you?

I say that if you don't test it, it will probably end up broken and outdated.

[bouweandela]

Note that the latest conda packages (v2.0.0b1) are noarch, so can probably also be installed on OSX, but no guarantee whatsoever.

[valeriupredoi]

yes, installed successfully and changed the two PRs: https://github.com/ESMValGroup/ESMValCore/pull/208 and https://github.com/ESMValGroup/ESMValTool/pull/1257
a :beer: for you for noarch-ing them packages

[valeriupredoi]

What say you?

I say that if you don't test it, it will probably end up broken and outdated.

identical installation as to linux64 now, so it will not end up outdated and broken, unless there is something majorly conked-up but in that case the linux64 will prob go tits up too; anyway, warnings say that it is at the user's risk and if they find issues they should report them; we need those instructions because there is a s**tton of OSX users (if it was me I'd tell them to change to Ubuntu but meh)

[valeriupredoi]

closing this because it is deprecated, reopen new issue in #1807