Esmvaltool: Tutorial Issue: running recipe_python.yml
Describe the bug
A clear and concise description of what the bug is. If you are developing a new diagnostic script, please provide a link to the code/branch on GitHub that you are working in.
I am running the tutorial using the complete esmval package on an ETHZ server:
(conda create -n esmvaltool -c conda-forge -c esmvalgroup esmvaltool)
The installed version is:
ESMValCore: 2.1.0
ESMValTool: 2.1.0
When running the first recipe, I get this error:
Diagnostic script examples/diagnostic.py failed with return code 1. In the log.
In the log, there is an AttributeError: 'NoneType' object has no attribute 'lower'.
Please attach
- The recipe that you are trying to run, you can find a copy in the
rundirectory in the output directory - The
main_log_debug.txtfile, this can also be found in therundirectory in the output directory
Re: Attaching the recipe. The .yml file type is not supported as an attachment so I've attached it as a .txt.
recipe_python.txt
main_log_debug.txt
almerrifield
All 27 comments
Hi @almerrifield thanks for reporting this. In the debug log one of the last lines is: See the log in /home/meranna/esmvaltool_output/recipe_python_20201126_100020/run/map/script1/log.txt. Could you attach that log file as well?
Perhaps this is similar to the recent issue we've seen here: https://github.com/ESMValGroup/ESMValTool/issues/1894
There the problem was related to a bug in cartopy v0.17.0, and it was solved by pinning cartopy like so:
conda create -n esmvaltool esmvaltool 'cartopy>=0.18' -c conda-forge -c esmvalgroup -y
Peter9192
on 26 Nov 2020
I'll give it a try, thank you!
almerrifield
on 26 Nov 2020
Unfortunately, upon running:
conda create -n esmvaltool esmvaltool 'cartopy>=0.18' -c conda-forge -c esmvalgroup -y
There seems to be a conflict between esmvaltool-python and cartopy version >= 0.18.
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package cartopy conflicts for:
cartopy[version='>=0.18']
esmvaltool -> esmvaltool-python -> cartopy
Note that strict channel priority may have removed packages required for satisfiability.
almerrifield
on 26 Nov 2020
The log is not super clear, but it appears to be a very similar plotting issue indeed.
I just tried and the command above does work on my system without conflicts. Did you remove the existing esmvaltool environment you made earlier? The command is to create a new environment - that usually works better than trying to update an existing environment. You could also try to create another environment alongside it like so:
conda create -n other_env_name esmvaltool 'cartopy>=0.18' -c conda-forge -c esmvalgroup -y
Peter9192
on 26 Nov 2020
I did remove the esmvaltool environment using:
conda remove --name esmvaltool --all
Even with a new environment name, the same conflict occurs...
almerrifield
on 26 Nov 2020
I had the same issue with this recipe, relating to:
shading = kwargs.pop('shading', 'flat').lower()
AttributeError: 'NoneType' object has no attribute 'lower'
Following the suggestion above, and installing a new environment, it switches to a different issue:
2020-11-26 23:20:52,591 [255391] INFO diagnostic.py,89 Processing variable air_temperature
2020-11-26 23:20:52,591 [255391] INFO diagnostic.py,91 Processing dataset BCC-ESM1
Traceback (most recent call last):
File "/home/ncs001/miniconda3/envs/esmvaltool2/lib/python3.8/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 105, in <module>
main(config)
File "/home/ncs001/miniconda3/envs/esmvaltool2/lib/python3.8/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 93, in main
cube = compute_diagnostic(input_file)
File "/home/ncs001/miniconda3/envs/esmvaltool2/lib/python3.8/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 45, in compute_diagnostic
return cube.collapsed('time', iris.analysis.MEAN)
File "/home/ncs001/miniconda3/envs/esmvaltool2/lib/python3.8/site-packages/iris/cube.py", line 3239, in collapsed
raise iris.exceptions.CoordinateCollapseError(msg)
iris.exceptions.CoordinateCollapseError: Cannot collapse a dimension which does not describe any data.
I'm not sure of exactly why, but I notice that when creating the environment as above:
conda create -n other_env_name esmvaltool 'cartopy>=0.18' -c conda-forge -c esmvalgroup -y
this produces an install with:
esmvalcore 2.0.0 py_0 esmvalgroup
esmvaltool-python 2.0.0 py_0 esmvalgroup
whereas if I do not specify the cartopy version, I end up with version 2.1.0 of the above packages. If instead I do
conda create -n esmvaltool3 esmvaltool 'cartopy>=0.18' 'esmvaltool>=2.1.0' 'esmvaltool-python>=2.1.0' -c conda-forge -c esmvalgroup -y
this allows it to function for me at least.
swartn
on 27 Nov 2020
@swartn that's very helpful information, thanks!
@bouweandela @valeriupredoi any thoughts on these version conflicts?
Peter9192
on 27 Nov 2020
@swartn Thank you for the recommendation! We are currently triaging the situation in-house with the following updates:
-On our servers, we as users cannot update the conda environment, but have access to conda 4.9.2
The following works to create the environment, but not run the recipe:
conda create -n esmvaltool_full -c conda-forge -c esmvalgroup esmvaltool
The following cannot create the environment, due to package conflicts:
conda create -n esmvaltool_c18 esmvaltool 'cartopy>=0.18' -c conda-forge -c esmvalgroup -y
Package cartopy conflicts for:
cartopy[version='>=0.18']
esmvaltool -> esmvaltool-python -> cartopy
The following cannot create the environment, due to more package conflicts:
conda create -n esmvaltool3 esmvaltool 'cartopy>=0.18' 'esmvaltool>=2.1.0' 'esmvaltool-python>=2.1.0' -c conda-forge -c esmvalgroup -y
@ruthlorenz has had a look too. She has found an additional issue associated with the config file.
When uncommenting the rootpath and drs to input data, the tab structure needs to be maintained. In the example, it is correct. However, the tabs are missing in the solution for "set the correct rootpath". The recipe also will not run without the correct tab structure...
almerrifield
on 27 Nov 2020
there is no need to pass all those dependency constraints at env creation stage - cartopy=0.18.0 comes naturally in the environment for a while now, if you just call the env creation with conda env create -n esmvaltool -f environment.yml and then install in dev modepip install -e .[develop] all should work fine, note that you don't need to install the Python module of esmvaltool either, that comes stock, R and Julia need post-installation only :+1:
valeriupredoi
on 27 Nov 2020
another thing - when we made the 2.1 release we checked if the simple install from conda via conda install -c conda-forge -c esmvalgroup esmvaltool works (and this is checked continuously via the Github Actions too) so really, no need to create an env first, in your conda (base) env just run conda install -c conda-forge -c esmvalgroup esmvaltool and you should be sorted, including cartopy=0.18.0 :+1: Pls let me know if any issues, happy to help!
valeriupredoi
on 27 Nov 2020
@valeriupredoi Thank you! Unfortunately re: conda install -c conda-forge -c esmvalgroup esmvaltool
I fear that this is another place I don't have the permissions to write to the server environment...
almerrifield
on 27 Nov 2020
another thing - when we made the 2.1 release we checked if the simple install from conda via
conda install -c conda-forge -c esmvalgroup esmvaltoolworks (and this is checked continuously via the Github Actions too) so really, no need to create an env first, in your conda(base)env just runconda install -c conda-forge -c esmvalgroup esmvaltooland you should be sorted, includingcartopy=0.18.0馃憤 Pls let me know if any issues, happy to help!
I would not recommend that on our servers since the base environment is mutual between all users (and I do not think the general user can install anything into it). We would need to ask our IT person to do it, and if I remember correctly we have talked about it before and decided against it.
Regarding pip install -e .[develop]: I do not think thats the point of the tutorial, is it? Is the tutorial not about a beginner to be able to get started without developing anything? The problem I see is that if building the environment correctly is already a big issue there is the danger that this will prevent people from using the tool because they cannot even get started....
Is this mostly an issue we have at ETH or is this/could this be more common?
Thanks for helping though :-)
ruthlorenz
on 27 Nov 2020
ah my apologies, I didn't know this a case of not enough permissions to use the base environment; I would recommend having your own conda installation - in all cases this is the preferred solution because it becomes hairy when certain packages need to be written to the pkgs dir of conda. Having said that, what does conda list cartopy say in terms of version, and if that is 0.17.x you could just install it via conda directly in the environment with conda install -c conda-forge cartopy=0.18, and if that throws an issue related to environment insolvability that means a bunch of the deps in the environment are old so a recreation of the environment is needed. @ruthlorenz - good point, how about you guys talk to the ETHZ sys admins and ask them to install esmvaltool centrally, we've installed it on JASMIN and DKRZ and now a user needs to just module load esmvaltool and that's all, they can run it (no development obviously, but running the tool is enough for the tutorial) :beer:
valeriupredoi
on 27 Nov 2020
When uncommenting the rootpath and drs to input data, the tab structure needs to be maintained. In the example, it is correct. However, the tabs are missing in the solution for "set the correct rootpath". The recipe also will not run without the correct tab structure...
@almerrifield Thanks for reporting this, I have opened an issue about it in the tutorial repository: https://github.com/ESMValGroup/ESMValTool_Tutorial/issues/170
bouweandela
on 27 Nov 2020
I think the problem is that cartopy 0.17 is not compatible with matplotlib 3.3. So if installing cartopy 0.18 is not possible, you could also try installing matplotlib 3.2 instead?
Another possibility would be to try mamba instead of conda to install the package. It has a much better dependency solver.
bouweandela
on 27 Nov 2020
good point @bouweandela - that's why I suggested recreating the environment since for a while now (more than a month) cartopy=0.18.0 gets installed at env creation point, with a succession of matplotlibs - 3.2 a wee back ago, now 3.3, but cartopy is at the latest version notwithstanding matplotlib :+1:
valeriupredoi
on 27 Nov 2020
@valeriupredoi We do have cartopy 0.17.0 in our base environment so I tried to install cartopy 0.18.0 using
conda install -c conda-forge cartopy=0.18
after creating the esmvaltool environment using
conda create -n esmvaltool -c conda-forge -c esmvalgroup esmvaltool
then entering it with
conda activate esmvaltool
- side question, why
conda activaterather thansource activateto load esmvaltool?
After a lengthy report of conflicts, cartopy 0.18 did not install :(
If it helps, here is the list of packages in the esmvaltool conda environment I was able to create:
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
_py-xgboost-mutex 2.0 cpu_0 conda-forge
_r-mutex 1.0.1 anacondar_1 conda-forge
antlr-python-runtime 4.7.2 py38h32f6830_1002 conda-forge
atk 2.36.0 ha770c72_4 conda-forge
atk-1.0 2.36.0 h0d5b62e_4 conda-forge
attrs 20.3.0 pyhd3deb0d_0 conda-forge
binutils_impl_linux-64 2.35.1 h17ad2fc_0 conda-forge
binutils_linux-64 2.35 hc3fd857_29 conda-forge
blosc 1.20.1 he1b5a44_0 conda-forge
bokeh 2.2.3 py38h578d9bd_0 conda-forge
boost-cpp 1.72.0 h8e57a91_0 conda-forge
brotli 1.0.9 he1b5a44_3 conda-forge
brotlipy 0.7.0 py38h8df0ef7_1001 conda-forge
brunsli 0.1 he1b5a44_0 conda-forge
bwidget 1.9.14 ha770c72_0 conda-forge
bzip2 1.0.8 h516909a_3 conda-forge
c-ares 1.17.1 h36c2ea0_0 conda-forge
ca-certificates 2020.11.8 ha878542_0 conda-forge
cairo 1.16.0 hcf35c78_1003 conda-forge
cartopy 0.17.0 py38h9cf8511_1015 conda-forge
cdo 1.9.8 h41c8164_9 conda-forge
cdsapi 0.3.1 pyhd3deb0d_0 conda-forge
certifi 2020.11.8 py38h578d9bd_0 conda-forge
cf-units 2.1.4 py38hab2c0dc_2 conda-forge
cffi 1.14.4 py38ha312104_0 conda-forge
cfitsio 3.470 h2e3daa1_7 conda-forge
cftime 1.3.0 py38h0b5ebd8_0 conda-forge
chardet 3.0.4 py38h924ce5b_1008 conda-forge
charls 2.1.0 he1b5a44_2 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.1 pyhd8ed1ab_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
cmocean 2.0 py_3 conda-forge
colorspacious 1.1.2 pyh24bf2e0_0 conda-forge
cryptography 3.2.1 py38h7699a38_0 conda-forge
curl 7.71.1 he644dc0_8 conda-forge
cycler 0.10.0 py_2 conda-forge
cython 0.29.21 py38h348cfbe_1 conda-forge
cytoolz 0.11.0 py38h25fe258_1 conda-forge
dask 2.30.0 py_0 conda-forge
dask-core 2.30.0 py_0 conda-forge
decorator 4.4.2 py_0 conda-forge
distributed 2.30.1 py38h578d9bd_0 conda-forge
eccodes 2.17.0 h59f7be3_1 conda-forge
ecmwf-api-client 1.6.1 pyhd3deb0d_0 conda-forge
eofs 1.4.0 py_0 conda-forge
esmf 8.0.0 nompi_hb0fcdcb_6 conda-forge
esmpy 8.0.0 nompi_py38hf0e99fa_1 conda-forge
esmvalcore 2.1.0 py_0 esmvalgroup
esmvaltool 2.1.0 0 esmvalgroup
esmvaltool-julia 2.1.0 0 esmvalgroup
esmvaltool-ncl 2.1.0 0 esmvalgroup
esmvaltool-python 2.1.0 py_0 esmvalgroup
esmvaltool-r 2.1.0 0 esmvalgroup
expat 2.2.9 he1b5a44_2 conda-forge
fftw 3.3.8 nompi_hfc0cae8_1114 conda-forge
fiona 1.8.13 py38h033e0f6_1 conda-forge
fire 0.3.1 pyh9f0ad1d_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 2.001 hab24e00_0 conda-forge
font-ttf-source-code-pro 2.030 hab24e00_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.4 h7ca028e_0 conda-forge
freexl 1.0.5 h516909a_1002 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
fsspec 0.8.4 py_0 conda-forge
g2clib 1.6.0 hf3f1b0b_9 conda-forge
gcc_impl_linux-64 7.5.0 hd9e1a51_17 conda-forge
gcc_linux-64 7.5.0 he2a3fca_29 conda-forge
gdal 3.0.4 py38h172510d_6 conda-forge
gdk-pixbuf 2.42.0 h0536704_0 conda-forge
geos 3.8.1 he1b5a44_0 conda-forge
geotiff 1.5.1 h05acad5_10 conda-forge
gettext 0.19.8.1 hf34092f_1004 conda-forge
gfortran_impl_linux-64 7.5.0 hfca37b7_17 conda-forge
gfortran_linux-64 7.5.0 ha081f1e_29 conda-forge
ghostscript 9.53.3 h58526e2_2 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.66.3 h58526e2_0 conda-forge
gobject-introspection 1.66.1 py38h4eacb9c_3 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
graphviz 2.42.3 h6939c30_2 conda-forge
gsl 2.6 hf94e986_0 conda-forge
gtk2 2.24.32 h586f36d_1 conda-forge
gts 0.7.6 h17b2bb4_1 conda-forge
gxx_impl_linux-64 7.5.0 h7ea4de1_17 conda-forge
gxx_linux-64 7.5.0 h547f3ba_29 conda-forge
harfbuzz 2.4.0 h9f30f68_3 conda-forge
hdf4 4.2.13 hf30be14_1003 conda-forge
hdf5 1.10.5 nompi_h7c3c948_1111 conda-forge
hdfeos2 2.20 h64bfcee_1000 conda-forge
hdfeos5 5.1.16 h8b6279f_6 conda-forge
heapdict 1.0.1 py_0 conda-forge
html5lib 1.1 pyh9f0ad1d_0 conda-forge
icu 64.2 he1b5a44_1 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagecodecs 2020.5.30 py38h63741c2_4 conda-forge
imageio 2.9.0 py_0 conda-forge
imagemagick 7.0.10_28 pl526h201ca68_0 conda-forge
iris 2.4.0 py38_0 conda-forge
isodate 0.6.0 py_1 conda-forge
jasper 1.900.1 h07fcdf6_1006 conda-forge
jbig 2.1 h516909a_2002 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
joblib 0.17.0 py_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.13.1 hbfbb72e_1002 conda-forge
jxrlib 1.1 h516909a_2 conda-forge
kealib 1.4.13 hec59c27_0 conda-forge
keepalive 0.5 py38h32f6830_4 conda-forge
kernel-headers_linux-64 2.6.32 h77966d4_13 conda-forge
kiwisolver 1.3.1 py38h82cb98a_0 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
lcms2 2.11 hcbb858e_1 conda-forge
ld_impl_linux-64 2.35.1 hed1e6ac_0 conda-forge
lerc 2.2 he1b5a44_0 conda-forge
libaec 1.0.4 he1b5a44_1 conda-forge
libblas 3.9.0 3_openblas conda-forge
libcblas 3.9.0 3_openblas conda-forge
libcurl 7.71.1 hcdd3856_8 conda-forge
libdap4 3.20.6 h1d1bd15_1 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-devel_linux-64 7.5.0 h42c25f5_17 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgdal 3.0.4 h3dfc09a_6 conda-forge
libgfortran-ng 7.5.0 hae1eefd_17 conda-forge
libgfortran4 7.5.0 hae1eefd_17 conda-forge
libglib 2.66.3 hbe7bbb4_0 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 hd79254b_1012 conda-forge
liblapack 3.9.0 3_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libnetcdf 4.7.4 nompi_h9f9fd6a_101 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libopenblas 0.3.12 pthreads_hb3c22a3_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 12.3 h5513abc_2 conda-forge
librsvg 2.50.2 h3442318_1 conda-forge
libspatialite 4.3.0a h2482549_1038 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-devel_linux-64 7.5.0 h4084dd6_17 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
libtiff 4.1.0 h4f3a223_6 conda-forge
libtool 2.4.6 h58526e2_1007 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libwebp 1.1.0 h76fa15c_4 conda-forge
libwebp-base 1.1.0 h36c2ea0_3 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxgboost 1.2.0 he1b5a44_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
libxslt 1.1.33 h31b3aaa_0 conda-forge
libzopfli 1.0.3 he1b5a44_0 conda-forge
lime 0.2.0.1 pyh9f0ad1d_0 conda-forge
llvmlite 0.34.0 py38h4f45e52_2 conda-forge
locket 0.2.0 py_2 conda-forge
lxml 4.6.1 py38hbb43d70_0 conda-forge
lz4-c 1.9.2 he1b5a44_3 conda-forge
make 4.3 hd18ef5c_1 conda-forge
markupsafe 1.1.1 py38h8df0ef7_2 conda-forge
matplotlib 3.3.2 0 conda-forge
matplotlib-base 3.3.2 py38h5c7f4ab_1 conda-forge
msgpack-python 1.0.0 py38h82cb98a_2 conda-forge
munch 2.5.0 py_0 conda-forge
natsort 7.1.0 pyhd8ed1ab_0 conda-forge
nc-time-axis 1.2.0 py_1 conda-forge
ncl 6.6.2 hfe5c2fd_21 conda-forge
nco 4.9.2 nompi_h5be8af1_2 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
netcdf-fortran 4.5.2 nompi_h45d7149_104 conda-forge
netcdf4 1.5.3 nompi_py38heb6102f_103 conda-forge
networkx 2.5 py_0 conda-forge
numba 0.51.2 py38hc5bc63f_0 conda-forge
numpy 1.19.4 py38hf0fd68c_1 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.3.1 h981e76c_3 conda-forge
openssl 1.1.1h h516909a_0 conda-forge
ossuuid 1.6.2 hf484d3e_1000 conda-forge
owslib 0.20.0 py_0 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandas 1.1.4 py38h0ef3d22_0 conda-forge
pango 1.42.4 h7062337_4 conda-forge
partd 1.1.0 py_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pcre2 10.35 h279444b_1 conda-forge
perl 5.26.2 h36c2ea0_1008 conda-forge
pillow 8.0.1 py38h70fbd49_0 conda-forge
pip 20.2.4 py_0 conda-forge
pixman 0.38.0 h516909a_1003 conda-forge
pkg-config 0.29.2 h36c2ea0_1008 conda-forge
poppler 0.67.0 h14e79db_8 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 12.3 h8573dbc_2 conda-forge
proj 7.0.0 h966b41f_5 conda-forge
prov 2.0.0 pyhd3deb0d_0 conda-forge
psutil 5.7.3 py38h8df0ef7_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
py-xgboost 1.2.0 py38h32f6830_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pydot 1.4.1 py38h32f6830_1003 conda-forge
pyepsg 0.4.0 py_0 conda-forge
pykdtree 1.3.4 py38h0b5ebd8_0 conda-forge
pyke 1.1.1 pyhd8ed1ab_1004 conda-forge
pynio 1.5.5 py38h031d99c_12 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 2.6.1.post1 py38h7521cb9_0 conda-forge
pyshp 2.1.2 pyh9f0ad1d_0 conda-forge
pysocks 1.7.1 py38h924ce5b_2 conda-forge
python 3.8.6 h852b56e_0_cpython conda-forge
python-cdo 1.5.3 py_0 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-stratify 0.1.1 py38hab2c0dc_1003 conda-forge
python_abi 3.8 1_cp38 conda-forge
pytz 2020.4 pyhd8ed1ab_0 conda-forge
pywavelets 1.1.1 py38hab2c0dc_3 conda-forge
pyyaml 5.3.1 py38h8df0ef7_1 conda-forge
r-base 4.0.2 h95c6c4b_0 conda-forge
r-curl 4.3 r40hcdcec82_1 conda-forge
r-udunits2 0.13 r40hcdcec82_1004 conda-forge
rdflib 5.0.0 py38h32f6830_3 conda-forge
readline 8.0 he28a2e2_2 conda-forge
requests 2.25.0 pyhd3deb0d_0 conda-forge
scikit-image 0.17.2 py38h0ef3d22_4 conda-forge
scikit-learn 0.23.2 py38h5d63f67_3 conda-forge
scipy 1.5.3 py38h828c644_0 conda-forge
seaborn 0.11.0 ha770c72_1 conda-forge
seaborn-base 0.11.0 pyhd8ed1ab_1 conda-forge
seawater 3.3.4 py_1 conda-forge
sed 4.8 he412f7d_0 conda-forge
setuptools 49.6.0 py38h924ce5b_2 conda-forge
shapely 1.7.1 py38hc7361b7_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
sortedcontainers 2.3.0 pyhd8ed1ab_0 conda-forge
sparqlwrapper 1.8.5 py38h32f6830_1004 conda-forge
sqlite 3.33.0 h4cf870e_1 conda-forge
statsmodels 0.12.1 py38h0b5ebd8_1 conda-forge
sysroot_linux-64 2.12 h77966d4_13 conda-forge
tbb 2020.2 hc9558a2_0 conda-forge
tblib 1.6.0 py_0 conda-forge
tempest-remap 2.0.3 nompi_h60f458c_108 conda-forge
termcolor 1.1.0 py_2 conda-forge
threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge
tifffile 2020.11.26 pyhd8ed1ab_0 conda-forge
tiledb 1.7.7 h8efa9f0_3 conda-forge
tk 8.6.10 hed695b0_1 conda-forge
tktable 2.10 hb7b940f_3 conda-forge
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py38h25fe258_0 conda-forge
tqdm 4.53.0 pyhd8ed1ab_1 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
tzcode 2020a h516909a_0 conda-forge
udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge
urllib3 1.25.11 py_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.35.1 pyh9f0ad1d_0 conda-forge
xarray 0.16.1 py_0 conda-forge
xerces-c 3.2.2 h8412b87_1004 conda-forge
xesmf 0.4.0 pyhd8ed1ab_0 conda-forge
xgboost 1.2.0 py38h950e882_0 conda-forge
xlsxwriter 1.3.7 pyh9f0ad1d_0 conda-forge
xorg-imake 1.0.7 0 conda-forge
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxaw 1.0.13 h516909a_1002 conda-forge
xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxmu 1.1.3 h516909a_0 conda-forge
xorg-libxpm 3.5.13 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.1.5 h516909a_1003 conda-forge
xorg-makedepend 1.0.6 he1b5a44_1 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yamale 2.2.0 pyh9f0ad1d_0 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zfp 0.5.5 he1b5a44_4 conda-forge
zict 2.0.0 py_0 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.5 h6597ccf_2 conda-forge
@almerrifield cheers for the detailed diagnosis! I have replicated your chain of commands and indeed, cartopy=0.18 can not be installed in that environment. The solution is: in your environment you have created, please execute:
conda uninstall cartopy
[press y to all that guff on the screen, don't be scared it needs to remove so many packages]
conda install -c conda-forge cartopy=0.18
conda deactivate
conda activate esmvaltool
conda install -c conda-forge iris
pip install esmvalcore
pip install esmvaltool [provided pynio and esmpy are installed in the base env]
You should then have a solid working environment with cartopy=0.18 and matplotlib=3.3.3;
A few notes:
- you should really have admin access to your anaconda distribution, and be able to create/remove environments and operate from the base environment; you can download miniconda3 and use it without the need to depend on the sysadmin's sudo privileges; you can then share your working environment with others too, by exporting three paths to various
libandbindirs (I can show you how to do that) - anaconda is a poor dependency package manager and it's really hard to work with it if you don't have full control over it; even then it is very fussy and throw a lot of hiss when versions don't even conflict; this is why I recommend for users that just want to run the tool to
conda install -c conda-forge -c esmvalgroup esmvaltool- this installs ESMValTool just like any other package and the benefit is that conda doesn't have to consult its dependency table too much since at release point, by building esmvaltool conda package, the env is pretty much set in its metadata file; of course this means you must have write access to the base env - as I recommended to Ruth, a central installation is very useful, management can be done easily and users have hassle-free access to it just by loading a module; maybe you guys should ask at ETHZ, am sure they'd be happy to do it for you (they have setup a conda tool so it looks like they're nice :grin: )
- about why
conda activateand notsource activate- they're pretty much the same in terms of functionality, the newerconda activateis meant to be faster, have a look at this stackoverflow post
Cheers :beer:
valeriupredoi
on 28 Nov 2020
@valeriupredoi Thank you for following up, I was very much looking forward to completing the cheers 馃嵒. All went smoothly until the pip installs:
pip install esmvalcore
ERROR: Failed building wheel for stratify
ERROR: Could not build wheels for stratify which use PEP 517 and cannot be installed directly
and as anticipated:
pip install esmvaltool
ERROR: Could not find a version that satisfies the requirement pynio (from esmvaltool) (from versions: none)
ERROR: No matching distribution found for pynio (from esmvaltool)
I think you are right that central installation may be necessary at this point.... Thanks to all again!
almerrifield
on 30 Nov 2020
ERROR: No matching distribution found for pynio (from esmvaltool)
yeah I could see that coming, my base env had it and conda finally had the decency to look in the base env prefix path too
@bouweandela and myself we'll release a bugfix version of Core with pinning cartopy to 0.18 (we'll try do that tomorrow, Bouwe had the idea to do it and I think it's welcome for people struggling to install in standard mode like yourself, but I do encourage you to ask the ETHZ sys admins to look into a maintenable central installation :+1: )
valeriupredoi
on 30 Nov 2020
Interesting, for the old version (2.1.0) conda wants to install cartopy 0.17 and matplotlib 3.3; mamba on the other hand cartopy 0.18 and matplotlib 3.3.
conda create --name esmvaltool_conda -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0
mamba create --name esmvaltool_mamba -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0
Sometimes the incompatibilities are difficult to know and can be very indirect (e.g. both cdo and cartopy pin proj). But the conflict conda shows do not really make sense and I also made this experience before... So thanks for pointing out mamba! I did not know this tool.
The new version of esmvaltool (2.1.1) works with both mamba and conda, so @almerrifield you can now install it with:
conda create --name esmvaltool -c esmvalgroup -c conda-forge esmvaltool
Feel free to ask or tag me directly when you have problems with setting up an environment. We can also discuss setting up a global esmval environment.
@valeriupredoi please don't recommend installing esmvaltool in the base environment. The base environment should basically only contain conda and its dependencies. See also in the conda docs:
You don't want to put programs into your base environment, though.
That's what environments are for & when you do conda create --name ... you start with a clean slate, so it's even better than trying to install something into base. Also a shared conda installation is no worse than your self-installed one. You get your personal package cache (pkgs folder) and envs directories. The only thing you don't have control over is the conda version.
mathause
on 2 Dec 2020
Oh I just found out - mamba does not enforce --strict-channel-priority so when doing
mamba create --name esmvaltool_mamba -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0
mamba wants to install cartopy from main and not from conda-forge so I'd be very careful for now using mamba because channel mixing can lead to even more difficult to debug problems.
mathause
on 2 Dec 2020
Sorry, couldn't leave it: your actual problem is pynio. pynio is no longer maintained (see conda-forge/pynio-feedstock#90). It conflicts with the newest version of netCDF, but also with cartopy 0.18. So the following fails:
mamba create --name test_pynio --override-channels --strict-channel-priority -c conda-forge pynio python cartopy=0.18
I see only one mention of pynio (and no import Nio in the code; also in the private and AR6 repo). So I'd highly recommend to remove this dependency.
mathause
on 2 Dec 2020
we have just released 2.1.1 today and I wanted to test carefully its full functionality before we posted here that's ready for use and it'd solve this issue, cheers @mathause for trying it out and posting your results!
@valeriupredoi please don't recommend installing esmvaltool in the base environment. The base environment should basically only contain conda and its dependencies.
yes and no: conda is a package manager first and a virt environment manager second (and it's doing a relatively poor job as a package manager) so there is no problem to have an installation in base if you are 1. a new user that is not familiar with virtual environments and package managers and 2. you are told how to create and update environments in the future and start learning how to update individual packages. At the end of the day the base env is another env. This is why, in such cases and for a lot of users that need only to use the tool, a central installation is preferred, where one can module load a certain module. Also note that base is barebones when you use miniconda so it is a clean slate more or less.
Also a shared conda installation is no worse than your self-installed one. You get your personal package cache (pkgs folder) and envs directories. The only thing you don't have control over is the conda version.
Careful here - if all you share are paths to executables and some lib dirs, it's perfectly fine to do it (this is how a module is constructed anyway). No other permissions, no possibility to install new packages. If you start from a 100% working environment you will be sure your users will have it working as well (provided they don't mix paths, that'd be havoc :grin: )
I see only one mention of pynio
Yes, well spotted, I've already told @bouweandela today, after some env solving headaches the previous days, we should look into removing that! :beer:
Cheers for the comments @mathause - good points overall, and good detective work! Hopefully @almerrifield will be able to get the installation working! @mathause if you have a plan for a central install at ETHZ that'd be great, I can share the module I built at JASMIN for that central install with you :+1:
valeriupredoi
on 2 Dec 2020
oh and apparently mamba beats conda boss level, didn't know about it myself -> @bouweandela gets a :beer:
valeriupredoi
on 2 Dec 2020
@bouweandela @mathause @valeriupredoi Success!! 馃嵒馃嵒馃嵒
conda create --name esmvaltool -c esmvalgroup -c conda-forge esmvaltool
threw an r-base SafetyError and a series of conda-forge/linux-64 ClobberErrors, but ultimately worked and the recipe ran!
Thank you all for continuing to follow up!
almerrifield
on 3 Dec 2020
excellent, cheers @almerrifield for the heads up and sticking with it! I created an issue for pynio removal https://github.com/ESMValGroup/ESMValTool/issues/1935 :+1:
valeriupredoi
on 3 Dec 2020
Related issues
bouweandela
路
4Comments
valeriupredoi
路
5Comments
valeriupredoi
路
4Comments
chris-to-pher
路
3Comments
valeriupredoi
路
4Comments
Most helpful comment
@bouweandela @mathause @valeriupredoi Success!! 馃嵒馃嵒馃嵒
conda create --name esmvaltool -c esmvalgroup -c conda-forge esmvaltoolthrew an r-base SafetyError and a series of conda-forge/linux-64 ClobberErrors, but ultimately worked and the recipe ran!
Thank you all for continuing to follow up!