Esmvaltool: Iris 2.0 and 2.1 does not work with netcdf4=1.4.0

[valeriupredoi]

when installing iris via anaconda with the usual conda install -c conda-forge iris this will automatically pick up iris=2.0 and now it will pick up netcdf4=1.4.0. Iris does not work with this latest version though:

In [1]: import iris
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-089b5db944e7> in <module>()
----> 1 import iris

~/anaconda3/lib/python3.6/site-packages/iris/__init__.py in <module>()
    111 import iris._constraints
    112 from iris._deprecation import IrisDeprecation, warn_deprecated
--> 113 import iris.fileformats
    114 import iris.io
    115

~/anaconda3/lib/python3.6/site-packages/iris/fileformats/__init__.py in <module>()
     27                                    UriProtocol, LeadingLine)
     28 from . import abf
---> 29 from . import um
     30 from . import name
     31 from . import netcdf

~/anaconda3/lib/python3.6/site-packages/iris/fileformats/um/__init__.py in <module>()
     29 from ._ff_replacement import um_to_pp, load_cubes, load_cubes_32bit_ieee
     30 from ._fast_load import structured_um_loading
---> 31 from ._fast_load_structured_fields import FieldCollation
     32 __all__ = ['um_to_pp', 'load_cubes', 'load_cubes_32bit_ieee',
     33            'structured_um_loading', 'FieldCollation']

~/anaconda3/lib/python3.6/site-packages/iris/fileformats/um/_fast_load_structured_fields.py in <module>()
     29 import itertools
     30
---> 31 from netCDF4 import netcdftime
     32 import numpy as np
     33

ImportError: cannot import name 'netcdftime'

reverting to netcdf4=1.3.1, it works fine.

@bjlittle are you Iris folk aware of this? I am assigning a coule guys @mattiarighi and @bouweandela just so we are aware of this when we go to iris 2.0 and this issue will not have been sorted :grin:

[bouweandela]

@valeriupredoi This seems to be purely an issue with iris. Maybe it would be better to open an issue on the github page of iris instead of on the esmvaltool page?

[valeriupredoi]

@bouweandela yep, agreed, man, I added the iris label, it's just for bookkeeping for us here just in case someone hits this issue and doesn't know how to fix it. I emailed @bjlittle about it as well, am sure the Iris guys are aware of it nonetheless :)

[bjlittle]

@valeriupredoi There has been a bit of necessary churn in iris for various reasons.

To cut a very long story short, we've now cut iris version 2.1, which unpins matplotlib to 2.2.2 and numpy to 1.14.3. This will work with the latest netcdf4 version 1.4.0 and pull in the new cftime package, which is a replacement for the netcdf4.netcdftime package. The netcdf4.netcdftime package has been removed from netcdf4 version 1.4.0 onwards, see https://github.com/Unidata/netcdf4-python/blob/master/Changelog#L12.

If you're going to use iris version 2.0 or less then you must pin netcdf4 to be <1.4

We also renamed cf_units package to be cf-units, and the nc_time_axis package to be nc-time-axis, and bumped both of these up to new release versions 2.0.2 and 1.1.0 respectively - they both also now pull in the new cftime package.

HTH clarify the situation...

[valeriupredoi]

@bjlittle thanks a bunch, man! Yes, we'll do that, once the switch from iris 1.13. So we'll go straight to 2.1 then. In factanything that uses nc-time-axis will not work unless the iris plot is changed or 2.1 is used, I had this issue on Friday with a diagnostic plotting and was quite bedazzled :grin:
I suggest keeping this issue and #404 open just as references for when we switch to 2.1. BTW 2.1 doesn't yet work with netcdf4=1.4

[bjlittle]

@valeriupredoi No worries.

Also, slightly confused about you saying that iris version 2.1 is not working with netcdf4 version 1.4.0.

According to travis-ci on our latest upstream/master branch of iris... all is working just fine.

Infact, we've just added a conda list --explict to our travis-ci logging, so that you can easily recreate the travis-ci conda environment locally, see here.

Perhaps you should compare and contrast the packages in our travis-ci conda environment with your local conda environment to see if there are any obvious differences...

[valeriupredoi]

hey @bjlittle have a looksee:

(esmvaltool_autoassess) [valeriu@jasmin-sci2 esmvaltool_autoassess]$ conda install -c conda-forge iris=2.1
Solving environment: done

# All requested packages already installed.

(esmvaltool_autoassess) [valeriu@jasmin-sci2 esmvaltool_autoassess]$ conda list netcdf4
# packages in environment at /home/users/valeriu/anaconda3/envs/esmvaltool_autoassess:
#
# Name                    Version                   Build  Channel
netcdf4                   1.3.1                    py36_3    conda-forge
(esmvaltool_autoassess) [valeriu@jasmin-sci2 esmvaltool_autoassess]$ conda install -c conda-forge netcdf4=1.4
Solving environment: done

## Package Plan ##

  environment location: /home/users/valeriu/anaconda3/envs/esmvaltool_autoassess

  added / updated specs: 
    - netcdf4=1.4


The following packages will be REMOVED:

    iris:    2.1.0-py36_0 conda-forge

The following packages will be UPDATED:

    netcdf4: 1.3.1-py36_3 conda-forge --> 1.4.0-py36_0 conda-forge

Not pretty :grin:

[bjlittle]

@valeriupredoi Awesome, thanks.

We simply forgot to unpin netcdf4 in the conda-forge iris recipe ... I'm on that right now!

Cheers :wink:

[valeriupredoi]

whoopsy :grin:

[bjlittle]

Yeah, guess we're human after all :stuck_out_tongue:

See https://github.com/conda-forge/iris-feedstock/pull/36

Cheers @valeriupredoi

[valeriupredoi]

what, I thought iris is the precursor of Skynet :grinning: Cheeers @bjlittle

[bjlittle]

@valeriupredoi My PR just got merged, so the iris fix should be available shortly from conda-forge.

[bjlittle]

@valeriupredoi The fix is now available from conda-forge.

Let me know if this resolves your issue.

[valeriupredoi]

it do indeed:

iris version:  2.1.0
cf_units version:  2.0.1
netCDF4 version:  1.4.0

cheers @bjlittle and the other iris eggheads :grin:

To be absolutely nitpicky, it would be nice if you guys pinned netcdf4=1.3.1 for iris=2.0 so people don't get confusado, at the moment with netcdf4=1.4 it will revert to iris 2.0 but not to netcdf4=1.3.1:

(esmvaltool_autoassess) [valeriu@jasmin-sci2 ~]$ conda install -c conda-forge iris=2.0
Solving environment: done

## Package Plan ##

  environment location: /home/users/valeriu/anaconda3/envs/esmvaltool_autoassess

  added / updated specs: 
    - iris=2.0


The following packages will be DOWNGRADED:

    iris: 2.1.0-py36_1 conda-forge --> 2.0.0-py36_1 conda-forge

Proceed ([y]/n)? y

[valeriupredoi]

am gunna close this, @bjlittle and the iris gang, great job, but pls bear in mind the last comment above :)
point to us -- go straight to iris 2.1+ when switching from 1.13