Iris: CF variable group not found in netCDF4 file; also not being able to load cube should tell the user why

[valeriupredoi]

Hi guys, I have been asked by @ledm why a certain NetCDF_v4 (priority 5) file can not be loaded by iris but ncdump works fine and in actuality it is a netCDF file with metadata and data in it (I am attaching the ncdump -h output; the results of load_raw or load are empty lists and load_cube returns iris.exceptions.ConstraintMismatchError: no cubes found). It proves out that the reason iris can not load any cube out of the file is iris.fileformats.netcdf.load_cubes() returns no cube since it can not find any CF variable ie the dict cf.cf_group.data_variables is totally empty. Why this is is beyond my paygrade, my two cents is that the attribute :Conventions = "CF-1.7 CMIP-6.2" ; is wonky. But you know better! Can I suggest one thing and ask anthoer pls:

• in such cases it'd be nice if iris actually told the user I can't bloody load the cube because I couldn't find any CF vars;
• why would it not be able to find any CF vars?
  Many thanks in advance! :beer:
  @bjlittle @pelson @lbdreyer
  problem_file.txt

[valeriupredoi]

UPDATE I changed the :Conventions via ncatted and still have the same loading issue, I also saw that the CMOR CMIP6_Ofx.json has in the header "Conventions": "CF-1.7 CMIP-6.2" so that should be fine. @ledm has reported that any currently available CMIP6 volcello file off the ESGF can not be loaded in iris as a cube...quite fishy :fish:

[valeriupredoi]

So this is how a volcello CMIP5 CF variable looks like as found by iris:

{'volcello': CFDataVariable('volcello', <class 'netCDF4._netCDF4.Variable'>
float32 volcello(lev, j, i)
    standard_name: ocean_volume
    long_name: Ocean Grid-Cell Volume
    comment: grid-cell volume ca. 2000.
    units: m3
    missing_value: 1e+20
    _FillValue: 1e+20
    associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_ocean_fx_MPI-ESM-LR_historical_r0i0p0.nc
    coordinates: lat lon
unlimited dimensions: 
current shape = (40, 220, 256)
filling on)}

is this happening because iris still works with CF=1.5 maybe?

[bjlittle]

@valeriupredoi Good to hear from you 😃

Looking at the ncdump that you provided there's nothing obviously wrong that I can see. It's certainly not related to the Conventions attribute.

Would it be possible for you to share the actual NetCDF file with me? Or explain how and where I can download it from? That's the only way that I'm going to easily progress this for you.

Thanks!

[ledm]

Hi Bill,

On JASMIN, I'm finding that this problem occurs will all the files that I have looked at matching:

ls /badc/cmip6/data/CMIP6/CMIP///historical/r/O/volcello/gr/latest/volcello*:

/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r10i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r10i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r11i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r11i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r1i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r1i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r2i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r2i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r3i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r3i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r4i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r4i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r5i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r5i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r6i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r6i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r7i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r7i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r8i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r8i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/historical/r9i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_CESM2_historical_r9i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r10i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r10i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r11i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r11i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r12i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r12i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r13i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r13i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r14i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r14i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r15i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r15i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r16i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r16i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r17i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r17i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r18i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r18i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r19i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r19i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r1i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r1i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r20i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r20i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r21i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r21i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r22i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r22i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r23i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r23i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r24i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r24i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r25i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r25i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r26i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r26i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r27i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r27i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r28i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r28i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r29i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r29i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r2i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r2i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r30i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r30i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r3i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r3i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r4i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r4i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r5i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r5i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r6i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r6i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r7i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r7i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r8i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r8i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorCPM1/historical/r9i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorCPM1_historical_r9i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorESM2-LM_historical_r1i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorESM2-LM/historical/r2i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorESM2-LM_historical_r2i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NCC/NorESM2-LM/historical/r3i1p1f1/Ofx/volcello/gr/latest/volcello_Ofx_NorESM2-LM_historical_r3i1p1f1_gr.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_185001-186912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_187001-188912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_189001-190912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_191001-192912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_193001-194912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_195001-196912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_197001-198912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_199001-200912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-CM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-CM4_historical_r1i1p1f1_gr_201001-201412.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_185001-186912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_187001-188912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_189001-190912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_191001-192912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_193001-194912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_195001-196912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_197001-198912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_199001-200912.nc
/badc/cmip6/data/CMIP6/CMIP/NOAA-GFDL/GFDL-ESM4/historical/r1i1p1f1/Omon/volcello/gr/latest/volcello_Omon_GFDL-ESM4_historical_r1i1p1f1_gr_201001-201412.nc

If you can't access jasmin, these files are also available on ESGF. MIP era: CMIP6, experiment: historical, variable: volcello. Thanks!

[valeriupredoi]

@bjlittle extra extra :grin: Man, I tried xarray and the file loads fine in it. I also put one of these volcello up for you to download easily at (it's weird.nc, no pun intended) https://github.com/NCAS-CMS/NCAS-Useful-Documentation/tree/master/iris :beer:

[valeriupredoi]

@bjlittle here's a clue I got from xarray (while trying to save the loaded file):

ValueError: Variable 'volcello' has multiple fill values [1e+20, 1e+20]. Cannot encode data.

and indeed asking for a type it looks like the fill value is both

<class 'numpy.float32'>
<class 'numpy.float64'>

[valeriupredoi]

where

fv = encoding.get('_FillValue') -> float32
mv = encoding.get('missing_value') -> float64

[ledm]

What I don't understand is why this would happen for so many different models? Is there an issue in the CMOR table?

[bjlittle]

@valeriupredoi Thanks. I'm offline pretty much today (only capable of servicing GitHub), but I'll spin up on this early tomorrow morning and investigate further.

[bjlittle]

@ledm What ever it is, it seems to be systemic.

[ledm]

Thanks @bjlittle, any help you can give us on this tomorrow would be great. It's a fairly urgent problem for us. We'll keep looking at it and post anything we find here - but we understand that you might be see anything until tomorrow. Cheers!

[bjlittle]

@ledm and @valeriupredoi Of course, this might be symptomatic of a dependency issue, rather than a data issue... just guessing as I have no laptop, only my mobile. What version of netcdf4-python are you guys using?

[bjlittle]

netcdf4-python version 1.5.3 was just released on 27 Oct 2019 and it addressed https://github.com/Unidata/netcdf4-python/issues/972... Which is in the area of fill-values 🤔

Might be relevant, might absolutely not...

[valeriupredoi]

netcdf4=1.5.1.2==py37h73a1b54_1==conda-forge

[valeriupredoi]

will attempt installing 1.5.3, just saw it released :beer:

[valeriupredoi]

Oh but noooo - it is python3.8 and we can't move to 3.8 just yet, I'll attempt anyways just to check the pure iris functionality

[bjlittle]

I don't think you can move to python3.8 just yet, all the dependencies aren't available yet... Well, not for iris

[bjlittle]

Also libnetcdf just recently changed as well, see https://www.unidata.ucar.edu/software/netcdf/docs/RELEASE_NOTES.html.

The latest available on conda-forge is 4.7.2. What version are you guys using?

[valeriupredoi]

(irisenv) [valeriu@jasmin-sci1 ~]$ conda list iris
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv:
#
# Name                    Version                   Build  Channel
iris                      2.2.1                    py38_0    conda-forge
(irisenv) [valeriu@jasmin-sci1 ~]$ conda list netcdf4
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv:
#
# Name                    Version                   Build  Channel
netcdf4                   1.5.3            py38hf3213d6_0    conda-forge
(irisenv) [valeriu@jasmin-sci1 ~]$ conda list python
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv:
#
# Name                    Version                   Build  Channel
antlr-python-runtime      4.7.2                 py38_1000    conda-forge
msgpack-python            0.6.2            py38hc9558a2_0    conda-forge
python                    3.8.0                h357f687_3    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
(irisenv) [valeriu@jasmin-sci1 ~]$ conda list libnetcdf
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv:
#
# Name                    Version                   Build  Channel
libnetcdf                 4.7.1           nompi_h94020b1_101    conda-forge

still no luck! BTW iris and py3.8 installed OK in the same env, and the import works fine. Can't upgrade to 4.7.2, the env refuses to solve at the moment

[valeriupredoi]

reverted to older versions:

(irisenv2) [valeriu@jasmin-sci1 ~]$ conda list iris
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv2:
#
# Name                    Version                   Build  Channel
iris                      2.2.1                    py37_0    conda-forge
(irisenv2) [valeriu@jasmin-sci1 ~]$ conda list netcdf4
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv2:
#
# Name                    Version                   Build  Channel
netcdf4                   1.4.2           py37had69b76_1001    conda-forge
(irisenv2) [valeriu@jasmin-sci1 ~]$ conda list libnetcdf
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv2:
#
# Name                    Version                   Build  Channel
libnetcdf                 4.6.2             hbdf4f91_1001    conda-forge
(irisenv2) [valeriu@jasmin-sci1 ~]$ conda list hdf5
# packages in environment at /home/users/valeriu/anaconda3R/envs/irisenv2:
#
# Name                    Version                   Build  Channel
hdf5                      1.10.4          nompi_h3c11f04_1106    conda-forge

still no joy! It doesn't seem to be a dependency issue :crying_cat_face:

[bjlittle]

Interesting...well if it's a data problem, then that might be potentially huge. Hope that's not the case...

[bjlittle]

How recent is the CMIP6 data?

[valeriupredoi]

the files that @ledm has listed have been uloaded either on Jun 11, Aug 2 or Nov 16

[bjlittle]

Does all volcello CMIP6 data have this issue? Or just "recent" data? Is that worth knowing?

[valeriupredoi]

all, but I just tested with xarray on a different file and didnt notice the _FillValue complaint. Not the fillvalue then. It looks as if volcello is not identified by the CF as a valid CF variable (well, the CF used by iris that is) -> the initial clue I got from yesterday where the CF data vars dictionary is empty :confounded:

[bjlittle]

@valeriupredoi Cool, leave it with me and I'll spin up on this tomorrow morning.

[valeriupredoi]

I found it!! So the actual data variables and promoted variables for volcello ie self.cf_group.data_variables, self.cf_group.promoted are empty (and this is what necdf.py is looking to load ie

        data_variables = (list(cf.cf_group.data_variables.values()) +
                          list(cf.cf_group.promoted.values()))

so those are empty - and this is because volcello is treated as a measure variable:

CFMeasureVariable('volcello', <class 'netCDF4._netCDF4.Variable'>

you need to promote it to real var, man :beer:

[bjlittle]

@valeriupredoi Could you please mind your use of language, this is a public forum.

iris is not recognising this netCDF variable as a cube because of this metadata:

variables:
    float volcello(lev, nlat, nlon) ;
        volcello:_FillValue = 1.e+20f ;
        volcello:cell_measures = "area: areacello volume: volcello" ;

So iris is actually doing the right thing, as volcello is self-declaring itself as a cell_measure, which is pretty odd. It's either data i.e., a cube, or it's not. It doesn't really make sense for it to be data and a cell measure of itself.

I'll have a think about all this, and whether there is a workaround or fix that might assist here. The fact that CMIP6 is now producing self-referencing metadata is questionable, but that's not going to make its data go out of distribution any time soon, or ever, for the community.

[valeriupredoi]

Could you please mind your use of language, this is a public forum.

sorry, edited it :grin:

So iris is actually doing the right thing, as volcello is self-declaring itself as a cell_measure,

Indeed, maybe create a list of variables that can be both ways? I think areacello is the same -> if I remember correctly @zklaus was telling us all about these cell measures that are variables as well :beer:

I have this in cf.py right at the end of CFReader._build_cf_groups()

        for cf_var in self.cf_group._cf_variables:
            if isinstance(self.cf_group._cf_variables[cf_var], CFMeasureVariable):
                meas_var = self.cf_group._cf_variables[cf_var]
                built_meas_var = _build(meas_var)
                self.cf_group.data_variables[cf_var] = meas_var

but it's not working -> any idea how to include volcello just temporarily so @ledm can run his jobs? :beer:

[zklaus]

Two quick comments:

1. > The fact that CMIP6 is now producing self-referencing metadata is questionable, but that's not going to make its data go out of distribution any time soon, or ever, for the community.

Never say never. The analogous entry in areacello was purged in version 01.00.17 of the data_specs so there is hope yet.

1. Maybe having volcello as a measure is exactly what @ledm needs? After all, usually you are looking for this to calculate something with another quantity and volcello shows correctly up as a cell_measure on the other quantity? At least that would be the ideal case. Last we checked, iris wasn't able to handle cell_measures explicitely, but if it is now, maybe it is time to check the conformance of ESMValTool as well.

[valeriupredoi]

@bjlittle so I managed to add the MeasureVariable as DataVariable to the cf_group by adding

                for cf_var in cf_group:
                    if isinstance(cf_group._cf_variables[cf_var], CFMeasureVariable):
                        data_var = CFDataVariable(cf_var,
                                                  cf_group[cf_var].cf_data)
                        cf_group[cf_var] = data_var

            # Add the CF group to the variable.
            cf_variable.cf_group = cf_group

        # Ignored variables are those that cannot be attached to a
        # data variable as the dimensionality of that variable is not
        # a subset of the dimensionality of the data variable.
        ignored = set()
        for cf_variable in six.itervalues(self.cf_group):
            _build(cf_variable)

but the final self.cf_group.data_variables is still empty, no clues why - I'll trust you know how to proceed from here. Cheers muchly for all the effort in this issue, man :beer:

[ledm]

So iris is actually doing the right thing, as volcello is self-declaring itself as a cell_measure, which is pretty odd. It's either data i.e., a cube, or it's not. It doesn't really make sense for it to be data and a cell measure of itself.

Is it worth raising this as an issue with the CMOR guys?

This is the CMOR table for "volcello" in CMIP6_Oyr.json:

        "volcello": {
            "frequency": "yr", 
            "modeling_realm": "ocean", 
            "standard_name": "ocean_volume", 
            "units": "m3", 
            "cell_methods": "area: mean where sea time: mean", 
            "cell_measures": "area: areacello volume: volcello", 
            "long_name": "Ocean Grid-Cell Volume", 
            "comment": "grid-cell volume ca. 2000.", 
            "dimensions": "longitude latitude olevel time", 
            "out_name": "volcello", 
            "type": "real", 
            "positive": "", 
            "valid_min": "", 
            "valid_max": "", 
            "ok_min_mean_abs": "", 
            "ok_max_mean_abs": ""
        }, 

[zklaus]

It's perhaps more an issue of the data request, but yeah, definitely worth raising there.

[bjlittle]

@valeriupredoi @ledm @zklaus

Just checked and this issue has been already been fixed in upstream/master of iris by https://github.com/SciTools/iris/pull/3386. This will be part of iris for the forthcoming releases, in both versions 2.3.0 (Python2 and Python3) and 3.0.0 (Python3 only).

If you have an environment with all the installed dependencies of iris, then you can just use the master of iris instead and it'll all work :wink:

[bjlittle]

I'm now happy to close this issue, if you guys are...

[ledm]

Before we do that, @valeriupredoi, how do we implement this fix in ESMValTool?

[ledm]

Actually, lets continue that discussion on the ESMValTool github:
https://github.com/ESMValGroup/ESMValTool/issues/1442

[valeriupredoi]

Cheers much Bill!! Lee, we can't implement it in an automated way just yet,
since the iris conda package is not yet to the master dev version, but I
guess we can grab the branch as Bill says and install from source, lemme
try do it myself and will tell you how you can do it too (I never installed
iris from source before)

Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom

On Wed, 20 Nov 2019, 10:52 Lee de Mora, [e-mail] wrote:

Before we do that, @valeriupredoi https://github.com/valeriupredoi, how
do we implement this fix in ESMValTool?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SciTools/iris/issues/3544?email_source=notifications&email_token=AG5EFIYO2GDB5FSSXQQHWMLQUUJFDA5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERSAZI#issuecomment-555950181,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG5EFI4SXJFBOHPMHXGKFPTQUUJFDANCNFSM4JOS456Q
.

[bjlittle]

Actually, lets continue that discussion on the ESMValTool github:
ESMValGroup/ESMValTool#1442

@ledm Yeah, that kinda make sense to me.

Ideally, you'd just want to conda install -c conda-forge iris=3.*, but that's not available just yet, well most likely not until next month.

Until then, perhaps you're happy to run with something like installing all the iris dependencies and then use something along the lines of pip install git+https://github.com/scitools/iris/ --no-deps... which should pull in the master version of iris. Something like that.

I'll keep this issue open for a while in case there is any follow-on chatter.

[valeriupredoi]

Cheers, Bill! In theory all deps should have been installed by conda in the
esmvaltool env, unless iris 3 has new deps vs 2.2.1. Ill try as soon as I
get in work. Much thanks to Bill for being so on the ball :beer:

Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom

On Wed, 20 Nov 2019, 10:59 Bill Little, [e-mail] wrote:

Actually, lets continue that discussion on the ESMValTool github:
ESMValGroup/ESMValTool#1442
https://github.com/ESMValGroup/ESMValTool/issues/1442

@ledm https://github.com/ledm Yeah, that kinda make sense to me.

Ideally, you'd just want to conda install -c conda-forge iris=3.*, but
that's not available just yet, well most likely not until next month.

Until then, perhaps you're happy to run with something like installing all
the iris dependencies and then use something along the lines of pip
install git+https://github.com/scitools/iris/ --no-deps... which should
pull in the master version of iris. Something like that.

I'll keep this issue open for a while in case there is any follow-on
chatter.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SciTools/iris/issues/3544?email_source=notifications&email_token=AG5EFI2U3HF2CK3FPP4BA3DQUUKCHA5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERSXHY#issuecomment-555953055,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG5EFI7H666LI76HZPCSAM3QUUKCHANCNFSM4JOS456Q
.

[valeriupredoi]

@bjlittle @ledm I have installed iris=3 and it loads the file like a charm :tada: Absolute ledgs, the iris folk :beer:
Lee, to get it installed, do these steps:

• get in the conda esmvaltool environment;
• remove the previously installed iris (2.2.1dev0)

rm -r $YOUR_ANACONDA_DIR/envs/esmvaltool/lib/python3.7/site-packages/iris/

NOTE: this will keep the source in the pkgs dir so you can reinstall it later; iris=3 installs in a dir called $YOUR_ANACONDA_DIR/anaconda3R/envs/esmvaltool/lib/python3.7/site-packages/scitools_iris-3.0.dev0-py3.7.egg/iris/

• get the gitball:

git clone https://github.com/SciTools/iris.git

• cd into iris, and install via python setup.py install

Loading of volcello files will be biutiful :grin:

[valeriupredoi]

@bjlittle go ahead and close this, man. Also, note that the installation via pip is not working due to

  ERROR: Failed building wheel for scitools-iris
  Running setup.py clean for scitools-iris
Failed to build scitools-iris
ERROR: Could not build wheels for scitools-iris which use PEP 517 and cannot be installed directly

which is something Ive run into before for other packages and not even related to iris, it's pip that is at fault, adding --no-binary opt is not useful either

[zklaus]

Alternatively, and to keep pips db in order, don't delete anything manually and instead of python setup.py install use pip install -e ..

I wrote the above before V's last comment appeared, but it is confusing for me because I always do my iris source installation with pip install -e . with no problem.

[valeriupredoi]

(esmvaltool) [valeriu@jasmin-sci2 iris]$ pip install -e . --no-deps
Obtaining file:///home/users/valeriu/iris
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: scitools-iris
  Found existing installation: scitools-iris 2.2.1.dev0
    Uninstalling scitools-iris-2.2.1.dev0:
      Successfully uninstalled scitools-iris-2.2.1.dev0
  Running setup.py develop for scitools-iris
    ERROR: Command errored out with exit status 1:
     command: /home/users/valeriu/anaconda3R/envs/esmvaltool/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/users/valeriu/iris/setup.py'"'"'; __file__='"'"'/home/users/valeriu/iris/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/users/valeriu/iris/
    Complete output (24 lines):
    running develop
    running egg_info
    writing lib/scitools_iris.egg-info/PKG-INFO
    writing dependency_links to lib/scitools_iris.egg-info/dependency_links.txt
    writing requirements to lib/scitools_iris.egg-info/requires.txt
    writing top-level names to lib/scitools_iris.egg-info/top_level.txt
    reading manifest file 'lib/scitools_iris.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'docs/iris/build'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    writing manifest file 'lib/scitools_iris.egg-info/SOURCES.txt'
    running build_ext
    Creating /home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.7/site-packages/scitools-iris.egg-link (link to lib)
    scitools-iris 3.0.dev0 is already the active version in easy-install.pth

    Installed /home/users/valeriu/iris/lib
     [Running in-place]
    /home/users/valeriu/anaconda3R/envs/esmvaltool/bin/python tools/generate_std_names.py etc/cf-standard-name-table.xml lib/iris/std_names.py
    /home/users/valeriu/anaconda3R/envs/esmvaltool/bin/python -c import os; from pyke.target_pkg import target_pkg; target_pkg.load = lambda *args, **kwargs: None; os.chdir(os.path.join('lib', 'iris', 'fileformats', '_pyke_rules')); from pyke import knowledge_engine; knowledge_engine.engine('')
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'pyke'
    error: command '/home/users/valeriu/anaconda3R/envs/esmvaltool/bin/python' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of scitools-iris
  Moving to /home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.7/site-packages/scitools_iris-2.2.1.dev0.dist-info/
   from /home/users/valeriu/anaconda3R/envs/esmvaltool/lib/python3.7/site-packages/~citools_iris-2.2.1.dev0.dist-info
ERROR: Command errored out with exit status 1: /home/users/valeriu/anaconda3R/envs/esmvaltool/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/users/valeriu/iris/setup.py'"'"'; __file__='"'"'/home/users/valeriu/iris/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

I use pip install too but always revert to python setup.py install when dealing with new lands, at least that I know how to make work better than pip :grin:

[zklaus]

So somehow pyke is missing. Hm.

[valeriupredoi]

it's not, it's in the conda env:

(esmvaltool) [valeriu@jasmin-sci2 ~]$ conda list pyke
# packages in environment at /home/users/valeriu/anaconda3R/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
pyke                      1.1.1                 py37_1001    conda-forge

but pip is somehow unable to find it, maybe it needs the pypi version, dunno, I have encountered this before and that's why we stayed with the conda-forge installation for iris and not the pypi one. Altho I seem to recall fixing it. Anyway - outside the scope of this issue - let's wait for the "official" release of iris 3 but it already looks very promising :grin:

[valeriupredoi]

...and now volcello is a data variable:

CFGROUP = {'volcello': CFDataVariable('volcello', <class 'netCDF4._netCDF4.Variable'>
float32 volcello(lev, nlat, nlon)
    _FillValue: 1e+20
    cell_measures: area: areacello volume: volcello
    cell_methods: area:
...

:smile:

[bjlittle]

@valeriupredoi @zklaus Ohh that interesting issues with pyke on installation...

I can recreate that locally, which is a bit of a :pig:

To be honest, I'm keen to purge our use of pyke from iris... sooner rather than later would be a very good move indeed. That said, I'll investigate this side issue of pyke causing install to barf. The pyke package is needed to build the pyke rules, but you should be able to build and install iris in a clean way. I think this is fixable, and on us to do that... so leave that with me.

In the meantime, I'll close this issue.

Thanks guys :+1: :beers: :smile:

[valeriupredoi]

No, thank you Bill :beer: The pyke issue is a non issue when installing
from conda so we'll be fine when you guys release the conda package. In the
meantime you'll be happy to hear I produced IPCC AR6 plots with
ESMValTool+iris3 :grin:

Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom

On Fri, 22 Nov 2019, 09:41 Bill Little, [e-mail] wrote:

@valeriupredoi https://github.com/valeriupredoi @zklaus
https://github.com/zklaus Ohh that interesting issues with pyke on
installation...

I can recreate that locally, which is a bit of a 🐷

To be honest, I'm keen to purge our use of pyke from iris... sooner
rather than later would be a very good move indeed. That said, I'll
investigate this side issue of pyke causing install to barf. The pyke
package is needed to build the pyke rules, but you should be able to
build and install iris in a clean way. I think this is fixable, and on us
to do that... so leave that with me.

In the meantime, I'll close this issue.

Thanks guys 👍 🍻 😄

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SciTools/iris/issues/3544?email_source=notifications&email_token=AG5EFI6HY7OUPXBIZLBNIJLQU6SM5A5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE5DQHI#issuecomment-557463581,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG5EFI62NDDYGPUHXSJIB6LQU6SM5ANCNFSM4JOS456Q
.

[zklaus]

Just a post scriptum on the pyke installation error: This happened to me when I set up an environment first with conda create -n evt python=3 and then did an update with conda env update -n evt --file environment.yml. Turns out in my case the reason was that the creation of the environment installed a python 3.8 which was downgraded to python 3.7 in the update step which worked out completely ok, except that pip was somehow confused in looking things up until I deactivated and reactivated the environment (I may even have logged out in between), at which point everything worked smoothly.

[valeriupredoi]

most interesting @zklaus - I shall proceed to actually try to figure out what's the hap here :beer: