Iris: Iris causes ValueError for empty spaces in long_name

[JuliaKukulies]

When using iris.load_cube(file, constraint) to read in a variable of a netCDF4 file which has an empty space in its long_name, iris raises ValueError: 'substring not found' when using the long_name as a constraint. The ValueError does not give any clear message how to solve the problem, which in this case is straight forward: the long_name should not contain any empty spaces or special characters. To solve this problem, one can define a new long_name, e.g. with NCO:

ncatted -a long_name,'varname',o,c,'new_long_name' in.nc out.nc

When, however, using the variable name as a constraint in iris.load_cube, no ValueError appears until you display the cube. In contrast to #3261, it worked to read in the data based on the variable name (instead of standard_name or long_name)m but any further operations with the cube object fail.

[bjlittle]

@JuliaKukulies Awesome, thanks for raising this issue. Much appreciated! :smile:

Could you provide an example of the constraint that you are using to load your example file, with the traceback, just to clarify.

Also, could you provide the output from a ncdump -h of the file, if possible.

Many thanks :+1:

[JuliaKukulies]

Yes, of course! :)

The ncdump -h output of the file is:

ncdump -h CNRR-GSISAT_hourly_rain_199706.nc4
netcdf CNRR-GSISAT_hourly_rain_199706 {
dimensions:
time = UNLIMITED ; // (720 currently)
west_east = 480 ;
south_north = 360 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "hours since 1997-1-1 0:0:0" ;
time:calendar = "standard" ;
time:axis = "T" ;
float prcp(time, south_north, west_east) ;
prcp:long_name = "CNRR-GSISAT hourly rain, 1997" ;
prcp:units = "mm/hr" ;
float LAT(south_north, west_east) ;
LAT:description = "LATITUDE, SOUTH IS NEGATIVE" ;
LAT:units = "degree_north" ;
float LON(south_north, west_east) ;
LON:description = "LONGITUDE, WEST IS NEGATIVE" ;
LON:units = "degree_east" ;

// global attributes:
:CDI = "Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.6" ;
:history = "Wed Jul 10 11:00:19 2019: ncks -A -v LON surface_20150112.nc 1997_mon06.nc4\n",
"Wed Jul 10 11:00:01 2019: ncks -A -v LAT surface_20150112.nc 1997_mon06.nc4\n",
"Wed Jul 10 10:51:41 2019: cdo splitmon CNRR-GSISAT_hourly_rain_1997.nc4 1997_mon" ;
:title = "CNRR-GSISAT hourly rain, 1997" ;
:CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ;
:history_of_appended_files = "Wed Jul 10 11:00:19 2019: Appended file surface_20150112.nc had no \"history\" attribute\n",
"Wed Jul 10 11:00:01 2019: Appended file surface_20150112.nc had no \"history\" attribute\n",
"" ;
:NCO = "netCDF Operators version 4.7.9 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)" ;

When I read in the file with iris.load_cube('CNRR-GSISAT_hourly_rain_199706.nc4 ', 'CNRR-GSISAT hourly rain, 2016'), I get:

ValueError Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
343 method = get_real_method(obj, self.print_method)
344 if method is not None:
--> 345 return method()
346 return None
347 else:
~/anaconda3/lib/python3.6/site-packages/iris/cube.py in _repr_html_(self)
2113 from iris.experimental.representation import CubeRepresentation
2114 representer = CubeRepresentation(self)
-> 2115 return representer.repr_html()
2116
2117 def __iter__(self):
~/anaconda3/lib/python3.6/site-packages/iris/experimental/representation.py in repr_html(self)
301 else:
302 self._get_bits(lines)
--> 303 content = self._make_content()
304
305 return self._template.format(header=header,
~/anaconda3/lib/python3.6/site-packages/iris/experimental/representation.py in _make_content(self)
272 colspan = 0
273 else:
--> 274 split_point = line.index(':')
275 title = line[:split_point].strip()
276 body = line[split_point + 2:].strip()
ValueError: substring not found

Actually, there are two fixes for this problem: 1. redefine the long_name with ncatted -a long_name,'varname',o,c,'new_long_name' in.nc out.nc or remove the other variables (LAT and LON). When there is only one float variable prcp that has a long_name with empty space, it works without problems to read in this file (so my suggestion that it has to do with the empty space might actually not be true):

ncdump -h CNRR-GSISAT_hourly_rain_201606.nc4
netcdf CNRR-GSISAT_hourly_rain_201606 {
dimensions:
time = UNLIMITED ; // (720 currently)
west_east = 480 ;
south_north = 360 ;
variables:
double time(time) ;
time:standard_name = "time" ;
time:units = "hours since 2016-1-1 0:0:0" ;
time:calendar = "standard" ;
time:axis = "T" ;
float prcp(time, south_north, west_east) ;
prcp:long_name = "CNRR-GSISAT hourly rain, 2016" ;
prcp:units = "mm/hr" ;

// global attributes:

:CDI = "Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.6" ;
:history = "Wed Jul 10 10:43:24 2019: cdo splitmon CNRR-GSISAT_hourly_rain_2016.nc4 2016_mon" ;
:title = "CNRR-GSISAT hourly rain, 2016" ;
:CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ;
}

What could be the reason for that?

[bjlittle]

@JuliaKukulies Awesome, thanks.

Okay, so you've given me enough to recreate the issue on my side. Which is brilliant :+1:

The problem is not to do with iris failing to load your data - it loads your data just fine. The issue is actually when iris attempts to render the repr of the loaded cube in HTML.

I'm assuming that you're seeing this issue whilst running in a Jupyter notebook or Jupyterlab, right?

[JuliaKukulies]

Yes, you are correct! The issue appears when I use Jupyter notebooks. Thank you very much for your help! Is there any way to prevent this error or is it the best to just redefine the long_name?

[bjlittle]

Digging a bit further, I believe the source of the issue is related to our iris.experimental.representation.CubeRepresentation failing to parse and render your global attributes from the NetCDF file. So for me this is a bug that we need to fix.

The temporary workaround for you at the moment (until we fix it) is to either, simply explicitly:

print(cube)

in a Jupyter notebook cell, to get the string representation of a cube rather than the HTML, or:

import iris

def callback(cube, field, fname):
    cube.attributes = None

cube = iris.load_cube('CNRR-GSISAT_hourly_rain_199706.nc4', 'CNRR-GSISAT hourly rain, 2016', callback=callback)

will remove the cube.attributes, and allow you to render your cube in HTML by entering:

cube

in a Jupyter notebook cell.

Or indeed, you could simply do:

import iris
cube = iris.load_cube('CNRR-GSISAT_hourly_rain_199706.nc4', 'CNRR-GSISAT hourly rain, 2016')
cube.attributes = None

[bjlittle]

@JuliaKukulies If you do either of the above temporary workarounds, that will save you the chore of using ncatted to tinker with the actual NetCDF file.

Is this a reasonable temporary workaround for now?

Apologies for the bug and inconvenience, but thanks for taking the time to report this, much appreciated indeed :smile:

[JuliaKukulies]

Sounds like a good and useful workaround which is definitely good enough for me :-) ! Thank you very much for your help! Your solution is indeed much better than ncatted ! And glad if I could contribute to notice this bug.

[pp-mo]

I think #3373 has fixed this.
@JuliaKukulies is there more going wrong here, or can we close this ??

[JuliaKukulies]

Many thanks! You can close the issue.

[lbdreyer]

Many thanks! You can close the issue.

Thanks @JuliaKukulies !