H5py: OSError: Unable to read attribute (No appropriate function for conversion path)

Created on 13 Jun 2016 · 8Comments · Source: h5py/h5py

I'm trying to open a netCDF4 file created by Xarray (made by opening a netCDF3 file and re-saving as netCDF4 using .to_netcdf) and I get the error below.

import h5py as h5
with h5.File('test.nc', 'r') as fd:
    print(list(fd['TMP_L103'].attrs.items()))

OSError                                   Traceback (most recent call last)
<ipython-input-5-bd3188829b86> in <module>()
      3 fd = h5.File('test.nc', 'r')
      4 
----> 5 print(list(fd['TMP_L103'].attrs.items()))

/Users/<me>/anaconda/lib/python3.5/site-packages/h5py/_hl/base.py in __iter__(self)
    336         with phil:
    337             for key in self._mapping:
--> 338                 yield (key, self._mapping.get(key))
    339 
    340 

/Users/<me>/anaconda/lib/python3.5/_collections_abc.py in get(self, key, default)
    593         'D.get(k[,d]) -> D[k] if k in D, else d.  d defaults to None.'
    594         try:
--> 595             return self[key]
    596         except KeyError:
    597             return default

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2582)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2541)()

/Users/<me>/anaconda/lib/python3.5/site-packages/h5py/_hl/attrs.py in __getitem__(self, name)
     77 
     78         arr = numpy.ndarray(shape, dtype=dtype, order='C')
---> 79         attr.read(arr, mtype=htype)
     80 
     81         if len(arr.shape) == 0:

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2582)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2541)()

h5py/h5a.pyx in h5py.h5a.AttrID.read (-------src-dir--------/h5py/h5a.c:5123)()

h5py/_proxy.pyx in h5py._proxy.attr_rw (-------src-dir--------/h5py/_proxy.c:915)()

OSError: Unable to read attribute (No appropriate function for conversion path)

This file in question is this.

Both h5dump and ncdump can read the attribute in question;

HDF5 "/Users/<me>/Downloads/test.nc" {
ATTRIBUTE "standard_name" {
   DATATYPE  H5T_STRING {
      STRSIZE 15;
      STRPAD H5T_STR_NULLTERM;
      CSET H5T_CSET_UTF8;
      CTYPE H5T_C_S1;
   }
   DATASPACE  SCALAR
   DATA {
   (0): "air_temperature"
   }
}
}

Note: this was first brought up in https://github.com/shoyer/h5netcdf/issues/16, but appears to actually be an issue in h5py

enhancement pull-request-welcome strinunicode usage

Source

mangecoeur

👍4 😕1

Most helpful comment

I can confirm that this works now, with h5py 2.10!

I tested with the strings.nc file from https://github.com/Unidata/netcdf-c/issues/298#issuecomment-238069437.

shoyer on 14 Sep 2019

🚀1 🎉1

All 8 comments

Newer netCDF4 version unfortunately make this issue a bit more serious as it causes an incompatibility between netCDF4 and h5py. netCDF4 version 1.2.3 introduced a new setncattr_string() method and the previously also existing setncattr() method will now always attempt to write bytes which h5py does not like. Discussion on the netCDF4 github: https://github.com/Unidata/netcdf4-python/issues/529

A great downside is that h5py is no longer able to read some netcdf files if they were produces with newer versions.

Short code snippet to reproduce:

import netCDF4
import h5py

print("netCDF4 version:", netCDF4.__version__)
print("h5py version:", h5py.__version__)

filename = "temp.h5"

with netCDF4.Dataset(filename, "w", format="NETCDF4") as f:
    f.setncattr_string("random", "test")
    # This, if written with netcdf4 >= 1.2.3, can no longer be read by h5py.
    f.setncattr("random_2", "test_2")

with h5py.File(filename, mode="r") as f:
    # Works.
    print(f.attrs["random"])
    # Fails.
    print(f.attrs["random_2"])

Output:

netCDF4 version: 1.2.4
h5py version: 2.6.0
['test']
Traceback (most recent call last):
  File "netcdf4_h5py_incompat.py", line 17, in <module>
    print(f.attrs["random_2"])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2582)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2541)
  File "/Users/lion/.miniconda3/envs/instaseis_py35/lib/python3.5/site-packages/h5py/_hl/attrs.py", line 79, in __getitem__
    attr.read(arr, mtype=htype)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2582)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir--------/h5py/_objects.c:2541)
  File "h5py/h5a.pyx", line 355, in h5py.h5a.AttrID.read (-------src-dir--------/h5py/h5a.c:5123)
  File "h5py/_proxy.pyx", line 36, in h5py._proxy.attr_rw (-------src-dir--------/h5py/_proxy.c:915)
OSError: Unable to read attribute (No appropriate function for conversion path)

Dump outputs:

$ ncdump temp.h5
netcdf temp {

// global attributes:
        string :random = "test" ;
        :random_2 = "test_2" ;
}



$ h5dump temp.h5
HDF5 "temp.h5" {
GROUP "/" {
   ATTRIBUTE "random" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): "test"
      }
   }
   ATTRIBUTE "random_2" {
      DATATYPE  H5T_STRING {
         STRSIZE 6;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "test_2"
      }
   }
}
}

krischer on 18 Jul 2016

The issue here is that recent versions of netCDF-C save the NC_CHAR dtype as fixed length UTF8 strings, which h5py cannot read.

shoyer on 7 Aug 2016

I get this error when I try to access an attribute value that is a string of any length. The keys work and I can get strings when they're in a list. I've tried this on Windows and Red Hat Linux, Python 2 and 3.