H5py: Python 3 string error

[CompPhysChris]

I'm one of the developers on the pycroscopy python package, and we make heavy use of hdf5 and h5py. We would like to make the switch to python3, but can't do so because of errors related to strings. We have datasets with attributes that are lists of strings, and these results in an error. Sending a single string works fine.

Test Code

import h5py
h5_f = h5py.File('test.h5', 'w')
h5_data = h5_f.create_dataset('some_data', (10,10), dtype=int)
h5_data.attrs['labels'] = ['X', 'Y']

Output

Traceback (most recent call last):
File "", line 1, in
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2798)
File "/usr/local/lib/python3.5/dist-packages/h5py/_hl/attrs.py", line 93, in __setitem__
self.create(name, data=value, dtype=base.guess_dtype(value))
File "/usr/local/lib/python3.5/dist-packages/h5py/_hl/attrs.py", line 171, in create
htype = h5t.py_create(original_dtype, logical=True)
File "h5py/h5t.pyx", line 1543, in h5py.h5t.py_create (/tmp/pip-huypgcah-build/h5py/h5t.c:18112)
File "h5py/h5t.pyx", line 1565, in h5py.h5t.py_create (/tmp/pip-huypgcah-build/h5py/h5t.c:17932)
File "h5py/h5t.pyx", line 1626, in h5py.h5t.py_create (/tmp/pip-huypgcah-build/h5py/h5t.c:17891)
TypeError: No conversion path for dtype: dtype('

I've seen other places where a manual encode/decode step was suggested, but this is not an option for us. While we can do this in our own code internal code, we are primarily a group that works with users at the Center for Nanophase Material Science (CNMS) at the Oak Ridge National Lab (ORNL). Our users have little to no programming knowledge and work primarily through jupyter notebooks.

Summary of the h5py configuration

h5py 2.7.0
HDF5 1.8.18
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609]
sys.platform linux
sys.maxsize 9223372036854775807
numpy 1.12.1

• Operating System: Linux Mint 18.1 Cinnamon 64-bit
• Where Python was acquired: apt-get

[rmvanhees]

A portable solution for your problem is to use the numpy function
'np.string_()'. This solution work for python v2 and v3.

Thus by replacing the last line of your code by:
h5_data.attrs['labels'] = np.string_(['X', 'Y'])

There is also a nice h5py documentation page on the usage of strings in
h5py:
http://docs.h5py.org/en/latest/strings.html

Succes, Richard

On 31 May 2017 at 20:51, CompPhysChris [e-mail] wrote:

I'm one of the developers on the pycroscopy
https://github.com/pycroscopy/pycroscopy python package, and we make
heavy use of hdf5 and h5py. We would like to make the switch to python3,
but can't do so because of errors related to strings. We have datasets with
attributes that are lists of strings, and these results in an error.
Sending a single string works fine.
Test Code

import h5py
h5_f = h5py.File('test.h5', 'w')
h5_data = h5_f.create_dataset('some_data', (10,10), dtype=int)
h5_data.attrs['labels'] = ['X', 'Y']

Output

Traceback (most recent call last):
File "", line 1, in
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
(/tmp/pip-huypgcah-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
(/tmp/pip-huypgcah-build/h5py/_objects.c:2798)
File "/usr/local/lib/python3.5/dist-packages/h5py/_hl/attrs.py", line 93,
in setitem
self.create(name, data=value, dtype=base.guess_dtype(value))
File "/usr/local/lib/python3.5/dist-packages/h5py/_hl/attrs.py", line
171, in create
htype = h5t.py_create(original_dtype, logical=True)
File "h5py/h5t.pyx", line 1543, in h5py.h5t.py_create
(/tmp/pip-huypgcah-build/h5py/h5t.c:18112)
File "h5py/h5t.pyx", line 1565, in h5py.h5t.py_create
(/tmp/pip-huypgcah-build/h5py/h5t.c:17932)
File "h5py/h5t.pyx", line 1626, in h5py.h5t.py_create
(/tmp/pip-huypgcah-build/h5py/h5t.c:17891)
TypeError: No conversion path for dtype: dtype('

I've seen other places where a manual encode/decode step was suggested,
but this is not an option for us. While we can do this in our own code
internal code, we are primarily a group that works with users at the Center
for Nanophase Material Science (CNMS) at the Oak Ridge National Lab (ORNL).
Our users have little to no programming knowledge and work primarily
through jupyter notebooks.
Summary of the h5py configuration

h5py 2.7.0
HDF5 1.8.18
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609]
sys.platform linux
sys.maxsize 9223372036854775807
numpy 1.12.1

• Operating System: Linux Mint 18.1 Cinnamon 64-bit
• Where Python was acquired: apt-get

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/h5py/h5py/issues/892, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADi7qsN54fU_6GuIb4AnjBKR7eARVNLuks5r_baxgaJpZM4NsCqP
.

[ssomnath]

Richard,

I tried your suggestion:

test = np.string_(['Hello','Bye'])
test
"[u'Hello', u'Bye']"
test[0]
'['
test[1]
'u'

So, this solution may not work for Chris. The h5py documentation page seems to have a simpler solution:

import h5py
h5_f = h5py.File('test_2.h5')

h5_f.attrs['labels'] = ['X', 'Y']
Traceback (most recent call last):
File "", line 1, in
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/Users/ilan/minonda/conda-bld/h5py_1490026561416/work/h5py/_objects.c:2846)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/Users/ilan/minonda/conda-bld/h5py_1490026561416/work/h5py/_objects.c:2804)
File "/Users/syz/anaconda/lib/python2.7/site-packages/h5py/_hl/attrs.py", line 93, in __setitem__
self.create(name, data=value, dtype=base.guess_dtype(value))
File "/Users/syz/anaconda/lib/python2.7/site-packages/h5py/_hl/attrs.py", line 171, in create
htype = h5t.py_create(original_dtype, logical=True)
File "h5py/h5t.pyx", line 1543, in h5py.h5t.py_create (/Users/ilan/minonda/conda-bld/h5py_1490026561416/work/h5py/h5t.c:18118)
File "h5py/h5t.pyx", line 1565, in h5py.h5t.py_create (/Users/ilan/minonda/conda-bld/h5py_1490026561416/work/h5py/h5t.c:17938)
File "h5py/h5t.pyx", line 1626, in h5py.h5t.py_create (/Users/ilan/minonda/conda-bld/h5py_1490026561416/work/h5py/h5t.c:17897)
TypeError: No conversion path for dtype: dtype('

h5_f.attrs['labels'] = [b'X', b'Y']
h5_f.attrs['labels'][0] == 'X'
True

Like Chris, I am only interested in (including lists of) single / multi character string attributes. I don't need to store a dataset of strings.

Would there be a problem to using this approach that I didn't think of?

[aragilar]

The np.string_ usage assumes that you are dealing with a single string, not a list of strings.
The safest usage would be to

h5_data.attrs['labels'] = np.encode(np.array(list_of_strings, type='U'), encoding='utf8')

noting that you should always be working with unicode (py3 strings, py2 unicode) rather than bytestrings (py3 bytes, py2 strings).

[ssomnath]

Thank you @aragilar and @rmvanhees

I tried a few things and I think, with the help of everyone, I managed to make an attribute using a list of strings and also extract the original list of strings from the attribute. I do wish this process was much easier. Please share if you can think of easier methods.

att_list = ['hello','bye']
h5_f.attrs['multi_strings'] = np.array(att_list, dtype='S')
h5_f.attrs['multi_strings'][0] == 'hello'
Out[66]: False
h5_f.attrs['multi_strings'][0] == np.string_('hello')
Out[67]: True
h5_f.attrs['multi_strings'][0] == b'hello'
Out[68]: True
h5_f.attrs['multi_strings'].dtype == 'str'
Out[69]: False
h5_f.attrs['multi_strings']
Out[70]: 
array([b'hello', b'bye'], 
      dtype='|S5')
cleaned_list = [str(x, 'utf-8') for x in h5_f.attrs['multi_strings']]
cleaned_list == att_list
Out[111]: True

Thanks!

[AminHP]

Thanks to @ssomnath

It would be Python v2 and v3 compatible by changing str(x, 'utf-8') to x.decode('utf-8').

[takluyver]

I think #1032 will fix this.