H5py: How to store multidimensional variable length array with h5py ?

[linxd5]

I have two item, the first item contains 2 pictures, the second item contains 3 pictures, each picture is encoded with 200 dim vector. So the first item is 2x200, the second item is 3x200. I don't want to mixture this two item using numpy.concatenate because it may lose information about which picture belongs to which item. So I choose to use h5py.special_dtype, below is my code:

import numpy as np
import h5py

data = np.array([np.random.randn(2, 200), np.random.randn(3, 200)])

dt = h5py.special_dtype(vlen=np.float32)
file = h5py.File('temp.h5','w')
file.create_dataset('data', data = data, dtype=dt)
file.close()

f_read = h5py.File('temp.h5', 'r')
print f_read['data'][:]

The result shows that h5py only store the first dimension, regradless of the second dimension. I can't find out what's wrong with my code ?

---

• Operating System : Ubuntu 14.04
• Python version : 2.7
• Where Python was acquired : system python
• h5py : 2.7.0
• HDF5 version : 1.8.18

[jh70zape]

I ran into the same problem. seems only 1D array can be written in .hdf5

[nilinykh]

Same issue here. Only 1D arrays of the arbitrary length can be stored in .hdf5 format, while what I want is to store 2D arrays of arbitrary length...

[toshas]

quick hack - create a separate dataset in the same file to store original tensor shape

[tacaswell]

from https://support.hdfgroup.org/HDF5/doc/UG/HDF5_Users_Guide-Responsive%20HTML5/index.html#t=HDF5_Users_Guide%2FDatatypes%2FHDF5_Datatypes.htm%23TOC_6_5_2_Definition_ofbc-14&rhtocid=6.3.0_2 (scroll down a bit)

6.5.2.2.3. Variable-length Datatypes

A variable-length (VL) datatype is a one-dimensional sequence of a datatype which are not fixed in length from one dataset location to another. In other words, each data element may have a different number of members. Variable-length datatypes cannot be divided, the entire data element must be transferred.

VL datatypes are useful to the scientific community in many different ways, possibly including:

• Ragged arrays: Multi-dimensional ragged arrays can be implemented with the last (fastest chang­ing) dimension being ragged by using a VL datatype as the type of the element stored.
• Fractal arrays: A nested VL datatype can be used to implement ragged arrays of ragged arrays, to whatever nesting depth is required for the user.
• Polygon lists: A common storage requirement is to efficiently store arrays of polygons with differ­ent numbers of vertices. A VL datatype can be used to efficiently and succinctly describe an array of polygons with different numbers of vertices.
• Character strings: Perhaps the most common use of VL datatypes will be to store C-like VL charac­ter strings in dataset elements or as attributes of objects.
• Indices (for example, of objects within a file): An array of VL object references could be used as an index to all the objects in a file which contain a particular sequence of dataset values.
• Object Tracking: An array of VL dataset region references can be used as a method of tracking objects or features appearing in a sequence of datasets.

so creating "fractal" data sets is possible, but harder to specify and for this use case, swapping the "ragged" axes to be last may work?

[kuangluvasuka]

One sidestep to store the variable-length matrix is to break down the matrix into n-row arrays.
This is what I did:

import h5py
import numpy as np
float32_t = h5py.special_dtype(vlen=np.dtype('float32'))
evolutionary_ = f.create_dataset('evolutionary', shape=(1, 3,), maxshape=(None, 3,), dtype=float32_t)
a = np.random.randn(1, 3, 4)
b = np.random.randn(1, 3, 6)

evolutionary_[0] = a
print('evo[0] is \n', evolutionary_.value)

evolutionary_.resize(3, axis=0)
evolutionary_[1] = b
print('evo[0,1,2] is\n', evolutionary_.value)

output:

evo[0] is
[[array([-0.3807321,  1.4716047], dtype=float32)
  array([-0.36868876, -0.4454193 ], dtype=float32)
  array([-0.24017805,  1.5490232 ], dtype=float32)]]

evo[0,1,2] is
[[array([-0.3807321,  1.4716047], dtype=float32)
  array([-0.36868876, -0.4454193 ], dtype=float32)
  array([-0.24017805,  1.5490232 ], dtype=float32)]
 [array([ 0.08650446,  0.561802  ,  1.607487  ,  1.2775166 , -0.9047034 ,
       -0.8518197 ], dtype=float32)
  array([-0.49097598, -0.6513362 , -0.74338317,  2.0841973 , -0.13526744,
       -0.46241984], dtype=float32)
  array([ 0.15678692, -2.0850842 , -0.9444737 ,  0.8307225 , -0.14368942,
        0.17739072], dtype=float32)]
 [array([], dtype=float32) array([], dtype=float32)
  array([], dtype=float32)]]

so matrix now is stored as array of arrays, and using np.stack(evolutionary_[0], axis=0) will revert it to matrix.

[hafiz031]

How did you know the hdf5 file is only storing the first dimension? Did you use "HDFView" software to open the file? If it is than I will say your file may be okay. Because this software cannot show multidimensional array. So the problem is with the software. I also fell into the same confusion. I will highly recommend to calculate the desired file size yourself and check if your file size is the same. If it is than probably your file is okay. See the comments to the question: https://stackoverflow.com/questions/52102616/images-saved-as-hdf5-arent-colored