H5py: Save jpeg images in h5py

[markusnagel]

Hello,

I currently deal with image datasets of about 1 million images. When saving them as numpy array (with dtype uint8) to h5py this would result in a dataset file of over 1TB which is very suboptimal.
My solution to this is to save the images as jpeg (that is there original format) which should reduce the dataset size to about 100GB. I want to use h5py since I then can use fuel (a dataset manage) to easily preprocess and iterate over the data.

So my only problem is now how to get the jpeg images into h5py. Reading the binary is easy:

fin = open(filename, 'rb')
binary_data = fin.read()

But I can not manage the save it in h5py. If I do:

f = h5py.File('foo.hdf5')
dt = h5py.special_dtype(vlen='str')
dset = f.create_dataset('binary', (100, ), dtype=dt)
dset[0] = binary_data

I get the following error

tid = h5t.py_create(dtype, logical=1)
...
ValueError: Size must be positive (Size must be positive)

If I change line 3 to dt = h5py.special_dtype(vlen=bytes) then I get ValueError: VLEN strings do not support embedded NULLs like in http://docs.h5py.org/en/latest/strings.html#variable-length-ascii. But avoiding nulls is not possible in a binary string from jpeg images.

Also changing the last line to dset[0] = np.void(binary_data) seems not to help (was suggested here: http://docs.h5py.org/en/latest/strings.html#how-to-store-raw-binary-data).

I in general I tried lots/all different things and combinations from these tutorials:
http://docs.h5py.org/en/latest/special.html
http://docs.h5py.org/en/latest/strings.html

It would be great if anyone could hint me to the correct solution.

Thanks a lot,
Markus

[tacaswell]

Have you looked at any of the compression options built into hdf5 (http://docs.h5py.org/en/latest/high/dataset.html#filter-pipeline)?

Also see https://www.hdfgroup.org/services/filters.html for other available filters.

[felix1028]

Markus,
Did you ever find a way to save the JPG file to your HDF5 file, or just used compression to bring down your file size? If you found a way to attach the files, I would appreciate any hints you may have.

Thanks,
Mariana

[markusnagel]

Yes I found a way to save them. The solution is to save the binary data as a numpy array of 'uint8', there it does not have the problem with the NULL bytes (I assume there they save the shape and do not use NULL as the end of the string).
So the working code is now:

import numpy as np
import h5py

filename = 'test.jpg'
fin = open(filename, 'rb')
binary_data = fin.read()

f = h5py.File('foo.hdf5')
dt = h5py.special_dtype(vlen=np.dtype('uint8'))
dset = f.create_dataset('binary_data', (100, ), dtype=dt)

# Save data string converted as a np array
dset[0] = np.fromstring(binary_data, dtype='uint8')

[felix1028]

Markus,
Thank you very much for the example. I now understand what you were trying to do -- read the image file and save it as an array to your hdf5 file. Thank you for the code, I learned something new!

Unfortunately my problem is a little different, I was asked to save the image itself to the hdf5 file (as a truecolor image in an h5py image class). So I'll have to keep digging :)
Thanks again,
Mariana

[meder411]

Markus,

Out of curiosity, how did you read the binary data from the dataset and convert back to image form? I'm trying to use scikit image or PIL for this, but there doesn't seem to be a good method.

Marc

[meder411]

Nevermind, I got it. For anyone else curious:

from PIL import Image
import io

img = Image.open(io.BytesIO(dset[0])) # This took ~30ms for a hi-res 3-channel image (~2000x2000)
img.show()

[raguiar2]

I created a script for this

https://github.com/raguiar2/jpg_to_h5/tree/master

[briland]

Thank you @markusnagel for the really useful explanation. Myself, I made use of numpy ndarrays to solve this issue. I am attaching a sample code below:

import h5py
import numpy as np

from PIL import Image
import io

file_name = 'sample_image.jpg'
hdf5_file = 'samples.hdf5'

with open(file_name, 'rb') as img_f:
    image_file = img_f.read()

print(type(image_file)) # <class 'bytes'>  

img_np_array = np.asarray(image_file)
print(type(img_np_array)) # <class 'numpy.ndarray'>

# store to hdf5 file
f = h5py.File(hdf5_file)
dset = f.create_dataset('binary_data', data=img_np_array)
f.close()

To read back the result from the hdf5 file:

f = h5py.File(hdf5_file)

dset_read = f.get('binary_data')
dset_read_np = np.array(dset_read)
img_res = Image.open(io.BytesIO(dset_read_np))
img_res.show()

f.close()