H5py: Overwrite of existing file not flushed to disk on close

[mjwillson]

We've seen some bugs where writes that overwrite an existing file aren't (immediately or synchronously at least) being flushed to disk when the File object is closed, either via .close or __exit__ from the context manager.

Is .close guaranteed to flush? I noticed that you do have a separate .flush method, and that .close doesn't explicitly call it.

[mminervini]

Hi @mjwillson,

I was wondering if you found an answer to your question (it would appear so, as you closed your own issue yourself not long after you opened it).

I am experiencing errors (KeyError: 'Unable to open object (bad object header version number)') when reading 2-3 GB HDF5 files that are created within a context manager.
When the program terminates gracefully, the HDF5 files are just fine, whereas when the program terminates with an exception being raised, I obtain the invalid h5 files (even though the exception is well past the context manager that creates the h5 file). So, I suspect it may have something to do with close() not flushing and an exception preventing the delayed flushing from ever happening.
Perhaps, flush() should be invoked before exiting the context manager, when writing mid-large files that are expected to survive the program, no matter how badly it terminates.

I hope you could shed some light based on you experience! :)

[mjwillson]

I'm afraid I don't remember now exactly why I closed this but I think I may have tracked the bug down to something other than h5py. I think it might have been that some other library (theano perhaps?) was causing the python interpreter to hang or die in a nasty way when an exception occurred. It would be good to confirm that h4py does flush on close though.

[aragilar]

h5py does not specify whether the file is flushed on close. HDF5 (assuming the file driver doesn't do anything odd), closes the file and flushes it when the are no more references to the file or its members. Due to how H5Fclose interacts with MPI (https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-Close), and the fact that it doesn't actually close the file in the case there are more references, means h5py cannot call it, instead it decrements the number of references to the file, which means it will be flushed and closed at some point.

The safest thing to do would be to call flush as suggested (we should determine whether calling flush before closing the file should be added to FileID.close).

[mminervini]

Actually, I'm using Keras (version 2.1.2) and TensorFlow (version 1.4.1) with Python 3.5.3 and h5py 2.7.1. However, upon further tests I realized that even when the program terminates without any exception I still get seemingly corrupt h5 files (when I try to read them, I get a KeyError: 'Unable to open object (bad object header version number)' exception). So it could be that TensorFlow/Theano are not the culprit in this case.
I found older issues related to overwriting an existing h5 file that resulted in similar errors (#121), but deleting previous files did not solve the problem.
I also added a flush() before exiting the context manager, but this still didn't solve the problem.

Eventually, I figured out that the shape of the tensors saved as datasets is making a difference...
For example, if I include datasets with dimensions such as (784, 1248, 768, 3), the resulting h5 file is good, regardless of overwriting or flushing. On the other hand, if I reshape the same original data with dimensions such as (392, 1248, 1504, 3), obtaining an h5 file of approximately the same size as before, when I try to read it I get the aforementioned error.

I couldn't find in the documentation any specifications related to limits on the dimensions of the datasets. Are there any known issues with respect to this?

[Chrisebell24]

I had issues with this error when putting h5py.File(fp, libver='latest') when creating the file. Once I switched it to not include the libver, it worked for me just fine. I also only had issues when trying to use a compression filter for gzip or lzf.