H5py: IOError: Unable to create file (unable to open file: name = 'feat.h5', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)

[wqn628]

hello，today i met a strange problem when i use the python script to read the h5py file.
There have been a python script which is reading the h5py file, if another python script starts reading the same h5py file, the error occurs as follows:

IOError: Unable to create file (unable to open file: name = './MTL_AM/jp0703_kr0829/src_0.4/jp0703_kr0829_train_sort10_0sp4_inputs_td0_ts100.h5', errno = 17, error message = 'File exists', flags = 15, o_flags = c2)
but if i copy this h5py and read the copy-file, there is no problem. However, this is a large file and i don't want to copy it. Are there any good solutions？
and here is my python environment
[@nmyjs_16_40 ~]$ python -c 'import h5py; print(h5py.version.info)' /search/speech/wangqingnan/Anaconda/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters

Summary of the h5py configuration

h5py 2.7.1
HDF5 1.10.2
Python 2.7.15 |Anaconda, Inc.| (default, May 1 2018, 23:32:55)
[GCC 7.2.0]
sys.platform linux2
sys.maxsize 9223372036854775807
numpy 1.14.3`
any help would be appreciated

[jerasman]

Probably the file was already opened in another terminal.

[wqn628]

@jerasman
I also think so, why can the same file not opened by several processes?

[1313e]

@wqn628 Because that requires the hdf5-library to be built with parallel extensions. You can open an HDF5-file for read-only in multiple processes, but you cannot open it for read/write in more than one process unless the library has been built with parallel enabled.
The reason for this is because HDF5-files write to the file the moment you close them, not the moment you tell them to write. Therefore, in serial, corruption would be very likely.

The best way to deal with this is by simply opening the HDF5-file for read-only in both Python scripts. That should work perfectly fine.

[tacaswell]

Closing as this is not an h5py bug, but a limitation of hdf5 as @1313e explained.

You may also be interested in SWMR (if you only need to write from one process, but read from many).

[mzechmeister]

I got this error after interruption of a program. Now, I cannot rerun the program, because the file was not closed properly. The file is still open as a zombie somewhere.

How can close the file???

PS: The file seems not to be listed by lsof!?
PS2: f = open('zombie.hdf'); f.closed() with python is possible; but the hdf error remains. Last time I had to delete the file or restart the computer.