H5py: RuntimeError when writing to /dev/null

[kalvdans]

This error occur in my unit tests, where I write data to /dev/null to throw it away. It only happens when writing empty datasets. Testcase:

import h5py

with h5py.File("/dev/null", "w") as f:
    f.create_dataset("foo", data=[])
    f.create_dataset("bar", data=[])

Output:

Traceback (most recent call last):
  File "h5py-dev-null.py", line 5, in <module>
    f.create_dataset("bar", data=[])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/chn/env/lib/python3.8/site-packages/h5py/_hl/files.py", line 475, in __exit__
    self.close()
  File "/home/chn/env/lib/python3.8/site-packages/h5py/_hl/files.py", line 457, in close
    h5i.dec_ref(id_)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5i.pyx", line 153, in h5py.h5i.dec_ref
RuntimeError: Can't decrement id ref count (unable to extend file properly, errno = 22, error message = 'Invalid argument')
Segmentation fault

The expected result is either to work correctly or fail with OSError or equivalent. Versions:

Ubuntu 20.04.1 LTS
h5py    3.1.0
HDF5    1.12.0
Python  3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.18.3
cython (built with) 0.29.21
numpy (built against) 1.17.5
HDF5 (built against) 1.12.0

[aragilar]

Does an equivalent version in C work? Otherwise the segfault may be coming from the underlying HDF5 library.

[takluyver]

I don't think the equivalent in C would work - HDF5 expects to be able to resize a file with ftruncate(), and that errors on /dev/null. I don't think it makes sense to special-case this in h5py - not least because there are a variety of files which HDF5 probably won't like: other special things in /dev, named pipes, unix sockets, etc.

If you want to test creating an HDF5 file without touching the disk, use the 'core' file driver with backing_store=False. Creating temp files with the default driver may also have a similar effect, either if the system is set up with tmpfs, or if the file is deleted again before the OS actually writes its buffers to disk.

[kalvdans]

Even if it needs ftruncate it should not crash with a Segmentation Fault.

[takluyver]

It shouldn't, but I've seen before that HDF5 doesn't handle errors when closing a file very well.

[kalvdans]

Did you spend time to report those to the HDF group? (much easier now when they have a public issue tracker)

[tacaswell]

With enough print statements I localized this to https://github.com/h5py/h5py/blob/fd708874ac59dd03ccd4cddc9911c37ef9c2fe80/h5py/_objects.pyx#L216 where we are passing a pointer back into libhdf5. My guess is that something is going wrong during opening, the file is not fully initialized at the c-level, something is not allocated, and then in close it blows up when trying to access that part of the structure.

While segfaults are quite bad, I am not convinced that trying to use /dev/null as the file is part of normal operations and the mitigation at the user level straight forward (do not do that). That said, this should be reported upstream (either they need to fail sooner on opening the file or more gracefully on close!). As you found this can you please take care of that @kalvdans ?

---

The changes in files.py move the segfault from Python tear down to the __exit__ on the context manager.

import faulthandler
import h5py
import time
import gc
import sys

faulthandler.enable()

try:
    with h5py.File("/dev/null", "w") as f:
        f.create_dataset("foo", data=[])
        f.create_dataset("bar", data=[])
except Exception:
    print('here')
    for j in range(100):
        gc.collect(2)
        n = j*j
        print(j)
    time.sleep(1)
    print('still here', flush=True)

✔ 17:36:57 $ git diff
diff --git a/h5py/_hl/files.py b/h5py/_hl/files.py
index 6fdb767e..834cd7e8 100644
--- a/h5py/_hl/files.py
+++ b/h5py/_hl/files.py
@@ -466,14 +466,23 @@ class File(Group):
                 id_list = [x for x in id_list if h5i.get_file_id(x).id == self.id.id]
                 file_list = [x for x in file_list if h5i.get_file_id(x).id == self.id.id]

+                errors = []
                 for id_ in id_list:
                     while id_.valid:
-                        h5i.dec_ref(id_)
+                        try:
+                            h5i.dec_ref(id_)
+                        except Exception as e:
+                            errors.append(e)
+                            break

                 for id_ in file_list:
+                    print(f'{id_=}', flush=True)
                     while id_.valid:
-                        h5i.dec_ref(id_)
-
+                        try:
+                            h5i.dec_ref(id_)
+                        except Exception as e:
+                            errors.append(e)
+                            break
                 self.id.close()
                 _objects.nonlocal_close()

diff --git a/h5py/_objects.pyx b/h5py/_objects.pyx
index de428e09..63fd714a 100644
--- a/h5py/_objects.pyx
+++ b/h5py/_objects.pyx
@@ -210,16 +210,22 @@ cdef class ObjectID:
         """ Manually close this object. """

         with _phil:
-            IF DEBUG_ID:
-                print("CLOSE - %d HDF5 id %d" % (self._pyid, self.id))
+            print('a')
+            print("CLOSE - %d HDF5 id %d" % (self._pyid, self.id))
+            print('b')
+            print('c')
+            print('d')
             if is_h5py_obj_valid(self) and (not self.locked):
+                print('e')
                 if H5Idec_ref(self.id) < 0:
                     warnings.warn(
                         "Reference counting issue with HDF5 id {}".format(
                             self.id
                         )
                     )
+            print('f')
             self.id = 0
+            print('g')

     def close(self):
         """ Close this identifier. """

✔ 17:52:50 $ python /tmp/womp-womp2.py
id_=<h5py.h5f.FileID object at 0x7f064f0bcb80>
a
CLOSE - 139665461695328 HDF5 id 72057594037927936
b
c
d
e
Fatal Python error: Segmentation fault

Current thread 0x00007f0666941740 (most recent call first):
  File "/home/tcaswell/source/bnl/h5py/h5py/_hl/files.py", line 486 in close
  File "/home/tcaswell/source/bnl/h5py/h5py/_hl/files.py", line 502 in __exit__
  File "/tmp/womp-womp2.py", line 12 in <module>
Segmentation fault (core dumped)

[kalvdans]

That said, this should be reported upstream (either they need to fail sooner on opening the file or more gracefully on close!). As you found this can you please take care of that @kalvdans ?

Sure! Thanks a ton for pinpointing the crash - should not be hard to make a C-only reproducer now. I will post a link here when it is done.

[tacaswell]

Awesome, thanks @kalvdans !