H5py: OSError raised for errors unrelated to system function calls

[miccoli]

Calling h5py.File on an existing file not in HDF5 format raises OSError:

>>> import h5py
>>> h5py.__version__
'2.9.0'
>>> h5py.File("/etc/passwd", "r")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)
  File ".../lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

This is semantically wrong: I expect OSError to be raised when a system call fails. From the OSError python docs (emphasis added by me):

This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

[takluyver]

I'm looking at the standard library for what other modules do when a file is not of an expected type

• zipfile declares its own BadZipFile exception type, inheriting from Exception
• tarfile declares its own TarError (and subclasses), inheriting from Exception
• gzip uses OSError for invalid files
• bz2 uses OSError
• dbm declares its own error type, inheriting from Exception
• json declares its own JSONDecodeError, inheriting from ValueError
• pickle uses TypeError
• plistlib uses TypeError
• pstats uses ValueError
• sqlite3 has its own Error (and subclasses), inheriting from Exception
• sunau has its own Error, inheriting from Exception
• wave has its own Error, inheriting from Exception

So it seems like the common pattern is to define your own exception type, with exceptions for the pure compression formats (which are taking liberties with OSError), and some modules following the load/dump API.

[miccoli]

@takluyver thanks for the excellent review of the standard library: at least h5py is not alone in (ab)using OSError! (I was completely unaware of this fact). An interesting read is also PEP 3151 -- Reworking the OS and IO exception hierarchy where the rationale of the exception hierarchy implemented in Python 3.3+ is explained.

Let me only add that, as a h5py user/python programmer I would prefer to write:

try:
    f = h5py.File('myfile.hdf5', 'r')
except h5py.BadHDF5File as exc:
    # handle HDF5 format errors
except OSError as exc:
    # handle os related errors

instead of

try:
    f = h5py.File('myfile.hdf5', 'r')
except OSError as exc:
    # guesswork to understand what went wrong

(Of course names and namespaces are only tentative, just to give an idea of what I think would be a sensible choice.)

[takluyver]

My hope is that if HDF5 can expose the POSIX errno, we can raise appropriate OSErrors (or subclasses like FileNotFoundError), and then maybe have an HDFError and possibly some subclasses for everything that doesn't come with an errno. I suspect that the attempt to translate HDF5 errors to canonical Python errors is probably pretty arbitrary.

However, I haven't seen any movement from HDF5 towards making the errnos part of the API. I'd be reluctant to redesign the exception handling without that.