Pandas: Can't import h5py with pandas

Created on 12 Feb 2018  Â·  17Comments  Â·  Source: pandas-dev/pandas

Here's what I'm currently doing:

import h5py as h5
import pandas as pd

def init_stores():
    """Check if h5 exists, create one if it doesn't"""
    for store_path in store_paths:
        try:
            with h5.File(store_path, 'r'):
                pass
        except OSError:
            with h5.File(store_path, 'w'):
                logger.info("Created h5 file: {}".format(store_path))
                pass

def update_hdf():
    with pd.HDFStore(store_path) as store:
        print("In the update function")
        exit()

And I'm getting this error:

Traceback (most recent call last):
  File "C:\venv\lib\site-packages\pandas\io\pytables.py", line 445, in __init__
    import tables  # noqa
  File "C:\venv\lib\site-packages\tables\__init__.py", line 90, in <module>
    from .utilsextension import (
ImportError: DLL load failed: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/.../hdf_stuff.py", line 93, in <module>
    main()
  File "C:/.../hdf_stuff.py", line 89, in main
    update_hdf()
  File "C:/.../hdf_stuff.py", line 39, in update_hdf
    with pd.HDFStore(store_path) as store:
  File "C:\venv\lib\site-packages\pandas\io\pytables.py", line 448, in __init__
    'importing'.format(ex=str(ex)))
ImportError: HDFStore requires PyTables, "DLL load failed: The specified procedure could not be found." problem importing

If I comment out the h5py import, it works.

IO HDF5 Needs Info

Most helpful comment

I managed to get pytables and h5py working together on windows10 by downgrading h5py to version 2.7.0.
pip uninstall h5py
pip install h5py=2.7.0

All 17 comments

Can you fill in your #stuff with what actually causes the failure? If I just be pass there, I don't get an exception.

Can you also include your version info + your h5py and HDF5 versions?

@TomAugspurger all I have is a print statement (edited) and it fails.

Versions:
python 3.6
pandas 0.22.0
h5py 2.7.1

I'm not sure how to see the HDF5 version. I've only used what comes with pandas.

OK, your edited version does not fail for me, so it's presumable an environment thing.

What version of pytables?

On mac / linux h5cc -showconfig will print out the HDF5 version. Pandas doesn't bundle HDF5. How'd you install pytables?

pip show tables
Version: 3.4.2

Installed with pip I believe

OK, I'm not sure if / how pytables or h5py wheels bundle HDF5.

On Mon, Feb 12, 2018 at 4:54 PM, Brendan Martin notifications@github.com
wrote:

pip show tables
Version: 3.4.2

Installed with pip I believe

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/19666#issuecomment-365091390,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIn3FxZe5e9N9NxfYTmQJnn_hIWcHks5tUMErgaJpZM4SCwgl
.

please read the docs
HDFStore only works with pytables and has no support with h5py at all

you cannot use h5 files at all - maybe we need a bug warning in the docs - have seen a couple of issues like this in last few months

The issue isn't about using h5py with HDFStore, it's about some kind of
process-level conflict between pytables (plus or minus something pandas
does) and something h5py does on import.

On Mon, Feb 12, 2018 at 5:22 PM, Jeff Reback notifications@github.com
wrote:

you cannot use h5 files at all - maybe we need a bug warning in the docs -
have seen a couple of issues like this in last few months

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/19666#issuecomment-365097615,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIkJ0P02AGp-fyBnc8il4W13XjO3vks5tUMfIgaJpZM4SCwgl
.

this was an issue ages ago, but IIRC has been resolved in newer builds of hdf5. In any event this is not a pandas issue, rather an install / conda / pip one.

Yeah, I vaguely remember that too, and the fact that I can't reproduce it
points pretty strongly at an environment issue.

On Mon, Feb 12, 2018 at 6:16 PM, Jeff Reback notifications@github.com
wrote:

this was an issue ages ago, but IIRC has been resolved in newer builds of
hdf5. In any event this is not a pandas issue, rather an install / conda /
pip one.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/19666#issuecomment-365107952,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHItjM3x049K4X_ICGyhOEFJ7q4ALbks5tUNRdgaJpZM4SCwgl
.

@TomAugspurger @jreback thanks for the help. Should I try to reinstall or just use pytables, or is there another solution? I'm not partial to using h5py since this is the first use in my project.

You might try conda. I know there was an issue with conda packages in the
past but I believe they were fixed.

Otherwise, you may need to restructure your program to avoid importing h5py
and pandas in the same process.
Let us know if you find anything out.

On Tue, Feb 13, 2018 at 9:01 AM, Brendan Martin notifications@github.com
wrote:

@TomAugspurger https://github.com/tomaugspurger @jreback
https://github.com/jreback thanks for the help. Should I try to
reinstall or just use pytables, or is there another solution? I'm not
partial to using h5py since this is the first use in my project.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pandas-dev/pandas/issues/19666#issuecomment-365292645,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIsWglMlEBs8OJkPXyLPJ3kxGv6Utks5tUaPRgaJpZM4SCwgl
.

@TomAugspurger I'm just using pytables open_file instead. Thanks for the help.

I managed to get pytables and h5py working together on windows10 by downgrading h5py to version 2.7.0.
pip uninstall h5py
pip install h5py=2.7.0

I suddenly started facing this issue. I did an uninstall followed by an install, and it fix the problem.
pip uninstall h5py
pip install h5py

The problem is with h5py packaged with Anaconda. I started with a fresh installation and still get the same error. Shame on Anaconda for not fixing bugs for months!

@nirvana1122 it's most likely not an issue with the conda package. If you have an example showing otherwise I'm sure the Anaconda team would appreciate a report.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

nathanielatom picture nathanielatom  Â·  3Comments

ericdf picture ericdf  Â·  3Comments

Ashutosh-Srivastav picture Ashutosh-Srivastav  Â·  3Comments

marcelnem picture marcelnem  Â·  3Comments

andreas-thomik picture andreas-thomik  Â·  3Comments