Openpmd-api: Splash HDF5 files with Zero Particles

[PrometheusPi]

Describe the bug
The string matching routine suggested in the documentation via "data_%T.h5" fails for mixed output wich filter strings as commonly used in PIConGPU.

To Reproduce
Assume you have data of the form simData_100.h5 and additionally filtered data in the form simData_filter_100.h5 in the same directory. While the suggested placeholder %T should select iterations only, it is confused with the additional string beta_ and throws the following run time error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-36-4ed3e1a89505> in <module>
      2 series = api.Series(
      3     path + "simData_%T.h5",
----> 4     api.Access_Type.read_only)

RuntimeError: Unsupported dataset class

Python:

import openpmd_api
path = "/scratch/ws/<...>/runs_LWFA_PWFA/069_<...>/simOutput/h5/"
series = api.Series(
    path + "simData_%T.h5", 
    api.Access_Type.read_only)

Expected behavior
Intuitively a expected %T to only match numbers (with and without leading zeros) and thus ignore additional letters (as used by filter names in PIConGPU).

Software Environment

• version of openPMD-api: 0.10.0-alpha
• installed openPMD-api via: conda-forge
• operating system: Red Hat Enterprise Linux Server release 7.4 (Maipo)
• machine: taurus ZIH TU Dresden
• name and version of Python implementation: Python 3.6.9 | Anaconda, Inc.
• version of HDF5: 1.10.5
• version of ADIOS1: 1.13.1
• name and version of MPI: no MPI used

Additional context
This effects use of all filtered openPMD data in PIConGPU.

[ax3l]

Thanks for the detailed report!
%T or %05T et al. are only placeholders for the iteration.

If your file reads simData_filter_100.h5 than your regex should read
path + "simData_filter_%T.h5".

Does that solve it?

[PrometheusPi]

No, the issue is not with reading the series "simData_filter_%T.h5" (that works fine) but with reading the "simData_%T.h5" series, while the other series (with beta) resides in the same directory.

[ax3l]

Interesting, I do not see an issue with the regex, besides that the . should read \.:
https://regex101.com/r/jwPEm7/2

Could it be possible we are matching other digit components of our path? Update: urgh, I might accidentally include your whole path you pass as prefix in the regex - without escaping it.
I'll have to fix the pattern matching.

As a quick work-around to keep you going, there are two ways you can proceed:

• simple and effective: cd into path first: import os; os.chdir(path) and only pass the filename to Series
• hacky: escape the /s in your path (and + or . if you have them) with one or two \\ to sanitize the regex :D

[PrometheusPi]

@ax3l Thanks for the hacky workaround - it is not an urgent issue - I just discovered it during a test.

[ax3l]

Just in order to test this properly: can you please provide me with the full path that you used? If it contains private info you don't want to post publicly, feel free to e-mail instead.

[PrometheusPi]

@ax3l You should have received an email.

[ax3l]

I provided a PR in #633 that should address your issue :)
Update: hm, still working on a reproducer for CI...

[ax3l]

Things that make me wonder in this report:

• is "beta" the filter you mention?
• did you move the files into separate directories and confirmed that it works then?
• did you try the cd workaround?

The latter two points are likely not the case since this error is not at all stemming from the iteration parsing but from the HDF5 dataset detection: https://github.com/openPMD/openPMD-api/blob/0.10.0-alpha/src/IO/HDF5/HDF5IOHandler.cpp#L476

Can you please provide recursive h5ls dumps of an example file? Does it contain maybe some unusual particle attributes or derived fields or so?

[PrometheusPi]

@ax3l Thank you for working on a fix.

Regarding your questions:

• is "beta" the filter you mention?
  Yes, beta is the name of the filter. Please excuse the confusion.
• did you move the files into separate directories and confirmed that it works then?
  No - I will test this as soon as possible.
• did you try the cd workaround?
  Yes - it does not solv the issue - I still get the same error.

FYI: While testing in ipython (and not jupyter) directly, I got the following warnings various times from both strings "simData_%T.h5" and "simData_beta_%T.h5":

Skipping non-standard attribute _global_size (Compound attribute type not supported)
Skipping non-standard attribute _global_start (Compound attribute type not supported)
Skipping non-standard attribute _size (Compound attribute type not supported)
Skipping non-standard attribute _start (Compound attribute type not supported)

h5ls -r simData_10000.h5 output:

/                        Group
/data                    Group
/data/10000              Group
/data/10000/fields       Group
/data/10000/fields/B     Group
/data/10000/fields/B/x   Dataset {768, 4032, 768}
/data/10000/fields/B/y   Dataset {768, 4032, 768}
/data/10000/fields/B/z   Dataset {768, 4032, 768}
/data/10000/fields/E     Group
/data/10000/fields/E/x   Dataset {768, 4032, 768}
/data/10000/fields/E/y   Dataset {768, 4032, 768}
/data/10000/fields/E/z   Dataset {768, 4032, 768}
/data/10000/fields/H_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/H_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/H_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/HeI_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/HeI_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/HeI_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/e_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/e_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/e_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/ef_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/ef_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/ef_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/en_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/en_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/en_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/ep1_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/ep1_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/ep1_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/ep2He_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/ep2He_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/ep2He_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/ep2_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/ep2_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/ep2_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/fields/n_chargeDensity Dataset {768, 4032, 768}
/data/10000/fields/n_energyDensity Dataset {768, 4032, 768}
/data/10000/fields/n_particleMomentumComponent Dataset {768, 4032, 768}
/data/10000/particles    Group
/data/10000/particles/H  Group
/data/10000/particles/H/boundElectrons Dataset {NULL}
/data/10000/particles/H/mass Group
/data/10000/particles/H/momentum Group
/data/10000/particles/H/momentum/x Dataset {NULL}
/data/10000/particles/H/momentum/y Dataset {NULL}
/data/10000/particles/H/momentum/z Dataset {NULL}
/data/10000/particles/H/particlePatches Group
/data/10000/particles/H/particlePatches/extent Group
/data/10000/particles/H/particlePatches/extent/x Dataset {192}
/data/10000/particles/H/particlePatches/extent/y Dataset {192}
/data/10000/particles/H/particlePatches/extent/z Dataset {192}
/data/10000/particles/H/particlePatches/numParticles Dataset {192}
/data/10000/particles/H/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/H/particlePatches/offset Group
/data/10000/particles/H/particlePatches/offset/x Dataset {192}
/data/10000/particles/H/particlePatches/offset/y Dataset {192}
/data/10000/particles/H/particlePatches/offset/z Dataset {192}
/data/10000/particles/H/position Group
/data/10000/particles/H/position/x Dataset {NULL}
/data/10000/particles/H/position/y Dataset {NULL}
/data/10000/particles/H/position/z Dataset {NULL}
/data/10000/particles/H/positionOffset Group
/data/10000/particles/H/positionOffset/x Dataset {NULL}
/data/10000/particles/H/positionOffset/y Dataset {NULL}
/data/10000/particles/H/positionOffset/z Dataset {NULL}
/data/10000/particles/H/weighting Dataset {NULL}
/data/10000/particles/HeI Group
/data/10000/particles/HeI/boundElectrons Dataset {NULL}
/data/10000/particles/HeI/mass Group
/data/10000/particles/HeI/momentum Group
/data/10000/particles/HeI/momentum/x Dataset {NULL}
/data/10000/particles/HeI/momentum/y Dataset {NULL}
/data/10000/particles/HeI/momentum/z Dataset {NULL}
/data/10000/particles/HeI/particlePatches Group
/data/10000/particles/HeI/particlePatches/extent Group
/data/10000/particles/HeI/particlePatches/extent/x Dataset {192}
/data/10000/particles/HeI/particlePatches/extent/y Dataset {192}
/data/10000/particles/HeI/particlePatches/extent/z Dataset {192}
/data/10000/particles/HeI/particlePatches/numParticles Dataset {192}
/data/10000/particles/HeI/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/HeI/particlePatches/offset Group
/data/10000/particles/HeI/particlePatches/offset/x Dataset {192}
/data/10000/particles/HeI/particlePatches/offset/y Dataset {192}
/data/10000/particles/HeI/particlePatches/offset/z Dataset {192}
/data/10000/particles/HeI/position Group
/data/10000/particles/HeI/position/x Dataset {NULL}
/data/10000/particles/HeI/position/y Dataset {NULL}
/data/10000/particles/HeI/position/z Dataset {NULL}
/data/10000/particles/HeI/positionOffset Group
/data/10000/particles/HeI/positionOffset/x Dataset {NULL}
/data/10000/particles/HeI/positionOffset/y Dataset {NULL}
/data/10000/particles/HeI/positionOffset/z Dataset {NULL}
/data/10000/particles/HeI/weighting Dataset {NULL}
/data/10000/particles/e  Group
/data/10000/particles/e/charge Group
/data/10000/particles/e/mass Group
/data/10000/particles/e/momentum Group
/data/10000/particles/e/momentum/x Dataset {263946130}
/data/10000/particles/e/momentum/y Dataset {263946130}
/data/10000/particles/e/momentum/z Dataset {263946130}
/data/10000/particles/e/particleId Dataset {263946130}
/data/10000/particles/e/particlePatches Group
/data/10000/particles/e/particlePatches/extent Group
/data/10000/particles/e/particlePatches/extent/x Dataset {192}
/data/10000/particles/e/particlePatches/extent/y Dataset {192}
/data/10000/particles/e/particlePatches/extent/z Dataset {192}
/data/10000/particles/e/particlePatches/numParticles Dataset {192}
/data/10000/particles/e/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/e/particlePatches/offset Group
/data/10000/particles/e/particlePatches/offset/x Dataset {192}
/data/10000/particles/e/particlePatches/offset/y Dataset {192}
/data/10000/particles/e/particlePatches/offset/z Dataset {192}
/data/10000/particles/e/position Group
/data/10000/particles/e/position/x Dataset {263946130}
/data/10000/particles/e/position/y Dataset {263946130}
/data/10000/particles/e/position/z Dataset {263946130}
/data/10000/particles/e/positionOffset Group
/data/10000/particles/e/positionOffset/x Dataset {263946130}
/data/10000/particles/e/positionOffset/y Dataset {263946130}
/data/10000/particles/e/positionOffset/z Dataset {263946130}
/data/10000/particles/e/weighting Dataset {263946130}
/data/10000/particles/ef Group
/data/10000/particles/ef/charge Group
/data/10000/particles/ef/mass Group
/data/10000/particles/ef/momentum Group
/data/10000/particles/ef/momentum/x Dataset {NULL}
/data/10000/particles/ef/momentum/y Dataset {NULL}
/data/10000/particles/ef/momentum/z Dataset {NULL}
/data/10000/particles/ef/particleId Dataset {NULL}
/data/10000/particles/ef/particlePatches Group
/data/10000/particles/ef/particlePatches/extent Group
/data/10000/particles/ef/particlePatches/extent/x Dataset {192}
/data/10000/particles/ef/particlePatches/extent/y Dataset {192}
/data/10000/particles/ef/particlePatches/extent/z Dataset {192}
/data/10000/particles/ef/particlePatches/numParticles Dataset {192}
/data/10000/particles/ef/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/ef/particlePatches/offset Group
/data/10000/particles/ef/particlePatches/offset/x Dataset {192}
/data/10000/particles/ef/particlePatches/offset/y Dataset {192}
/data/10000/particles/ef/particlePatches/offset/z Dataset {192}
/data/10000/particles/ef/position Group
/data/10000/particles/ef/position/x Dataset {NULL}
/data/10000/particles/ef/position/y Dataset {NULL}
/data/10000/particles/ef/position/z Dataset {NULL}
/data/10000/particles/ef/positionOffset Group
/data/10000/particles/ef/positionOffset/x Dataset {NULL}
/data/10000/particles/ef/positionOffset/y Dataset {NULL}
/data/10000/particles/ef/positionOffset/z Dataset {NULL}
/data/10000/particles/ef/weighting Dataset {NULL}
/data/10000/particles/en Group
/data/10000/particles/en/charge Group
/data/10000/particles/en/mass Group
/data/10000/particles/en/momentum Group
/data/10000/particles/en/momentum/x Dataset {1774}
/data/10000/particles/en/momentum/y Dataset {1774}
/data/10000/particles/en/momentum/z Dataset {1774}
/data/10000/particles/en/particleId Dataset {1774}
/data/10000/particles/en/particlePatches Group
/data/10000/particles/en/particlePatches/extent Group
/data/10000/particles/en/particlePatches/extent/x Dataset {192}
/data/10000/particles/en/particlePatches/extent/y Dataset {192}
/data/10000/particles/en/particlePatches/extent/z Dataset {192}
/data/10000/particles/en/particlePatches/numParticles Dataset {192}
/data/10000/particles/en/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/en/particlePatches/offset Group
/data/10000/particles/en/particlePatches/offset/x Dataset {192}
/data/10000/particles/en/particlePatches/offset/y Dataset {192}
/data/10000/particles/en/particlePatches/offset/z Dataset {192}
/data/10000/particles/en/position Group
/data/10000/particles/en/position/x Dataset {1774}
/data/10000/particles/en/position/y Dataset {1774}
/data/10000/particles/en/position/z Dataset {1774}
/data/10000/particles/en/positionOffset Group
/data/10000/particles/en/positionOffset/x Dataset {1774}
/data/10000/particles/en/positionOffset/y Dataset {1774}
/data/10000/particles/en/positionOffset/z Dataset {1774}
/data/10000/particles/en/weighting Dataset {1774}
/data/10000/particles/ep1 Group
/data/10000/particles/ep1/charge Group
/data/10000/particles/ep1/mass Group
/data/10000/particles/ep1/momentum Group
/data/10000/particles/ep1/momentum/x Dataset {NULL}
/data/10000/particles/ep1/momentum/y Dataset {NULL}
/data/10000/particles/ep1/momentum/z Dataset {NULL}
/data/10000/particles/ep1/particleId Dataset {NULL}
/data/10000/particles/ep1/particlePatches Group
/data/10000/particles/ep1/particlePatches/extent Group
/data/10000/particles/ep1/particlePatches/extent/x Dataset {192}
/data/10000/particles/ep1/particlePatches/extent/y Dataset {192}
/data/10000/particles/ep1/particlePatches/extent/z Dataset {192}
/data/10000/particles/ep1/particlePatches/numParticles Dataset {192}
/data/10000/particles/ep1/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/ep1/particlePatches/offset Group
/data/10000/particles/ep1/particlePatches/offset/x Dataset {192}
/data/10000/particles/ep1/particlePatches/offset/y Dataset {192}
/data/10000/particles/ep1/particlePatches/offset/z Dataset {192}
/data/10000/particles/ep1/position Group
/data/10000/particles/ep1/position/x Dataset {NULL}
/data/10000/particles/ep1/position/y Dataset {NULL}
/data/10000/particles/ep1/position/z Dataset {NULL}
/data/10000/particles/ep1/positionOffset Group
/data/10000/particles/ep1/positionOffset/x Dataset {NULL}
/data/10000/particles/ep1/positionOffset/y Dataset {NULL}
/data/10000/particles/ep1/positionOffset/z Dataset {NULL}
/data/10000/particles/ep1/weighting Dataset {NULL}
/data/10000/particles/ep2 Group
/data/10000/particles/ep2/charge Group
/data/10000/particles/ep2/mass Group
/data/10000/particles/ep2/momentum Group
/data/10000/particles/ep2/momentum/x Dataset {NULL}
/data/10000/particles/ep2/momentum/y Dataset {NULL}
/data/10000/particles/ep2/momentum/z Dataset {NULL}
/data/10000/particles/ep2/particleId Dataset {NULL}
/data/10000/particles/ep2/particlePatches Group
/data/10000/particles/ep2/particlePatches/extent Group
/data/10000/particles/ep2/particlePatches/extent/x Dataset {192}
/data/10000/particles/ep2/particlePatches/extent/y Dataset {192}
/data/10000/particles/ep2/particlePatches/extent/z Dataset {192}
/data/10000/particles/ep2/particlePatches/numParticles Dataset {192}
/data/10000/particles/ep2/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/ep2/particlePatches/offset Group
/data/10000/particles/ep2/particlePatches/offset/x Dataset {192}
/data/10000/particles/ep2/particlePatches/offset/y Dataset {192}
/data/10000/particles/ep2/particlePatches/offset/z Dataset {192}
/data/10000/particles/ep2/position Group
/data/10000/particles/ep2/position/x Dataset {NULL}
/data/10000/particles/ep2/position/y Dataset {NULL}
/data/10000/particles/ep2/position/z Dataset {NULL}
/data/10000/particles/ep2/positionOffset Group
/data/10000/particles/ep2/positionOffset/x Dataset {NULL}
/data/10000/particles/ep2/positionOffset/y Dataset {NULL}
/data/10000/particles/ep2/positionOffset/z Dataset {NULL}
/data/10000/particles/ep2/weighting Dataset {NULL}
/data/10000/particles/ep2He Group
/data/10000/particles/ep2He/charge Group
/data/10000/particles/ep2He/mass Group
/data/10000/particles/ep2He/momentum Group
/data/10000/particles/ep2He/momentum/x Dataset {NULL}
/data/10000/particles/ep2He/momentum/y Dataset {NULL}
/data/10000/particles/ep2He/momentum/z Dataset {NULL}
/data/10000/particles/ep2He/particleId Dataset {NULL}
/data/10000/particles/ep2He/particlePatches Group
/data/10000/particles/ep2He/particlePatches/extent Group
/data/10000/particles/ep2He/particlePatches/extent/x Dataset {192}
/data/10000/particles/ep2He/particlePatches/extent/y Dataset {192}
/data/10000/particles/ep2He/particlePatches/extent/z Dataset {192}
/data/10000/particles/ep2He/particlePatches/numParticles Dataset {192}
/data/10000/particles/ep2He/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/ep2He/particlePatches/offset Group
/data/10000/particles/ep2He/particlePatches/offset/x Dataset {192}
/data/10000/particles/ep2He/particlePatches/offset/y Dataset {192}
/data/10000/particles/ep2He/particlePatches/offset/z Dataset {192}
/data/10000/particles/ep2He/position Group
/data/10000/particles/ep2He/position/x Dataset {NULL}
/data/10000/particles/ep2He/position/y Dataset {NULL}
/data/10000/particles/ep2He/position/z Dataset {NULL}
/data/10000/particles/ep2He/positionOffset Group
/data/10000/particles/ep2He/positionOffset/x Dataset {NULL}
/data/10000/particles/ep2He/positionOffset/y Dataset {NULL}
/data/10000/particles/ep2He/positionOffset/z Dataset {NULL}
/data/10000/particles/ep2He/weighting Dataset {NULL}
/data/10000/particles/n  Group
/data/10000/particles/n/boundElectrons Dataset {263946240}
/data/10000/particles/n/mass Group
/data/10000/particles/n/momentum Group
/data/10000/particles/n/momentum/x Dataset {263946240}
/data/10000/particles/n/momentum/y Dataset {263946240}
/data/10000/particles/n/momentum/z Dataset {263946240}
/data/10000/particles/n/particlePatches Group
/data/10000/particles/n/particlePatches/extent Group
/data/10000/particles/n/particlePatches/extent/x Dataset {192}
/data/10000/particles/n/particlePatches/extent/y Dataset {192}
/data/10000/particles/n/particlePatches/extent/z Dataset {192}
/data/10000/particles/n/particlePatches/numParticles Dataset {192}
/data/10000/particles/n/particlePatches/numParticlesOffset Dataset {192}
/data/10000/particles/n/particlePatches/offset Group
/data/10000/particles/n/particlePatches/offset/x Dataset {192}
/data/10000/particles/n/particlePatches/offset/y Dataset {192}
/data/10000/particles/n/particlePatches/offset/z Dataset {192}
/data/10000/particles/n/position Group
/data/10000/particles/n/position/x Dataset {263946240}
/data/10000/particles/n/position/y Dataset {263946240}
/data/10000/particles/n/position/z Dataset {263946240}
/data/10000/particles/n/positionOffset Group
/data/10000/particles/n/positionOffset/x Dataset {263946240}
/data/10000/particles/n/positionOffset/y Dataset {263946240}
/data/10000/particles/n/positionOffset/z Dataset {263946240}
/data/10000/particles/n/weighting Dataset {263946240}
/data/10000/picongpu     Group
/data/10000/picongpu/idProvider Group
/data/10000/picongpu/idProvider/nextId Dataset {4, 8, 6}
/data/10000/picongpu/idProvider/startId Dataset {4, 8, 6}
/header                  Group

h5ls -r simData_beta_30000.h5 output:

/                        Group
/data                    Group
/data/30000              Group
/data/30000/fields       Group
/data/30000/particles    Group
/data/30000/particles/en_all Group
/data/30000/particles/en_all/charge Group
/data/30000/particles/en_all/mass Group
/data/30000/particles/en_all/momentum Group
/data/30000/particles/en_all/momentum/x Dataset {2048}
/data/30000/particles/en_all/momentum/y Dataset {2048}
/data/30000/particles/en_all/momentum/z Dataset {2048}
/data/30000/particles/en_all/particleId Dataset {2048}
/data/30000/particles/en_all/particlePatches Group
/data/30000/particles/en_all/particlePatches/extent Group
/data/30000/particles/en_all/particlePatches/extent/x Dataset {192}
/data/30000/particles/en_all/particlePatches/extent/y Dataset {192}
/data/30000/particles/en_all/particlePatches/extent/z Dataset {192}
/data/30000/particles/en_all/particlePatches/numParticles Dataset {192}
/data/30000/particles/en_all/particlePatches/numParticlesOffset Dataset {192}
/data/30000/particles/en_all/particlePatches/offset Group
/data/30000/particles/en_all/particlePatches/offset/x Dataset {192}
/data/30000/particles/en_all/particlePatches/offset/y Dataset {192}
/data/30000/particles/en_all/particlePatches/offset/z Dataset {192}
/data/30000/particles/en_all/position Group
/data/30000/particles/en_all/position/x Dataset {2048}
/data/30000/particles/en_all/position/y Dataset {2048}
/data/30000/particles/en_all/position/z Dataset {2048}
/data/30000/particles/en_all/positionOffset Group
/data/30000/particles/en_all/positionOffset/x Dataset {2048}
/data/30000/particles/en_all/positionOffset/y Dataset {2048}
/data/30000/particles/en_all/positionOffset/z Dataset {2048}
/data/30000/particles/en_all/weighting Dataset {2048}
/data/30000/picongpu     Group
/data/30000/picongpu/idProvider Group
/data/30000/picongpu/idProvider/nextId Dataset {4, 8, 6}
/data/30000/picongpu/idProvider/startId Dataset {4, 8, 6}
/header                  Group

(they start not at zero, thus there is also no 10000 iteration output)

[ax3l]

Thanks for the details!

Skipping non-standard attribute ...

This is fine. We do not support compound HDF5 attributes (yet) and libSplash used some of them for internal information instead of using just array attributes. It's a warning since we cannot read these attributes, but that's about it. You can ignore those, since you are not trying to read those splash reminiscences anyway.

h5ls -r simData_10000.h5

Thank you for the details! Maybe it's related to the empty data sets. In order to make sure, can you please to the following:

• copy the simData_10000.h5 file into an empty directory
• try to open this file alone with the usual series syntax with openPMD-api
• if you got the same error, proceed below, otherwise check which iteration file is actually causing the problem
• if you identified a good file for this issue, can you please h5compress a copy of that file and place it readable for me on hypnos?

[ax3l]

ping @PrometheusPi

[PrometheusPi]

@ax3l I copied (hard link) all non-_beta_ hdf5 files in a directory and still get the same error.

Code:

import openpmd_api as api
import numpy as np

path = "h5/"
series = api.Series(
    path + "simData_%T.h5", 
    api.Access_Type.read_only)

data structure/tree

.
├── h5
│   ├── simData_0.h5
│   ├── simData_100000.h5
│   ├── simData_10000.h5
│   ├── simData_105000.h5
│   ├── simData_110000.h5
│   ├── simData_115000.h5
│   ├── simData_120000.h5
│   ├── simData_125000.h5
│   ├── simData_130000.h5
│   ├── simData_135000.h5
│   ├── simData_140000.h5
│   ├── simData_145000.h5
│   ├── simData_150000.h5
│   ├── simData_15000.h5
│   ├── simData_155000.h5
│   ├── simData_160000.h5
│   ├── simData_165000.h5
│   ├── simData_170000.h5
│   ├── simData_175000.h5
│   ├── simData_180000.h5
│   ├── simData_185000.h5
│   ├── simData_190000.h5
│   ├── simData_195000.h5
│   ├── simData_200000.h5
│   ├── simData_20000.h5
│   ├── simData_205000.h5
│   ├── simData_210000.h5
│   ├── simData_215000.h5
│   ├── simData_220000.h5
│   ├── simData_225000.h5
│   ├── simData_230000.h5
│   ├── simData_235000.h5
│   ├── simData_240000.h5
│   ├── simData_245000.h5
│   ├── simData_250000.h5
│   ├── simData_25000.h5
│   ├── simData_255000.h5
│   ├── simData_260000.h5
│   ├── simData_265000.h5
│   ├── simData_270000.h5
│   ├── simData_275000.h5
│   ├── simData_280000.h5
│   ├── simData_285000.h5
│   ├── simData_290000.h5
│   ├── simData_295000.h5
│   ├── simData_300000.h5
│   ├── simData_30000.h5
│   ├── simData_35000.h5
│   ├── simData_40000.h5
│   ├── simData_45000.h5
│   ├── simData_50000.h5
│   ├── simData_5000.h5
│   ├── simData_55000.h5
│   ├── simData_60000.h5
│   ├── simData_65000.h5
│   ├── simData_70000.h5
│   ├── simData_75000.h5
│   ├── simData_80000.h5
│   ├── simData_85000.h5
│   ├── simData_90000.h5
│   └── simData_95000.h5
└── test_openPMD-api.ipynb

error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-8425feb474c6> in <module>
      2 series = api.Series(
      3     path + "simData_%T.h5",
----> 4     api.Access_Type.read_only)

RuntimeError: Unsupported dataset class

[PrometheusPi]

iteration 10000 alone throws the same error
iteration 0 too

[PrometheusPi]

@ax3l you can download the simData_0.h5 file from my /bigdata/hplsim/scratch/.../.

[ax3l]

So files at step 0 and 10000 seam to produce the error.
Can you update your comment with the path you want to download, please? :) Update: oh, you mean your user dir.

[ax3l]

I addressed both the read problem with somewhat curiously formatted zero-particle entried in libSplash as well as the very verbose messages on startup in #641

[PrometheusPi]

@ax3l Yes I meant my user dir which contains my user name which I stated as /.../.

[ax3l]

Merged #641 to dev to address the problems you experienced.
Please feel free to report if this works for you :)

[PrometheusPi]

@ax3l Is there a way to test this new version using conda?

[ax3l]

Yes, I just released 0.10.3 which contains the PR and its now available on conda-forge, too.

(In order to test dev versions before they are released just build from source or use Spack.)

[ax3l]

@PrometheusPi Were you able to read your files with the latest release?

[PrometheusPi]

@ax3l Please excuse the late reply. Yes, your fix solved the issue for me.

[ax3l]

Nice, thanks for testing!