Openpmd-api: Editing a data000X00 series

[ax3l]

Describe the bug
Editing files in an existing series that is zero-padded file names fails due to a mismatch in file names (the zeroes are forgotten during write).

To Reproduce

Python:

import openPMD

# $ ls samples/git-sample
# data00000100.h5  data00000200.h5  data00000300.h5  data00000400.h5  data00000500.h5

series = openPMD.Series("samples/git-sample/data%T.h5", openPMD.Access_Type.read_write)

series.iterations[42]
# <openPMD.Iteration of at t = '0.000000 s'>

series.flush()

HDF5-DIAG: Error detected in HDF5 (1.10.2) thread 0:
  #000: H5F.c line 511 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1450 in H5F_open(): unable to open file: time = Fri Jun 15 00:10:23 2018
, name = 'samples/git-sample/data100.h5', tent_flags = 1
    major: File accessibilty
    minor: Unable to open file
  #002: H5FD.c line 733 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #003: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file: name = 'samples/git-sample/data100.h5', errno = 2, error message = 'No such file or directory', flags = 1, o_flags = 2
    major: File accessibilty
    minor: Unable to open file
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-8118d0dafbde> in <module>()
----> 1 series.flush()

RuntimeError: Failed to open HDF5 file samples/git-sample/data100.h5

Expected behavior
It should actually try to access to the data00000100.h5 file but it skips the leading zeros.

Software Environment:

• version of openPMD-api: 0.2.1-dev as of 402698012d2587a716b1273bb31333e86e38d4e5
• installed openPMD-api via: from source with gcc 6.3.0
• operating system: Debian GNU/Linux 9.4 (stretch)
• machine: my thinkpad laptop
• name and version of Python implementation: CPython 3.6.5
• version of HDF5: 1.10.2
• version of ADIOS1: 1.13.1
• name and version of MPI: OpenMPI 3.1.0

[C0nsultant]

The problem happens here
https://github.com/openPMD/openPMD-api/blob/7501e3435913216812f3efe8707b30892b3aa915/src/Iteration.cpp#L109

The solution will be straight forward, but there is more to this issue:
To have our created files be consistent, we will need to detect the amount of zero-padding in the name of existing Iterations in the Series.
We also will need to introduce zero-padding in our own files with an option to let the user specify the amount of padding.

https://github.com/openPMD/openPMD-api/blob/7501e3435913216812f3efe8707b30892b3aa915/src/Series.cpp#L588-L589
should also only trigger for Accesstype::READ_ONLY

[ax3l]

The first to point is backen specific and given by the openend files.

For the third one I remember we discussed a little when allowing zero-padding in the standard. I think we could keep everything in the file format as we have it so far with %T (see first aspect). %T matches all and is not zero padded on create.

Additionally, we could allow syntax such as %0<N>T as regex in Series' parameter filepath, e.g. %05T for 00100, 00200, ... matching in both (strict) reads and zero-padded creates. Didn't yt.load() have a similar syntax for the same purpose?

What do you think?

[C0nsultant]

Additionally, we could allow syntax such as %0<N>T

I liked the idea so much, I implemented a PoC in #270. It's backward compatible to %T, which is great.
What do you think about other padding characters that are specified as x in

%x\dT

Didn't yt.load() have a similar syntax for the same purpose?

IIRC they used wildcards.

[ax3l]

I implemented a PoC

nahaaaice!

What do you think about other padding characters that are specified as x in

probably not needed :)

IIRC they used wildcards.

true, I think it's *