Openpmd-api: The best way to resize record component?

[KseniaBastrakova]

According to openPMD-api, to write record component, one needs to first create a record component structure and after assign values to the record component.

Python:

mpiDims = [4]
x_particle_values_first_part = np.arange(2, dtype=np.float32)
x_particle_values_second_part = np.arange(2, dtype=np.float32)
d = Dataset(partial_particlePos.dtype, extent=mpiDims)
electrons["position"]["x"].reset_dataset(d)
electrons["position"]["x"][0:2] = x_particle_values_first_part
electrons["position"]["x"][2:4] = x_particle_values_second_part 

I am currently modifying my particle_reduction from directly using hdf5 to openPMD-api. I got the following problem: I process and write values of every patch separately, and in the process I don't know the resulting size of the full particles' dataset before finishing the last patch.

When I used h5py, I did this like

self.hdf_file[node_name].resize(new_size).

What is the best way to do it in openPMD-api? As a note, I do know an upper bound size of my output data, so perhaps something like preallocating this size and then scaling down to the actual resulting size can work?

[ax3l]

Hi Ksenia,

for performance reasons, re-sizable data sets are not (yet) supported: #510 (But generally, ADIOS as well as HDF5 support resizable data sets.)
Indeed, if you can somehow estimate the upper size or count before writing the extend this would be the best solution. Are you potentially able to iterate the patches twice, once quickly accumulating the size meta data for extend and once for "heavy" data reads?

[KseniaBastrakova]

Hi, Axel
yes, I can estimate the upper size before writing the extend. But is it possible to reduce zero-values fust? Because now, my problem is that I got a dataset with result values and zero-tail.

Are you potentially able to iterate the patches twice, once quickly accumulating the size meta data for extend and once for "heavy" data reads?

No, for performance reasons, it's impossible now.

[ax3l]

Yes, so I am currently not working on that feature due to it's known lack of performance (and we current tune the latter). But PRs for #510 are certainly welcome!

Can you point me to the exact source code lines you refer to? Maybe we find a suitable way to redesign your iterations.

[sbastrakov]

@ax3l i believe the difficulty here is that for all the reduction algorithms we use, there is no way of knowing the resulting number of macroparticles before finishing the algorithm, so it's rather algorithmic feature and does not come from a particular implementation. So to me it looks like the sad way to go is to first create a dataset with upper bound size and write results there, and after everything is done to do another one with a proper size and copy. (I guess that's what the HDF5 resize-based implementation did anyways)

[KseniaBastrakova]

Yes, @sbastrakov is right: half of the algorithms are random-based(and the exact number of particles may vary from run to run) and another half is cell-based, so we also can't estimate number of particles without complete algorithm.

I think it's a best solution here: create a dataset with bound size, and after finishing algorithm, create another one and copy. I will do so.

[ax3l]

I guess that's what the HDF5 resize-based implementation did anyways.

What HDF5 does for resizable extents is chunking. Similarly, ADIOS keeps your data as it is (in chunks as you provide them, maybe aggregates) and "shapes"/extents are just some voluntary meta information on top of it.

I think it's a best solution here: create a dataset with bound size, and after finishing algorithm, create another one and copy. I will do so.

Sounds legit, sorry that I cannot provide this feature right now in the API (maybe you want to contribute this or have a student that can work on it?).