Adios2: Resize variables within a step after the first Engine::Put

[franzpoeschel]

Why is this feature important?
In some simulations, the extent of a variable cannot be known at the beginning of a step.
Currently, the openPMD API requires users to specify e.g. the total number of particles in an iteration / a step before writing the first particle. Likewise, ADIOS2 fixes the shape of a variable upon the first call to Engine::Put(). If ADIOS variables could be extended within a step, we could expose this functionality in the openPMD API to support such workflows.

What is the potential impact of this feature in the community?
ADIOS2 and the openPMD API would become accessible to a wider range of simulations that require IO workflows not yet supported by either.

Is your feature request related to a problem? Please describe.
Needed for implementing PR 829 in the openPMD API.

Changing a variable's shape within a step is currently silently ignore which can be considered a bug:

#include <adios2.h>
#include <vector>

int
main()
{
    using namespace adios2;
    ADIOS adios;
    IO IO = adios.DeclareIO( "IO" );
    Engine engine = IO.Open( "file", Mode::Write );

    Variable< int > var =
        IO.DefineVariable< int >( "var", { 10 } );
    std::vector< int > data( 20 );

    engine.BeginStep();

    var.SetSelection( { { 0 }, { 10 } } );
    engine.Put( var, data.data() );

    // engine.EndStep();

    // engine.BeginStep();

    var.SetShape({20});
    var.SetSelection( { { 10 }, { 20 } } );
    engine.Put( var, data.data() );

    engine.EndStep();
}

Result:

> bpls file/ -ad
  int32_t  var  {10}
    (0)    0 0 0 0 0 0
    (6)    0 0 0 0

Describe the solution you'd like and potential required effort
The above example should produce (or be rewritable to a similarly structured code to produce) a data dump that respects the reshaping of the variable. As discussed with @pnorbert, this might require (1) changes to the BP4 format or (2) heuristics on the reading end if no such changes are applied.

Describe alternatives you've considered and potential required effort
Implementing such a feature directly in the openPMD API without support from ADIOS2 would require needlessly duplicating efforts such as buffering data, whereas implementing it directly in ADIOS2 would open up such workflows to all users of ADIOS2 and openPMD. Current workarounds such as forcibly splitting one step into several lead to data dumps that are awkward to read.

[williamfgc]

@franzpoeschel would declaring two variables a reasonable workaround? Something like var/shape1 and var/shape2. I don't know how a variable would be presented if the shape changes within a step as it becomes a hierarchy rather than truly multidimensional array per step. Yeah, the current behavior is a bug. Thanks for catching that.

[franzpoeschel]

That workaround could be applied in two manners probably, both of which have their downsides:

• Such simulations could simply write several datasets /data/0/E/x_1, /data/0/E/x_2, /data/0/E/x_3, …. Not really a good solution since it requires (1) writing applications to manually circumvent these current restrictions and (2) reading applications to piece datasets back together. This would mean that such datasets would only be readable by applications that go through the effort to do that.
• We could write such helper variables internally and piece datasets back together internally. Since the openPMD API needs to adhere to the openPMD standard in the datasets that it produces, this cannot really be done without extending the openPMD standard to allow for the distribution of a logical dataset over several physical ones.

Since @ax3l has been mainly in contact with teams asking for such a feature, I should probably ask for his input on this one, too.

[ax3l]

Yep, writing multiple vars is not a good solution to this, just imagine the people that try to implement this in Fortran if we standardized this in openPMD. openPMD-api is "just" a reference implementation of openPMD-standard, a lot of people in the community implement against the lower-level file format APIs.

[williamfgc]

I think there is two separate aspects here: i) the higher-level data model presented in openPMD and ii) how variables are laid out in memory (BP buffer) and disk at the backend level. They don't have to match 1-to-1, in fact that's why backends can be easily plugged in openPMD as HDF5 and adios2 have very different in-memory and disk data models.

The trouble I see is how a variable with multiple shapes in a single step is presented (sub-variable?) to the user. For example, shape1=4x2 and shape2=1x4 would have undefined regions. How is it read back? This is regardless if the stitching for reading is done at the backend or frontend.

Hence, my suggestion to handle this at the openPMD level as an initial workaround and just have two separate variables at the adios2 level (which is already implemented). Building a new production quality serializer/deserializer is not a trivial task. BTW, how is this handle with the HDF5 backend? Hope this helps.

[pnorbert]

Working on it. We always supported the idea to change the Shape between variable definition time and first Put() but you are basically extending it to the point of EndStep. This is doable on the read side by looking at the shape of the last block in the file not the first one. But it turns out to be messy handling changing shapes so I am not giving a solution just yet.

[franzpoeschel]

Thanks for looking into this :)

[pnorbert]

@franzpoeschel Can you please test this feature? I am a bit worried about this fix, as I had to rewrite handling dimensions over steps differently on the read side.

[franzpoeschel]

@pnorbert I tested this PR with the current ADIOS2 master now and things seem to be working, parallel as well as serial. This is great, thanks again!