Adios2: Auto-generated info for directory hierarchy in ADIOS output

[pnorbert]

Challenge: reconstruct the hierarchy at read time fast. OpenPMD spends 30 seconds to do that currently.

Idea: ADIOS could save the hierarchy as a string variable as a JSON/YAML document at write time, that either the application can use to generate the hierarchy or ADIOS could use it to generate a tree instead of the plain map of variables and attributes.

This could speed up the variable/variable and attribute/variable association functions at read time as well.

@ax3l @psychocoderHPC @franzpoeschel @BeyondEspresso

[williamfgc]

This is now in master, please test if this has an impact on the current read performance. Thanks.

[ax3l]

cc @franzpoeschel can you please test this? :)
(also, feel free to share a little benchmark with little data and much meta-data in openPMD-api)

[williamfgc]

@NAThompson , this could be a good reading benchmark to cover openPMD use case. To provide some background, @NAThompson has been helping us with setting benchmarks with google benchmark.

[franzpoeschel]

Thanks for adding this! Reaping the benefits of this PR in openPMD will require some further changes in its usage of ADIOS2, since hierarchical groups are currently not represented as variables and this implementation is hence not directly applicable to inquiring attributes associated with openPMD groups. Once I find the time, I will apply the necessary modifications and report back.

[williamfgc]

@franzpoeschel thanks for the follow up. Just to make sure we are on the same page, this is not adding variable/variable hierarchy, but "caching" the current variable/attribute hierarchy. Basically, if you were using the InquireAttribute<T>( "att", "var" "/"); signature you should be able to test it.

[franzpoeschel]

Since InquireAttribute<T>( "att", "var", "/"); was previously a mere prefix search, it was also possible to call e.g. InquireAttribute<T>( "timeUnitSI", "/data/800", "/"); to inquire an attribute /data/800/timeUnitSI even if no variable /data/800 had never been defined. This was likely non-standard usage and it is good if I get rid of it, but this PR has actually broken the openPMD ADIOS2 backend this way.
Before making use of this PR, I will first need to make sure to define empty variables also for openPMD groups and not only for actual datasets.

[williamfgc]

@franzpoeschel I see, thanks for the explanation and sharing the bug report. Looking forward to follow up on this PR. Thanks!

[franzpoeschel]

As mentioned, I will have to use dummy variables of zero extent for implementing this. Just like ADIOS1, ADIOS2 will not store variables if no data has been written in a step. I guess this means, that for implementing this, I will have to call dummy writes on those variables per step? Example:

adios2::ADIOS adios;
{
    auto IO = adios.DeclareIO( "IO write" );
    auto var =
        IO.DefineVariable< int >( "empty dataset", { 0 }, { 0 }, { 0 } );
    auto engine = IO.Open( "empty.bp", adios2::Mode::Write );
    engine.BeginStep();
    IO.DefineAttribute< int >( "attr", 10, "empty dataset" );
    // engine.Put( var, 0 );
    engine.EndStep();
}
{
    auto IO = adios.DeclareIO( "IO read" );
    auto engine = IO.Open( "empty.bp", adios2::Mode::Read );
    engine.BeginStep();
    auto map = IO.AvailableAttributes("empty dataset");
    std::cout << "available attributes: ";
    for (auto const & pair : map )
    {
        std::cout << pair.first << ", ";
    }
    std::cout << std::endl;
    engine.EndStep();
}

This will output:

available attributes: 

But if I uncomment the commented line, this shows:

available attributes: attr, 

Also, bpls will output

  int32_t  empty dataset       scalar

instead of nothing at all.

[williamfgc]

@franzpoeschel Since the intention is not to write the Variable in the first place, why not just passing the attribute name IO.DefineAttribute< int >( "empty dataset/attr", 10 ); That way the attribute is associated with the entire dataset.

Also, I thought the slowdown is caused by attributes associated with actual variables, not the attributes that belong to the entire dataset (these are put in a hash so search should be constant, but it's always better to measure). Is it the latter or the former?

[franzpoeschel]

A real dataset written by the openPMD API may for example contain the following attributes:

  string    /data/900/particleBoundary                                                                         attr
  string    /data/900/particleBoundaryParameters                                                               attr
  string    /data/900/particleInterpolation                                                                    attr
  string    /data/900/particlePush                                                                             attr
  double    /data/900/particleShape                                                                            attr
  string    /data/900/particleSmoothing                                                                        attr

There is no variable /data/900 since within openPMD, that is not a dataset (which would be represented with a variable), but a hierarchical group, possibly containing further groups and datasets. When parsing an openPMD Series, the challenge is, given a group /data/900, to inquire the backend, in this case ADIOS2, for the attributes associated with that group. At this point, I don't yet know the available attributes seen above.
Ideally, I would be able to use the improvements coming with PR #1821 to efficiently inquire what attributes are available in a group. But since there is no variable corresponding with /data/900, the call to IO.AvailableAttributes("/data/900") returns with an empty map. What the ADIOS2 backend in the openPMD API currently does is exactly what you suggest: Pass the attribute globally during write-time. But this means that in order to figure out what attributes are available of the form /data/900/*, I currently need to do a global linear scan.
So to answer your question, the slowdown comes from attributes that are currently not associated with a variable, but should likely be associated with at least something. Hence, my idea was to add a dummy variable whose only purpose is bundling those attributes.

[williamfgc]

@franzpoeschel thanks for the example and explanation. So you want to extract by prefix, not by variable name, is that a fair assumption?

But this means that in order to figure out what attributes are available of the form /data/900/*, I currently need to do a global linear scan.

Are you using std::map::lower_bound? If not, it might be worth applying it on the returned std::map from IO::AvailableAttributes(); and measure if there is any performance difference. See this post.

[franzpoeschel]

Thanks for the suggestion, this seems like a more straightforward option, without requiring ugly workarounds. This might mean that we will not – or only partially – use the changes made with PR #1821.

[pnorbert]

@franzpoeschel Can we close this issue?

[franzpoeschel]

From my side, yes.
Thanks again for your help! @pnorbert