Adios2: Why do local and global arrays behave differently in regards to overwriting existing blocks?

[Bella42]

Hi,
I have a question regarding the behaviour of arrays in ADIOS2.

When writing a local array such as follows, then there is a block for each put call. The put does not overwrite any existing block but creates a new one with a incremented blockID as shown with bpls. However, when using a global array the behaviour is different and the blocks are somewhat still visible through the metadata information, e.g. such as the min and max values but the actual data is overwritten.
Is this difference intentional and if so, why is that?

Local Array Put:

const size_t Nglobal = 4;

std::vector<double> v1(Nglobal);
std::vector<double> v2(Nglobal);
std::vector<double> v3(Nglobal);
std::vector<double> v4(Nglobal);

adios2::ADIOS adios(adios2::DebugON);
adios2::IO io = adios.DeclareIO("Output");
io.SetEngine("bp4");

adios2::Variable<double> varV0 =
    io.DefineVariable<double>("v0", {}, {}, {Nglobal});

adios2::Engine writer = io.Open(fileName, adios2::Mode::Write);

for (size_t i = 0; i < Nglobal; i++)
{
    v1[i] = 11 + 1 * 0.1 + i * 100;
    v2[i] = 22 + 2 * 0.1 + i * 100;
    v3[i] = 33 + 3 * 0.1 + i * 100;
    v4[i] = 44 + 4 * 0.1 + i * 100;
}

for (int i = 0; i < 1; i++)
{
    writer.BeginStep();

    writer.Put<double>(varV0, v1.data(), adios2::Mode::Sync);
    writer.Put<double>(varV0, v2.data(), adios2::Mode::Sync);
    writer.Put<double>(varV0, v3.data(), adios2::Mode::Sync);
    writer.Put<double>(varV0, v4.data(), adios2::Mode::Sync);

    writer.EndStep();
}
writer.Close();

bpls output for local array:

./bin/bpls -d -l LocalArray.bp
  double   v0    3*[4]*{4} = 11.1 / 344.4
        step 0: 
          block 0: [0:3] = 11.1 / 311.1
    (0)    11.1 111.1 211.1 311.1 
          block 1: [0:3] = 22.2 / 322.2
    (0)    22.2 122.2 222.2 322.2 
          block 2: [0:3] = 33.3 / 333.3
    (0)    33.3 133.3 233.3 333.3 
          block 3: [0:3] = 44.4 / 344.4
    (0)    44.4 144.4 244.4 344.4 

Global Array Put (the same apart from variable definition)

...
 adios2::Variable<double> varV0 =
        io.DefineVariable<double>("v0", {Nglobal}, {0}, {Nglobal});
...

bpls output for global array:

./bin/bpls -d -l GlobalArray.bp
  double   v0    3*{4} = 11.1 / 344.4
    (0,0)    44.4 144.4 244.4 344.4 44.4 144.4
    (1,2)    244.4 344.4 44.4 144.4 244.4 344.4

./bin/bpls -dD -l GlobalArray.bp
  double   v0    3*{4} = 11.1 / 344.4
        step 0: 
          block 0: [0:3] = 11.1 / 311.1
    (0)    44.4 144.4 244.4 344.4 
          block 1: [0:3] = 22.2 / 322.2
    (0)    44.4 144.4 244.4 344.4 
          block 2: [0:3] = 33.3 / 333.3
    (0)    44.4 144.4 244.4 344.4 
          block 3: [0:3] = 44.4 / 344.4
    (0)    44.4 144.4 244.4 344.4 
        step 1: 
          block 0: [0:3] = 11.1 / 311.1
    (0)    44.4 144.4 244.4 344.4 
          block 1: [0:3] = 22.2 / 322.2
    (0)    44.4 144.4 244.4 344.4 
          block 2: [0:3] = 33.3 / 333.3
    (0)    44.4 144.4 244.4 344.4 
          block 3: [0:3] = 44.4 / 344.4
    (0)    44.4 144.4 244.4 344.4 

[pnorbert]

Global Arrays are local arrays with extra metadata to organize the blocks into a (virtual) global space. This allows writing and reading back data with spatial selection. With Local Arrays one can only read with block selection (which is still available for Global Arrays).

The storage of global arrays is the same as local arrays, so new blocks NEVER overwrite data. In the ADIOS format and API, there is no possibility to overwrite data.

Logically, you may attempt to overwrite data in a global array using a spatial selection, but what happens is that you will have multiple blocks in the file referring to the same spatial selection, and at read time you will end up getting one of them. But you have no control over that which one you get. So you should never assume you have overwritten/updated any data.

[pnorbert]

Your Global Array example is incorrect. You specify a 1D array of size {4} but then you write 4 arrays of 4 elements into it. Global Array gives you the way to organize that 4x4 = 16 elements into either a 1D array of size {16}, or a 2D array of size {4,4} and use offsets in varv0.SetSelection() to place them into the global space.

[Bella42]

@pnorbert First of all, thank you very much for your fast and detailed replies. I am a bit unsure whether I understood them.

I expected the local array to behave exactly as it has. Not overwriting anything but creating a new block for each put. However, I am still not sure why this is not possible in the same way for global arrays.
I did not want to overwrite the data but it looks like it did which is what got me to wonder. See the third bpls output where the actual data is just that of the forth vector. The metadata looks as expected.

I thought it would be possible to write a variable (local or global) several times during one step which is what I wanted to illustrate with writing data from the four different vectors so that I could see which data is actually written. So is the answer to my confusion just that I am not supposed to write one variable several times in a step?

[pnorbert]

Yes, ADIOS is step based. In one step you are supposed to write one or many variables, but each one once. If you cannot organize your data into neat global arrays, then you have the option to write local arrays. In that case, you can write as many blocks as you please in a single step. They count as "many blocks of a single variable in a single step".

Mind you, the writing is happening in EndStep, Put() only buffers data so you are limited by memory how many blocks you can write.

In any case, Global Arrays is a sugar syntax to enable you to stop thinking in terms of blocks. You write your data and read your data with spatial selections. Sticking to blocks in your way of thinking/coding just does not make sense when using global arrays. You have the local arrays for that.

Your example code is just testing around but I cannot see what you are trying to achieve in your application. So what is the application? Can you represent your output data in global arrays, so that you can define both write and read operations with spatial selections?

[williamfgc]

@Bella42 yes, local arrays are designed to be independent, I'm currently using them for unstructured mesh in which each domain is independent and it's a mesh on its own (it this helps). Global Arrays are designed for continuity across MPI communicators (adios2 being a MPI library), each rank contains an offset and count and the ideal case is that there is no excluding overlap (having ghost cells is inclusive overlap as they are physically the same values). The mutually exclusive case for global arrays is rare, I don't recall if it was the last block or the first block that gets exposed. Multiple blocks per step of a variable was designed for block-based local arrays mostly (I recall it was a lot of work to support this), not so much for global arrays as it's rare (although we can do a block selection, but I don't recall implementing this). It's better to create separate global variables as they are more like a set of variables with different meaning or masks to the same thing (something like var/level0, var/level1). We can help you with your use-case as @pnorbert suggested. BTW, here are the docs. Overall, adios2 semantics tries to resemble common I/O patterns in HPC simulations, but would be interesting to see new I/O patterns. Thanks!

[Bella42]

@pnorbert @williamfgc Many thanks for clearing that up for me! This discussion has been really helpful!
I am currently not actually writing an application using ADIOS2 but an engine which stores the metadata of self-describing data formats such as the BP format differently as part of my PhD. Since I am not trying to create a new format but rather research existing ones my engine needs to be able to mimic the behaviour of the BP3 and BP4 engines. That is why I was testing what is happening in this possibly obscure cases to make sure my engine also works the same to the application as the BP engines with parallel applications.

[williamfgc]

Since I am not trying to create a new format but rather research existing ones my engine needs to be able to mimic the behaviour of the BP3 and BP4 engines.

@Bella42 , you might be interested in @pnorbert 's excellent Python 3 bp4dbg util which bypasses entirely the ADIOS2 library. It's essentially backwards engineering the BP4 format to verify the assumed standard is met after generation (I caught a few bugs in the library with it). There is not such a thing for BP3, there is the adios1.x bpdump utility, but I think it's still calling the adios1.x library internals.

[Bella42]

@williamfgc Thanks for pointing that out! Seems very interesting and useful :-)

[Bella42]

Yes, ADIOS is step based. In one step you are supposed to write one or many variables, but each one once. If you cannot organize your data into neat global arrays, then you have the option to write local arrays. In that case, you can write as many blocks as you please in a single step. They count as "many blocks of a single variable in a single step".

@pnorbert Just to make sure I understand you correctly for local arrays, do you mean "write as many blocks as you please in a single step" or "write as many blocks as you please in a single step but at most one block per process"? Or to ask differently, how many blocks can I write with one process in one step?

[pnorbert]

Yes, as many blocks per process as you want. You need to call var.SetSelection() to set the size of each block before the Put() calls if their sizes are changing. You are only limited by the memory for how many you can buffer in one step.

[Bella42]

@pnorbert Thanks again for taking the time! I really appreciate all the answers :-)

[williamfgc]

This test shows what @pnorbert described. This is OK for global arrays as the blocks don't overlap and at the end they populate the entire shape. Hope it helps.