Adios2: Closing engines in different orders across MPI ranks (potential bug on master)

[franzpoeschel]

Describe the bug
I noticed that our testcases in the openPMD API will hang if using ADIOS2 master. I found out that we apparently do not close engines in a consistent order across MPI ranks (which is very likely a bug on our side), but interestingly, ADIOS2 in version v2.6.0 took no issue with that.

Since this means that there is a workflow that is now no longer possible, I thought I'd rather report this.

To Reproduce
Use MPI with two ranks, write to two files and close the engines in different orders:

#include <adios2.h>
#include <iostream>
#include <mpi.h>
#include <string>
#include <vector>

adios2::Engine
writeOneFile(
    adios2::ADIOS & adios,
    std::string const & name,
    std::vector< int > & data,
    std::string const & engine_type,
    int mpi_rank )
{
    adios2::IO IO = adios.DeclareIO( name );
    IO.SetEngine( engine_type );
    auto var = IO.DefineVariable< int >( "var", { 5 } );
    adios2::Engine engine = IO.Open( name, adios2::Mode::Write );
    return engine;
}

int
main( int argsc, char ** argsv )
{
    MPI_Init( &argsc, &argsv );
    MPI_Comm comm = MPI_COMM_WORLD;
    int mpi_rank, mpi_size;
    MPI_Comm_rank( comm, &mpi_rank );
    MPI_Comm_size( comm, &mpi_size );

    std::cout << "Initialized MPI" << std::endl;

    std::string engine_type = "bp4";
    if( argsc == 2 )
    {
        engine_type = argsv[ 1 ];
    }
    adios2::ADIOS adios{ comm };
    std::vector< int > data( 5, 17 );

    std::vector< adios2::Engine > engines;
    engines.reserve( 2 );
    for( unsigned i = 0; i < 2; ++i )
    {
        engines.push_back( writeOneFile(
            adios, std::to_string( i ) + ".bp", data, engine_type, mpi_rank ) );
    }

    switch( mpi_rank )
    {
        case 0:
            for( auto it = engines.begin(); it != engines.end(); ++it )
            {
                it->Close();
            }
            break;
        case 1:
            for( auto it = engines.rbegin(); it != engines.rend(); ++it )
            {
                it->Close();
            }
            break;
    }
    MPI_Finalize();
}

If running with v2.6.0, this will produce two empty data dumps, if running on master, this will hang.

Expected behavior
I'm not sure what to expect here. Whether this is a bug or wrong usage is up to you, we should fix it in the openPMD API anyway.

Desktop (please complete the following information):
Observed on multiple platforms with multiple compilers.

[pnorbert]

Probably we don't emphasize this enough in the documentation but Open, BeginStep, EndStep and Close are collective operations, so you should expect a hang whenever mixing up these calls in different processes. Just because one engine (BP4) does only call local functions in Close() when everything global has already been done, it is still not safe to assume you can treat it as a local function.

For 2.6 vs master: the default aggregator scheme has changed. 1 file per process changed to 1 file per compute node (shared memory domain). And now BP4's Close() is always collective unless you set the parameter AggregatorRatio=1.

[franzpoeschel]

Alright, thanks for the fast reply. Then I'll fix it in openPMD.

[williamfgc]

@franzpoeschel thanks for sharing. Open/Close must be collective calls, they copy the MPI comm used. Even if you don't write from all ranks (that's perfectly fine) here is the docs with a caution note. There is an exception when using MPI_COMM_SELF, but that's a special case.