Adios2: BP3 and BP4 (and BP5) File Magic

[ax3l]

By supporting ADIOS1 and ADIOS2 in openPMD-api for file backends, we seem to cause confusion for users in corner-cases where both the BP3 engine is not 100% compatible as well as situations where we slightly choose a different data model for ADIOS1 and ADIOS2 in openPMD-api (e.g. for internal representation of boolean types).

Ref.: https://github.com/ComputationalRadiationPhysics/picongpu/issues/3378

I wonder if BP3 files have a file-magic that we could probe for runtime detection and/or if we should start to call all ADIOS2 files we write and read ".bp4" so we can spare the file probing and continue relying on "file (directory) suffixes".

Or could we call a standalone ADIOS2 API call with a string argument identical to IO::Open and it will return us if this is BP3/BP4/newer?

Suggestions very welcome :)

[williamfgc]

I would just support BP4 and ADIOS2 for the long term as that's where all the currents efforts are put. BP3 hasn't been touched in a long time and there are efforts to migrate SST and climate to BP4. Is this correct, @pnorbert ?

[ax3l]

I agree with that, but have to transition existing users and their data gracefully: https://github.com/ComputationalRadiationPhysics/picongpu/issues/3378#issuecomment-708176759

[williamfgc]

@ax3l let me understand the request, my understanding is that the default virtual file engine detects the BP3/BP4 type at read. Do you need to expose this type in the API?

[germasch]

You can detect the difference between BP3 and BP4 by checking whether abc.bp is a file or a directory. I don't know if that helps -- this cannot tell the difference between a BP3 file written by ADIOS1 vs ADIOS2. There is a magic that encodes the ADIOS2 version at the very end of some of the written files -- I don't remember the details, but I can check if it's helpful.

I'd also support naming the next iteration .bp5, unless backward and forward compatibility is ensured. I've been bit by the "same suffix, different format" between BP3 and BP4 more than once, in particular one gets unintuitive errors when rerunning something and switching between BP3 and BP4. IIRC something like "cannot create/open file" when trying to overwrite and existing BP4 output with a BP3 output of the same name, since the name is already taken for a directory, and hence a file of the same name cannot be created.

[pnorbert]

File = BP3
Directory = BP4

adios2/source/core/IO.cpp: 533

    if ((engineTypeLC == "file" || engineTypeLC == "bpfile" ||
         engineTypeLC == "bp" || isDefaultEngine))
    {
        if (helper::EndsWith(name, ".h5", false))
        {
            engineTypeLC = "hdf5";
        }
        else if (mode == Mode::Read)
        {
            if (adios2sys::SystemTools::FileIsDirectory(name))
            {
                engineTypeLC = "bp4";
            }
            else
            {
                if (helper::EndsWith(name, ".bp", false))
                {
                    engineTypeLC = "bp3";
                }
                else
                {
                    /* We need to figure out the type of file
                     * from the file itself
                     */
                    if (helper::IsHDF5File(name, comm, m_TransportsParameters))
                    {
                        engineTypeLC = "hdf5";
                    }
                    else
                    {
                        engineTypeLC = "bp3";
                    }
                }
            }
        }
        else
        {    if ((engineTypeLC == "file" || engineTypeLC == "bpfile" ||
         engineTypeLC == "bp" || isDefaultEngine))
    {
        if (helper::EndsWith(name, ".h5", false))
        {
            engineTypeLC = "hdf5";
        }
        else if (mode == Mode::Read)
        {
            if (adios2sys::SystemTools::FileIsDirectory(name))
            {
                engineTypeLC = "bp4";
            }
            else
            {
                if (helper::EndsWith(name, ".bp", false))
                {
                    engineTypeLC = "bp3";
                }
                else
                {
                    /* We need to figure out the type of file
                     * from the file itself
                     */
                    if (helper::IsHDF5File(name, comm, m_TransportsParameters))
                    {
                        engineTypeLC = "hdf5";
                    }
                    else
                    {
                        engineTypeLC = "bp3";
                    }
                }
            }
        }
        else
        {
            engineTypeLC = "bp4";
        }
    }
            engineTypeLC = "bp4";
        }
    }