Picongpu: openPMD adios2 output missing

[PrometheusPi]

I just ran a default LWFA simulation on summit using ADIOS2 (adios2/2.5.0 module) and openPMD-api 0.12.0. PIConGPU compiled without problems and the interface for the openPMD plugin was created. However, no *.bp output files are created. The openPMD directory is completely empty. I ran both:

 --openPMD.period 5000:10000:2500 --openPMD.file simData --openPMD.ext bp       

 --openPMD.period 5000:10000:2500 --openPMD.file simData --openPMD.ext bp --openPMD.json '{ "adios2": { "engine": { "type": "file", "parameters": { "BufferGrowthFactor": "1.2", "InitialBufferSize": "2GB" } } } }' 

The openPMD directory is created and the simulation reaches 100%, but no output is created.
stdout and stderr give no clue of an error.

@ax3l @psychocoderHPC @franzpoeschel Any ideas what might cause this problem?

[ax3l]

ADIOS2 does not create or need .bp meta files anymore. You can operate completely on the directory it creates, even without a call on bpmeta.

This is identical to ADIOS1 behavior with have_metadata_file=0 (.disable-meta 1) - here you just had to run bpmeta before you could read the data again.

[PrometheusPi]

@ax3l Sorry for the confusion: no data at all is created. (I will correct the post above.)

[ax3l]

@PrometheusPi ok, I'll let @franzpoeschel investigate this, since it seems PIConGPU specific.

Also double-check that all backends you expect in openPMD-api, i.e. the ADIOS2 backend, are available, please: https://openpmd-api.readthedocs.io/en/0.12.0-alpha/citation.html#dependent-software

[franzpoeschel]

using ADIOS2 (adios2/2.5.0 module) and openPMD-api 0.12.0

openPMD 0.12.0 requires ADIOS 2.6. Can you verify whether your openPMD build includes an ADIOS backend?

[franzpoeschel]

@ax3l We might want to consider throwing an exception in that case just like ADIOS1 does instead of silently doing nothing…

[PrometheusPi]

@franzpoeschel The cluster only provides adios2/2.5.0 and adios2/2.4.0.
What version of openPMD-api should I use with adios2/2.5.0?

[PrometheusPi]

Okay - adios 2 2.5.5 works with openPMD-api 0.11.1 - I installed it (ctest failed with MPI) but now I get the following compile error:

/.../picongpu/include/pmacc/../picongpu/plugins/openPMD/openPMDWriter.def(113): error #135: namespace "openPMD" has no member "Access"

[PrometheusPi]

I will remove 1fb4d57607e5192e2734883ab47b3c5f961ba094 and try again

[PrometheusPi]

Next error:

/.../picongpu/include/pmacc/../picongpu/plugins/openPMD/openPMDWriter.hpp(142): error #289: no instance of constructor "openPMD::Series::Series" matches the argument list
            argument types are: (std::__cxx11::string, openPMD::AccessType, MPI_Comm, std::__cxx11::string)

[benjha]

I think @ax3l deployed the latest OpenPMD software stack for Summit. This can be activated by

module load ums
module load ums-aph114
module load openpmd-api/0.12.0

[franzpoeschel]

Yeah, the openPMD plugin in PIConGPU has always relied on 0.12.0 features in the openPMD API, so your best bet is going to be installing ADIOS2.6.0 manually for now. I'm using something like this atm:

# set PREFIX to your preferred install prefix
install_adios2() {
    if [ ! -d ADIOS2 ]; then
        git clone https://github.com/ornladios/ADIOS2
    fi
    cd ADIOS2
    git checkout master
    if [ ! -d build ]; then
        mkdir build
        cd build
        cmake .. -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_BUILD_TESTING=OFF \
            -DCMAKE_INSTALL_PREFIX="$PREFIX" -DADIOS2_USE_Fortran=OFF \
            -DADIOS2_USE_BZip2=OFF -DADIOS2_USE_PNG=OFF \
            || wait 1 && cmake .
    else
        cd build
    fi
    make -j `nproc` install
    cd ../..
}

EDIT: Or try @benjha's suggestion first, that one looks a bit easier ;)

[PrometheusPi]

The module provided by @ax3l worked like a charm. ADIOS2 data could be produced on summit using the openPMD plugin.

[ax3l]

Excellent, glad this helps.
Just be aware this it built against the gcc/6.4.0 software stack if you should update yours at some point.
aph114 is our ECP project for WarpX, this is the software stack (including MPI and HDF5 version): https://warpx.readthedocs.io/en/latest/building/summit.html

I requested ADIOS2 2.6.0 deployment on Summit and also triggered workflows to get newer ADIOS2 releases faster, but they seem to be still in the pipeline.

We might want to consider throwing an exception in that case just like ADIOS1 does instead of silently doing nothing…

In order to avoid some confusion and focus maintainance, I discussed with @franzpoeschel today that we will disable ADIOS1 writes though openPMD-api in PIConGPU with a clear error message at runtime.

While we can skim off the old libSplash HDF5 plugin soon, we will keep the original ADIOS1 plugin in PIConGPU for a bit longer, for purely conservative reasons. We will do no efforts on writing through openPMD-api via ADIOS1, since this diverts resources we need to put in:

• (primary) getting ADIOS2 to the same level of performance and battle-proven features; we are probably there or very close already; please also perform benchmarks and check scalability and large per-node writes with parallel compressors
• (bridge) maintaining the PIConGPU ADIOS1 plugin with minimal patches, which has a stable and battle-proven implementation
• (bridge) maintain compatibility to read ADIOS1 data in openPMD-api from a single source (instead of two)

The goal has to be to use openPMD-api with ADIOS2 (and HDF5 for smaller runs) and triage if there are issues left that I have not already caught while we are using it in WarpX. I am optimistic that we have pretty good stability for ADIOS2 at this point, so it's good to apply it on a broader basis now.

[sbastrakov]

@ax3l FYI PIConGPU already does not have a standalone HDF5 output plugin (#3361) and the plan is to remove all HDF5 / libSplash in other plugins in two weeks (#3357). So the only way to output anything in HDF5 format would be via openPMD-API. I don't think it causes conflicts with anything you wrote regarding ADIOS or openPMD-API side, just the HDF5 removal part is partialy happened already and partially will happen real soon hopefully.

[ax3l]

Thx for the heads-up @sbastrakov. Looks good.