Picongpu: openPMD adios1 output defective

[PrometheusPi]

When writing openPMD output using the openPMD plugin and the bp output format (ADIOS) on hemera, PIConGPU creates defective output files at least under the following condition:

• using the recommended modules (including openpmd/0.11.1-cuda102)
• using the flags:
  
  
  
  • from the LWFA example 8.cfg
  
  
  • or from TBG_macros.cfg
  
  

When loading the output via openpmd-api in python, the following error occurs:

import openpmd_api as io
import numpy as np

series = io.Series(
    "/.../_LWFA_openPMD_adios/_run/001_LWFA_openPMD_bp_noDetails/simOutput/openPMD/simData_000000.bp",
    io.Access_Type.read_only)

the following error is given (using openPMD-api 0.12.0-alpha):

ValueError                                Traceback (most recent call last)
<ipython-input-8-219039f61eab> in <module>
----> 1 series = io.Series(
      2     "/.../_LWFA_openPMD_adios/_run/001_LWFA_openPMD_bp_noDetails/simOutput/openPMD/simData_000000.bp",
      3     io.Access_Type.read_only)

ValueError: ERROR: characteristic ID 1 not supported

cc'ing @ilomi

I will compile my own openPMD-api and avoid the system module on hemera as a first step.
@franzpoeschel @sbastrakov @psychocoderHPC How did you read the openPMD output in PIConGPU

[PrometheusPi]

This error might possibly be caused by using ADIOS1 instead of ADIOS2 - this will be my next check.

@benjha This might effect the data set, I created for your workshop. Did you try reading it already? I will test it as soon as possible.

[sbastrakov]

@PrometheusPi weird, I do not see a place in openPMD-api where such an error message (or rather word "characteristic") is thrown.

[PrometheusPi]

@sbastrakov I had the similar issue - I was looking through the openPMD-api warnings and found nothing.

[PrometheusPi]

With my self-build openPMD-api 0.12.0 the same error occurred (on hemera).

[sbastrakov]

Perhaps this error comes from ADIOS itself. E.g. this may be related: https://github.com/ornladios/ADIOS/blob/f33031ee206d552b286674d0f1641b9aca6b9100/src/core/adios_bp_v1.h#L24
(but I do not have experience with that, just resulted from a quick search through sources, and the comment at value 1 suggests it's not supported, but still a shot in the dark)

[PrometheusPi]

@benjha This error occurs in the data I provided you. I will switch to adios2 for your your data instead.

[franzpoeschel]

I unfortunately don't have a lot of time to look into this in detail right now, but two observations:

1. This error (most certainly) comes from ADIOS and not the openPMD API
2. The output appears to be in file-based iteration layout (the default in the openPMD plugin), which you would usually load as 'simData_%06T.bp' and not 'simData_000000.bp'.

If this doesn't help, I'll probably find some time for some error-tracking on wednesday.

[sbastrakov]

Ah, that's a nice observation from @franzpoeschel !
@KseniaBastrakova just last week struggled with the same thing

[sbastrakov]

So in PIConGPU output only the simData_%06T.bp is an openPMD series, but not a separate file. And whether a file is a series by itself or not is written in its attribute, which PIConGPU sets up corrrespondingly during the output.

[PrometheusPi]

If I use simData_%06T.bp instead, I still get the same error.

[benjha]

@ax3l

[PrometheusPi]

Despite the issue with the openPMD-api python module, the backend seems to produce valid ADIOS(1) output. Using bpls -r works fine and shows all data. I will check if the data can be loaded with the adios python module directly.

[PrometheusPi]

Update: there are no issues loading the openPMD plugin data via adios-python module. Either there is something broken in the python implementation of the openPMD-api or we miss an openPMD required attribute that causes a fatal crash. Either way, we do not write nonsense 🎉 - accessing the data is just less convenient right now.

[PrometheusPi]

@benjha The files provided for your workshop are valid ADIOS1 files.

[ax3l]

Is it possible that you are writing ADIOS1 files and that the Hemera module does not support ADIOS2? Let's request ADIOS 2.6.0 there and an openPMD 0.12.0 module.

If you load ADIOS1 data with openPMD-api, better "prefer" it's ADIOS1 backend for .bp handling for 100% parity:

export OPENPMD_BP_BACKEND="ADIOS1"

There are a thousand little nimbles from ADIOS internal compression to data model in openPMD that I cannot guarantee would work in a round-trip right now.

Note: ADIOS1 is effectively deprecated and end-of-life. I only keep the backend around so we can read old ADIOS1 plugin data. Please do not create new ADIOS1 data if not absolutely necessary.

[PrometheusPi]

@ax3l Thanks for pointing out the missing system variable.
If export OPENPMD_BP_BACKEND="ADIOS1" is set before loading the openpmd-api module, the error does not occur.

[PrometheusPi]

@benjha If reading our openPMD ADIOS1 output, setting OPENPMD_BP_BACKEND="ADIOS1" in your environment solves the issue. (Just tested it.)

[sbastrakov]

Did you check if hemera has only ADIOS1 or both 1 and 2? I am just a little confused, as the linked documentation suggests in the former case "ADIOS1" would be the default value. However, apparently setting manually fixed the problem, so either Hemera actually has ADIOS2, or maybe the documentation does not paint the full picture (maybe because of intricacies mentioned by @ax3l )

[PrometheusPi]

@sbastrakov I agree with you - from the documentation I thought that this only effects writing new files and that the openPMD-api is capable of determining given files on its own (or at least give a understandable warning). For PIConGPU, I just used the default ADIOS 1 - 1.13.1 on hemera. For python, I build via conda, thus both ADIOS1 and ADIOS2 are available. I am not sure to what ADIOS the openPMD module links to, most likely the 1.13.1. As far as I know, hemera only provides ADIOS 1 1.13.1 as module.

[sbastrakov]

Btw hemera also has python support in the same module. E.g. this works out of the box once the module (and its dependencies) is loaded, and outputs

openPMD-api: 0.12.0-dev
openPMD-api backend variants: {'adios1': True, 'adios2': False, 'hdf5': True, 'json': True, 'mpi': True}

[ax3l]

Did you check if hemera has only ADIOS1 or both 1 and 2? I am just a little confused, as the linked documentation suggests in the former case "ADIOS1" would be the default value. However, apparently setting manually fixed the problem, so either Hemera actually has ADIOS2, or maybe the documentation does not paint the full picture (maybe because of intricacies mentioned by @ax3l )

ADIOS2 is the default for .bp handling if both ADIOS2 and ADIOS1 support is present. If you only compile ADIOS1 and try to handle .bp files it will use ADIOS1 for that. If you only compile ADIOS2 and try to handle .bp files it will use ADIOS2 for that. If you compile both it will prefer ADIOS2 by default.

from the documentation I thought that this only effects writing new files and that the openPMD-api is capable of determining given files on its own (or at least give a understandable warning).

ADIOS might have missed a chance to call their ADIOS1 files .bp3 and ADIOS2 files .bp4 or newer. We currently don't probe individual files and decide on ADIOS by the file ending, since ADIOS2 has a BP3 engine to read old BP3 files. As it turns out, a lot of intricacies in the way the latter is implemented make it complicated to rely on it, otherwise we could easily fall back.

I think that ADIOS2 in newer versions actually does not care about the file name anymore at all. We could investigate to just call them ".bp4" on our own to relax this confusion. Maybe we can also probe a sample file before read for its file magic and try to force ADIOS1 for reads... I'll ask upstream (ADIOS devs) for advise.

Just for curiosity you could make a small reproducer ADIOS1 file (BP3) available and report it as an issue in openPMD-api for volunteers to jump in. I just tried it and cannot see the problem immediately when I try to read example ADIOS1 files through the ADIOS2 engine in openPMD-api. Maybe someone has time to investigate the exact error and we can work-around it a little. But rather switch the backend for now.

openPMD-api backend variants: {'adios1': True, 'adios2': False, 'hdf5': True, 'json': True, 'mpi': True}

Please request an ADIOS 2.6.0 module on Hermera and then they can build openPMD-api against it. There is no ADIOS2 support on Herma's openPMD-api module: 'adios2': False. Please point cluster support to the readme dependency list or the equivalent page in the manual - I guess their ADIOS2 module is too old or they did not provide it.

TL;DR

• do not write new ADIOS1 files please unless you absolutely must
• please add ADIOS2 to hemera and write ADIOS2 files; this is the default if you write .bp series and ADIOS2 is compiled in
• if you must read old ADIOS1 files, openPMD-api still supports this end-of-life format just so the transition is easier; but if you enabled both ADIOS2 and ADIOS1 in openPMD-api, then please switch at runtime to the ADIOS1 backend for reading with export OPENPMD_BP_BACKEND="ADIOS1" (you can also set this from Python)

[PrometheusPi]

As mentioned before, the issue cam from the fallback option using ADIOS1. With #3393 this has been solved on hemera. I will thus close this issue.