Picongpu: ADIOS Particles are not fully openPMD-compatible

[KseniaBastrakova]

I create ADIOS checkpoints during the simulation.

TBG_checkpoint="--checkpoint.period 1000 --checkpoint.backend adios"
I got some ADIOS file, as a result.

checkpoint_0.bp                 
checkpoint_0.bp.dir             
checkpoint_1000.bp              
checkpoint_1000.bp.dir          
checkpoint_2000.bp
.....

When I tryed open one of ADIOS file, i got error:
series = openpmd_api.Series(checkpoint_0.bp, openpmd_api.Access_Type.read_only)

Series constructor called with explicit iteration suggests loading a single file with groupBased iteration encoding. Loaded file is fileBased.
Traceback (most recent call last):
File "hdf_to_gdf.py", line 488, in
hdf_to_gdf(args.hdf, args.gdf, args.max_cell, args.species)
File "hdf_to_gdf.py", line 31, in hdf_to_gdf
series_hdf = openpmd_api.Series(hdf_file_directory, openpmd_api.Access_Type.read_only)
RuntimeError: Unexpected Attribute datatype for 'timeOffset'

i'm using ADIOS 1.13.1
And openPMD latest (d68e9fc2),
PicOnGPU latest (d340812)

[sbastrakov]

@KseniaBastrakova as far as I understand, with HDF5 checkpoints it all works fine?

[ax3l]

Oh dang, I think the 7 should read 1 here:
https://github.com/ComputationalRadiationPhysics/picongpu/blob/5373b6e64decb304f42e25a4b69f06e51d38222a/include/picongpu/plugins/adios/writer/ParticleAttributeSize.hpp#L139

Copy-paste-upsi :)

@KseniaBastrakova do you like to change this and provide a PR for it?

[PrometheusPi]

Great finding @KseniaBastrakova :+1: .
I assume that this bug would probably also prevent restarting from ADIOS files. Is this correct?
(I currently encounter restart issues with ADIOS on taurus ml - but I did not yet check the readability of the adios files.)

[sbastrakov]

@PrometheusPi I don't think it does, since restarting currently uses ADIOS (or HDF5) directly, not via openPMD-API. As long as our writer is consistent with the reader, should be fine.

[KseniaBastrakova]

Oh dang, I think the 7 should read 1 here:

https://github.com/ComputationalRadiationPhysics/picongpu/blob/5373b6e64decb304f42e25a4b69f06e51d38222a/include/picongpu/plugins/adios/writer/ParticleAttributeSize.hpp#L139

Copy-paste-upsi :)

@KseniaBastrakova do you like to change this and provide a PR for it?

Yes, I'm going to do that

[ax3l]

@KseniaBastrakova thanks, please feel free to ping me on the PR. Any chance to get this done today already? :)

Re: Restarts.
This only affects a particle attribute that is mostly ignored, especially in our restarts. We do not have a finished validator script for BP3 files, which is why this slipped in.

[sbastrakov]

@ax3l today @KseniaBastrakova is traveling, so I doubt.

[ax3l]

No problem, opened in #2312 because I need it myself, too.

Testing welcome, will go to sleep now ;)

[sbastrakov]

Fixed by #3120.

[ax3l]

Did you try this is readable with openPMD-api now?

[sbastrakov]

@ax3l good point, sorry for closing too early.

[ax3l]

Quick test on my data before/after #3120. Reading a series that only contains:

• [x] fields (before): ok
• [x] particles (before): fail with above error
• [x] fields (after): ok
• [x] particles (after): waiting for my simulation to generate data... still failing with another error.

[sbastrakov]

@ax3l has your test simulation finished?

[ax3l]

Still have this error:

ERROR: Attribute '/data/40000/particles/e_onLaserAxis/particles_info/unitSI' is not found!
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-08c242bf7112> in <module>
----> 1 s = api.Series("electrons_%T.bp", api.Access_Type.read_only)

RuntimeError: Internal error: Failed to get ADIOS1 attribute during attribute read

We could probably ignore this and throw a warning instead in openPMD-api... Although currently we do not encourage to store non-openPMD-data inside the reserved paths that are parsed, so this would not pass the validator nor reading of the standard anyway: https://github.com/openPMD/openPMD-standard/issues/115

The main problem here is that ADIOS particles do not yet use openPMD particle patches, I never had time to finish that last part in the ADIOS I/O set since it affects restarts and I already spend a lot of time making the backend safe under changing MPI distributions between runs. Particle patches also solve that, someone just has to do it.

Quick fix for PIConGPU: move the particles_info entries to a path that is not interpreted by openPMD, as documented in https://github.com/openPMD/openPMD-standard/issues/115 above.

Potential hack for PIConGPU: add the required attributes, although useless, to the particles_info variable. It does not have proper particle shape (extent) though, which could cause some more problems down the line.

Quick hack for existing data: remove the /data/<step>/particles/<species>/particles_info object or add the attributes manually. Removing the particles_info object in restart files will render them non-restartable, so maybe backup the data first.

I will in parallel also look if we can hack something in the API to continue parsing on invalid records, but do not have much time this week so no guarantees: https://github.com/openPMD/openPMD-api/issues/620

[ax3l]

@KseniaBastrakova I pushed work-around that ignores the particles_info object of PIConGPU to openPMD-api:
https://github.com/openPMD/openPMD-api/pull/635 (merged to dev)

Still your observation is valid: this object should not be in that location and violates openPMD, so we can leave this issue open until this is fixed.

You can test the development version of openPMD-api either by

• building from source or
• loading all dependencies, such as ADIOS1, and installing via pip install git+https://github.com/openPMD/openPMD-api.git@dev or
• installing via spack install openpmd-api@develop

Feedback if this patch fixes your problem (and/or causes new problems) is appreciated.

[KseniaBastrakova]

thanks, @ax3l , now I ran into other problems see here, after solving them I will try

[ax3l]

I just released 0.10.2 to aide you further and safe you from compiling yourself: https://github.com/openPMD/openPMD-api/issues/636#issuecomment-566757281

Just run

python3 -m pip install --upgrade openpmd-api

as usual and this will contain the fix so you can try quicker.

[KseniaBastrakova]

I still have this problem after update 0.10.2
Reading with:

series_adios = openpmd_api.Series(adios_name, openpmd_api.Access_Type.read_only)

Still results in the same error

RuntimeError: Unexpected Attribute datatype for 'timeOffset'

I have checked that version is __version__ = '0.10.2-alpha' in init.py

[ax3l]

Confusing, I tested this on my own files and had no problems.
Did you write new files with PIConGPU and included the bugfix from PR #3120?

[KseniaBastrakova]

Confusing, I tested this on my own files and had no problems.
Did you write new files with PIConGPU and included the bugfix from PR #3120?

now, I'm pretty sure, that it was my mistake, sorry.

I tested it again, everything works and I had no problems on the last version PIConGPU

[ax3l]

Hooray, that's great! :)