Openpmd-api: H5Tset_size(): size must be positive

[LDAmorim]

Hi,

I have been using openpmd-api (much appreciated tool!), but, in a recent installation, my code got an error message connected to it (well HDF5) and I tested using your example file and found that same error.
Since I am not able to understand how to solve it and I haven't found similar topics in other issues/PRs (here, or in spack or in HDF5's repos), I decided to open this issue.

Performed steps
Installed spack with the commands below (directly copied from history command with added comments):

(...)
1909  spack install gcc
 1910  spack compiler find
# Forgot to load it first ...
 1911  spack load gcc
 1912  spack compiler find
 1913  gcc --version
# Output was 11.1.0
 1914  spack install cmake
 1915  spack load cmake
 1916  spack install mpich
 1917  spack load mpich
 1918  vi /home/lidipiamorim/work/opt/spack/etc/spack/defaults/packages.yaml
# Changed line to mpi: [mpich, openmpi] so that spack compiles and runs with mpich instead of openmpi by default
 1919  spack install openpmd-api@dev
# Previous line gave an error (since I've been changing multiple spack/hdf5/openpmd-api versions to solve the issue I am reporting here, I just disregarded the dev branch option)
 1920  spack clean --all
 1921  spack install openpmd-api
# Successful installation (apparently) so I continued to the python installation of it
(...)

Then I used the following files (the .cc one I copied from https://openpmd-api.readthedocs.io/en/latest/usage/parallel.html#writing) - except with correct extensions (instead of .txt):

opmd_build.sh.txt
opmd_run.sh.txt
test_opmd.cc.txt
CMakeLists.txt

This led to the output file and errors below:

output.txt

HDF5-DIAGHDF5-DIAG: Error detec: Error detected in ted in HDF5 (1.10.7) MPI-process 0:HDF5 (1.10.7) MPI-process 1:
  #000
  #000: H5T.c line: H5T.c line 2159 i 2159 in H5Tset_size(): size must be positive
n H5Tset_size(): size must be positive
    maj    major: Invalid arguments to routine
    mior: Invalid arguments to routine
    minor: nor: Bad value
Bad value
HDF5-DIAG:HDF5-DIAG: Error  Error detected in detected in HDF5HDF5 (1.10.7) MPI-pro (1.10.7) MPI-proccess 1:
  #000ess 0:
  #000: H5T.c: H5T.c line 2159 in  line 2159 in H5Tset_size(): size must be positive
    major:H5Tset_size(): size must be positive
    majo Invalid arguments to routine
    minor: Bad valuer: Invalid arguments to routine
    minor:
 Bad value
HDF5-DIAG: Error HDF5-DIAG: detected in Error detected HDF5 (1.10.7) MPI-procein HDF5 (1.10.7) MPI-pss 1:
  #000: H5T.c lrocess 0:
  ine 2159 in H5Tset_size()#000: H5T.c line 2159: size must be positive
    major: Invalid arguments to routine
     in H5Tset_size(): size must be positive
    mminor: Bad value
ajor: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.10.7) MPI-process 1:
  #000: H5T.c line 2159 in H5Tset_size(): size must be positive
    major: Invalid arguments to routine
    minor: HDF5-DIAG: Error detected in HDF5 (1.10.7) MPBad value
I-process 0:
  #000: H5T.c HDF5-DIAG: Error dline 2159 in H5Tset_size(): size must be positive
etected in HDF5 (1.10.7) M    major: Invalid arguments to routine
    minoPI-process 1:
r: Bad value
  #000: H5T.c line 2159 in H5Tset_size(): HDF5-DIAG: size must be positive
    major: Invalid arguments to routine
    miError detected nor: Bad value
in HDF5 (1.10.7) MPI-proceHDF5-DIAG: Errss 0:
  #000: H5T.c lor detected in HDF5 ine 2159 in H5Tset_size(): (1.10.7) MPI-processsize must be positive
    major: Invalid arguments to routine
    m 1:
  #000: H5T.c lininor: Bad value
e 2159 in H5Tset_size(): size must be positive
    major: Invalid arguments to routine
    minoHDF5-DIAG: Error: Bad value
r detected in HDF5 (1.10.7) MPI-process 0:
  #000: H5T.c line 2159 in H5Tset_size(): size must be positive
HDF5-DIAG: Error    major: Invalid arguments to routine
    mino detected in HDF5 (r: Bad value
1.10.7) MPI-process 1:
  #000: H5T.c line 2159 in H5Tset_size(): size must be positive
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.10.7) MPI-process 0:
  #000: H5T.c line 2159 in H5Tset_size(): size must be positive
    major: Invalid arguments to routine
    minor: Bad value

These errors didn't occur if I commented out everything in test_opmd.cc between the "Ctreated an empty series ..." cout and series.flush (i.e. lines 45 to 72).
And occurred when I kept the iterations mesh construction (i.e. un-commenting lines 45 to 50).

These errors also don't seem to appear with the serial example.

System environment

• operating system: Ubunto (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0)
• machine: Local Windows 10
• name and version of Python implementation: python installed via spack
• output of spack debug report:

• Spack: 0.16.2-3140-ca1d1c427c
• Python: 3.8.2
• Platform: linux-ubuntu20.04-skylake
• Concretizer: original

Additional context
I will try to follow @ax3l 's suggestion of using environments with spack and will report back on it asap.

[LDAmorim]

Update on trying to use an environment: didn't go very well - in https://github.com/spack/spack/issues/24520

[ax3l]

Hi Diana,

I tested your example and that looks good to me. I think that's a system issue mismatching compiler and package for your Ubuntu/WSL setup. I commented a spack suggestion now in the linked spack issue.

Alternatively, you should be able to spin up a full development env from the Ubuntu/WSL packages alone:

sudo apt update
sudo apt install build-essential ca-certificates cmake curl g++ gnupg libopenmpi-dev libhdf5-openmpi-dev libadios-dev python3 python3-numpy python3-mpi4py python3-pandas pkg-config wget

[LDAmorim]

Thanks @ax3l ! I just saw your reply. I'm testing it now.

[LDAmorim]

Hi @ax3l ,

Your suggestion seems to work.
I didn't report this back because I was finding an issue with 1) spack activate openpmd-api in the environment where I build it with python.
The workaround I found was just to load the module every time I use that environment.
In case you're curious this was the environment I made:

$ spack find -c
==> In environment jn
==> Root specs
git  graphviz  mercurial  openpmd-api +python  py-matplotlib  py-notebook  py-numpy  py-scipy  py-sphinx  py-sphinx-rtd-theme  py-sphinxcontrib-programoutput  py-sympy  python

==> Concretized roots
-- linux-ubuntu20.04-skylake / [email protected] ------------------------
[email protected]       [email protected]       [email protected]  [email protected]  [email protected]            [email protected]  [email protected]
[email protected]  [email protected]  [email protected]    [email protected]   [email protected]  [email protected]

And it did had some conflict notice when updating the view, but it seems to work well in spite of that.

The other issue was with loading data - I was getting everything = 0 - but I think this is a problem on my side that I think I know how to fix.

So you may close this issue if you want!
Thank you for all your help

[ax3l]

Hi @LDAmorim, thanks for the update! Glad to hear you made good progress! :tada:

I was finding an issue with 1) spack activate openpmd-api in the environment where I build it with python.
...
And it did had some conflict notice when updating the view

If you install everything in one go, do you still get conflicts?

sudo apt update
sudo apt install build-essential curl gcc git g++ make cmake

Then this:

spack env create openpmd-dev
spack env activate openpmd-dev
spack add cmake
spack add mpich
spack add openpmd-api
spack add python
spack add py-pip
spack install

Then optionally:

python -m pip install matplotlib yt openpmd-viewer # etc etc ...

Adding more software after a first spack install can sometimes be troublesome (a problem that will be eased a lot in the next Spack releases with the new clingo concretizer.) So make sure to add all the things you need from spack before you hit spack install for best results.

The other issue was with loading data - I was getting everything = 0 - but I think this is a problem on my side that I think I know how to fix.

Curious, feel free to share more if need be.
Generic tip with zeros in data: Make sure that you flush() before reading.

Glad to see you got further!

[LDAmorim]

Hi @ax3l ,

So I think I did the "one go" option, but I might have been confusing on my previous post, sorry for that.

I did the update of the ubunto gcc as you had mentioned before.
I cleaned everything I previously had from spack to make a completely new install.
Then I created several different environments as I need:

1) for simulations with openpmd-api + mpich + gcc + cmake (worked completely well)
2) for python analysis with python + matplotlib + scipy + sympy + openpmd-api + python notebook
3) sphynx (which I ended up having to merge with previous one)
4) doxygen (worked completely well)

Only the python one gave a conflict and it still worked, it just didn't update the final view.
So I just need to run spack load openpmd-api every time I use it.
But it works well after that!

This is what I get from spack find -c (as far as concretized roots) in it, in case you need this information:

==> Concretized roots
-- linux-ubuntu20.04-skylake / [email protected] ------------------------
[email protected]   [email protected]  [email protected]  [email protected]
[email protected]  [email protected]    [email protected]

---

Regarding the curious incident, I think I might have discussed this before with you (sorry!), when writing openPMD files from say 2 parallel ranks, if one rank has for example 3 particles and the other has none (and so doesn't know for example the particles mass), then I can't use .makeConstant() as I do .storeChunk(). Because using it lead to the run stalling (no error message).

Instead I just need to merge the information of the variable (say mass) from the ranks with particles to all of the ranks, to use .makeConstant().

Thank you for all your help @ax3l !
Please feel free to let me know if I didn't understand your message or if I was confusing in this reply.

[ax3l]

Only the python one gave a conflict and it still worked, it just didn't update the final view.
So I just need to run spack load openpmd-api every time I use it.

Good one - you could post the exact packages that you created for this environment as a Spack issue, that's probably an upstream issue that can be figured out.

In the meantime, you could build a simple environment like this for 2.) that gets most things from pypi:

spack env create pythonstuff
spack env activate pythonstuff
spack add openpmd-api +python
spack add python
spack add py-pip
spack install
# this should hopefully not yet create a conflict

# get the rest via:
python -m pip install matplotlib yt openpmd-viewer sympy scipy jupyter

That will still install in the spack environment and you re-activate with spack env activate pythonstuff.

Regarding the curious incident, I think I might have discussed this before with you (sorry!), when writing openPMD files from say 2 parallel ranks, if one rank has for example 3 particles and the other has none (and so doesn't know for example the particles mass), then I can't use .makeConstant() as I do .storeChunk(). Because using it lead to the run stalling (no error message).

Instead I just need to merge the information of the variable (say mass) from the ranks with particles to all of the ranks, to use .makeConstant().

Thank you for all your help @ax3l !
Please feel free to let me know if I didn't understand your message or if I was confusing in this reply.

That's interesting and could be similar to #961 .
Do you write .makeConstant() from all MPI ranks?
We have a new Iteration::open() method (in dev/0.14.0) that you could call first #862 in case you do non-collective writes (i.e., some ranks do not call the storeChunk() or flush() methods.

[ax3l]

I just tried the latter and there is more to it.
If you write HDF5, both setAttribute as well as makeConstant need to be called from all MPI ranks. That's a limitation of the HDF5 data model that we have no control over:
https://openpmd-api.readthedocs.io/en/latest/details/mpi.html

So even if you write zero particles from some MPI-ranks, make sure that all of them go through the code path that calls setAttribute and makeConstant calls.

(and so doesn't know for example the particles mass)
...
Instead I just need to merge the information of the variable (say mass) from the ranks with particles to all of the ranks, to use .makeConstant().

Yes, exactly. Given that limitation for HDF5 attributes (which we use for constants as well), this means you need to MPI_Bcast those constants before your write, sorry for that inconvenience.

[LDAmorim]

Hi,
Thanks for the replies!

Yes, that's a great idea to use py-pip instead and still very clean. I'll try it.
Please feel free to close this issue, whenever you want since we got several options to solve it now.

Regarding the write issue, it is just as you put it in your last message.
After merging the variable information to get it available to all MPI ranks, I was able to call makeConstant() from all of them and generate the openPMD files correctly.
This was not the case with variables like position of particles, using storeChunk(), where c_offset and c_extent are specified.
Thank you for explaining the HDF5 view of this, it does make sense.

Also, I didn't use MPI_Bcast before because I wasn't sure a priori which ranks could end up having data ... but I'll try it as well now, thanks!

[ax3l]

Glad this helped you.

Also, I didn't use MPI_Bcast before because I wasn't sure a priori which ranks could end up having data ... but I'll try it as well now, thanks!

That's right, broadcast only makes sense if you know for sure which rank has at least one particle (and thus knows the mass) upfront.