Openpmd-api: building with warpx (spack, HPC with `module`s)

[hightower8083]

Hi all (and in particular @ax3l )

I've been having some painful experience building warpx with openpmd support on our cluster at WIS. Here is what i'm doing:

• clone, bootstrap and source spack
• load a recent gcc compiler module load gcc/8.3.0
• find/add this [email protected] to spack and add lib paths to .spack/linux/compilers.yaml (that appeared to be necessary):

extra_rpaths:
- /apps/RH7U2/gnu/gcc/8.3.0/lib/gcc/x86_64-pc-linux-gnu/8.3.0/
- /apps/RH7U2/gnu/gcc/8.3.0/lib64/

• install openPMD-api via spack (set cmake to 3.13, cause default-now 3.15 is not stable):
  spack install openpmd-api %[email protected] -shared -json -python ^hdf5+mpi ^openmpi ^[email protected]
• go and build warpx with openPMD, make -j 4 DIM=2 USE_OPENPMD=TRUE

It does all compilation without any remarks and then, on the linking stage fails with:

/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(HDF5IOHandler.cpp.o): In function `openPMD::HDF5IOHandlerImpl::readAttribute(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)16>&)':
HDF5IOHandler.cpp:(.text+0x4830): undefined reference to `H5free_memory'
HDF5IOHandler.cpp:(.text+0x483a): undefined reference to `H5free_memory'
/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(HDF5IOHandler.cpp.o): In function `openPMD::HDF5IOHandlerImpl::createDataset(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)7> const&)':
HDF5IOHandler.cpp:(.text+0x751a): undefined reference to `H5P_CLS_DATASET_CREATE_ID_g'
/home/labs/malkalab/feigor/src/warpx_directory/openPMD-install/lib64/libopenPMD.a(ParallelHDF5IOHandler.cpp.o): In function `openPMD::ParallelHDF5IOHandlerImpl::ParallelHDF5IOHandlerImpl(openPMD::AbstractIOHandler*, ompi_communicator_t*)':
ParallelHDF5IOHandler.cpp:(.text+0x5ea): undefined reference to `H5P_CLS_DATASET_XFER_ID_g'
ParallelHDF5IOHandler.cpp:(.text+0x605): undefined reference to `H5P_CLS_FILE_ACCESS_ID_g'
ParallelHDF5IOHandler.cpp:(.text+0x625): undefined reference to `H5Pset_dxpl_mpio'
ParallelHDF5IOHandler.cpp:(.text+0x643): undefined reference to `H5Pset_fapl_mpio'
collect2: error: ld returned 1 exit status
make: *** [main2d.gnu.TPROF.MPI.OMP.ex] Error 1

this all looks like an ldconfig problem, but i've no idea how to tweak with it..

[ax3l]

Hi,

I am releasing a new openPMD-api version today that specifically is tailored to make warpx usage easier. Will update warpx as well. Stay tuned!

[hightower8083]

cool, will it be available at spack right away?

Anyway i still don't get the reason for this issue -- I have got it build on a few other machines (including macbook), and had no problems at all! Could it be gcc8.3 related thing or module load is failing to get correct linker flags (ones for HDF5)?

I've also tried manual building of openpmd-api (following the instructions ) , and it required to set CC and CXX paths for cmake:
CC=/apps/RH7U2/gnu/gcc/8.3.0/bin/gcc CXX=/apps/RH7U2/gnu/gcc/8.3.0/bin/g++ cmake ../openPMD-api -DopenPMD_USE_PYTHON=OFF -DopenPMD_USE_JSON=OFF -DCMAKE_INSTALL_PREFIX=../openPMD-install/ -DBUILD_SHARED_LIBS=OFF
otherwise it recognised only base system's [email protected] and found no openmpi and hdf5

I'll keep this issue open till will check the new API if its OK with you.

[ax3l]

Long story short: build +shared for now :)

[ax3l]

So, I am at the lab now.

CMake 3.15 & MPI

Addressed in https://github.com/spack/spack/pull/12129 with Spack very soon.

cool, will it be available at spack right away?

yes! actually, the spack default should also be +shared. Did you intentionally change that?

Don't disable json, it's fun and no real overhead/dep. to build! :)

build warpx

should be no problem with shared libs, but I am adding some additional handling for its (quite manual) build system with #532 this release. It's basically just lacking the propagation of public dependencies in shared builds.

I'll keep this issue open till will check the new API if its OK with you.

absolutely! :)

[hightower8083]

yes! actually, the spack default should also be +shared. Did you intentionally change that?

Don't disable json, it's fun and no real overhead/dep. to build! :)

nope, just followed the manual..

[ax3l]

Will update that as well then ;)

Ok, just do

module purge
spack uninstall --all openpmd-api
spack install openpmd-api
spack load openpmd-api

[hightower8083]

that did it, thanx @ax3l ! Will be waiting for more updates ! (e.g. particles dump 🙄)

[ax3l]

Wonderful!
Yes, that might become my job soon ;)

[ax3l]

CMake builds in spack are also fixed now, please update the spack source tree to get those into effect.

Thanks for reporting!

[hightower8083]

well, it might be not over just yet)) now i'm getting
./waprx.ex: error while loading shared libraries: libhdf5.so.8: cannot open shared object file: No such file or directorywhen i run it

[ax3l]

did you build warpx with spack (which would have an rpath)? Otherwise, spack load -r openpmd-api (note the -r).

[hightower8083]

don't know why haven't noticed it yesterday. So after all's installed, if i do

spack load openpmd-api +shared %[email protected] ^mpich

it does finish the WarpX build with SUCCESS (from the source, not spack) but linker gives a warning

/usr/bin/ld: warning: libhdf5.so.103, needed by /home/labs/malkalab/feigor/src/spack/opt/spack/linux-rhel7-x86_64/gcc-8.3.0/openpmd-api-develop-vd3ps4csd2a3zrrbdafrqmokffbk5cta/lib64/../lib64/libopenPMD.so, may conflict with libhdf5.so.8

(and i suspect that it means /usr/lib64/libhdf5.so.8)

Then when i run it interactively on the access machine, it crashes with

./waprx.ex: error while loading shared libraries: libgfortran.so.5: cannot open shared object file: No such file or directory

and if i submit it to our LSF, it returns

./waprx.ex: error while loading shared libraries: libhdf5.so.8: cannot open shared object file: No such file or directory

Adding the recursive load spack load -r does not affect this behaviour, and if i use spack load openpmd-api -shared %[email protected] ^mpich it falls back to the initial error, and if i build without openPMD all runs fine 🤔

[ax3l]

I think spack load -r openpmd-api should be sufficient, since you only installed one version (otherwise, install the unused ones).

You seam to build WarpX without the Spack build system, so it's using its very manual build system and picks up random things. Please make sure you load with -r so it sees the spack HDF5 then, then build again with that (and also run with loaded -r).

You might also want to experiment with USE_HDF5=FALSE or setting HDF5_HOME to the Spack HDF5 location.

[hightower8083]

well at some point it failed to build even variant without openPMD, so i've freaked out and have replaced all spack and sources =)
so i've :

• started from a fresh [email protected] built with spack using the default [email protected] -- there seems to be a bunch of things installed system-wide with it (e.g. hdf5)
• added mpich, hdf5^mpich, and openpmd-api -python ^[email protected] ^mpich.
• loaded hdf5 and openpmd-api with -r option

it still builds with a warning

/usr/bin/ld: warning: libhdf5.so.103, needed by /home/labs/malkalab/feigor/src/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/openpmd-api-develop-xndyau5eowkmts4a2l43vqrzvvtaxv4y/lib64/../lib64/libopenPMD.so, may conflict with libhdf5.so.8

and doing export HDF5_HOME=/path/to/spack/hdf5 before build does not affect it..

Now when i execute the code, it actually runs but till the openPMD use and fails with:

  Writing plotfile diags/plotfiles/plt00000
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: ../../src/H5P.c line 275 in H5Pcreate(): not a property list class
    major: Invalid arguments to routine
    minor: Inappropriate type

...

terminate called after throwing an instance of 'std::runtime_error'
  what():  Internal error: Failed to set HDF5 dataset transfer property
SIGABRT

There is a trace in Backtrace file, but i'm sure its appropriate to post it here.

well i assume that it just our system is haunted, and machines are always stronger than humans.. unless this new PR could resolve it ultimately Do you advise to pull it and check right away? 🤓

BTW, i've tried spack install warpx and it it fails to get it from github:

==> Cloning git repository: https://github.com/ECP-WarpX/WarpX.git at tag dev
Fetching tags only, you probably meant:
  git fetch --tags
error: pathspec 'dev' did not match any file(s) known to git.
==> Fetching from [git] https://github.com/ECP-WarpX/WarpX.git at tag dev failed.

adding @master gets the master brunch, but when i build it it gives bunch of warnings like that

fatal: Not a git repository (or any parent up to mount point /tmp)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

and when it reaches FieldIO

./Source/Diagnostics/FieldIO.cpp: In function 'std::unique_ptr<amrex::MultiFab> getInterpolatedScalar(const am$
./Source/Diagnostics/FieldIO.cpp:680:68: error: no matching function for call to 'amrex::NodeBilinear::interp($
                         finebx, refinement_vector, {}, {}, {}, 0, 0);

Not sure if its a glitch or some part of the curse 🤷‍♂️ And anyways is it possible to build openpmd version with spack? don't see it in spack info

UPD: this last part was fixed by spack install git, spack load git

[ax3l]

Can you disable the USE_HDF5=OFF of WarpX (make USE_HDF5=OFF ...)? That might be pulling the other HDF5 in.
Can you confirm?

If so, we can work on harmonizing what they decide to pick up in the Makefiles.

[hightower8083]

hmm, are you sure about this USE_HDF5=OFF flag (e.g. not USE_HDF5=FALSE)? I also had impression that hdf5 choice is hard-set in OPENPMD of WarpX.

Anyway, with openpmd-api built with default options (spack install openpmd-api), when I do make -j 1 DIM=2 USE_OPENPMD=TRUE USE_HDF5=OFF (or FALSE) it does not seem to propagate to compiler flags -- at the end WarpX linker still warns

/usr/bin/ld: warning: libhdf5.so.103, needed by /home/labs/malkalab/feigor/src/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/openpmd-api-develop-xndyau5eowkmts4a2l43vqrzvvtaxv4y/lib64/../lib64/libopenPMD.so, may conflict with libhdf5.so.8

and simulation starts but fails at openpmd diag writing with

HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: ../../src/H5P.c line 275 in H5Pcreate(): not a property list class
    major: Invalid arguments to routine
    minor: Inappropriate type

With another build, with -hdf5, the WarpX make finishes without warnings, but when I run example it also fails at writing diag (which makes sense) with

terminate called after throwing an instance of 'std::runtime_error'
  what():  openPMD-api built without HDF5 support
SIGABRT

[ax3l]

(e.g. not USE_HDF5=FALSE)?

Ah yes, it's hard coded and does not understand anything than that.

Anyway, with openpmd-api built with default options (spack install openpmd-api)

On which system are you computing? Can you show me what an ls in the installed lib directory shows? (Spack package's root location: spack find -p openpmd-api).

Are you certain you are using my branch from this PR: https://github.com/ECP-WarpX/WarpX/pull/239 (now in dev)?
What's in your environment's LIBRARY_PATH? Something manually added there?

[hightower8083]

it works! thanx for baring this one with me!

[ax3l]

Great to read that, you are welcome! :)