Openpmd-api: Segfault when reading libSplash file

Created on 5 Mar 2021 · 12Comments · Source: openPMD/openPMD-api

Describe the bug, To Reproduce
Steps:

Use the HDF5 plugin in PIConGPU to write data (that plugin is now removed, last commit containing it is 9180c81ba). Field data is sufficient to demonstrate.
Use Pip to download the current openpmd_api
Run openpmd-ls dump%T.h5 or similar
Observe a segfault

Expected behavior
Should not segfault. A self-built instance of the openPMD API reads the file just fine.

Software Environment

openpmd_api 0.13.2 from pip
operating system: observed on HZDR Hemera cluster (CentOS) and on a local x64 Ubuntu 20.10 machine
name and version of Python implementation: Python 3.8.6 from Ubuntu packages
version of *: whatever it is that the Pip package uses

Additional context
Valgrind seems to suggest there's something going wrong during attribute reading:

==164527== Invalid free() / delete / delete[] / realloc()
==164527==    at 0x483DA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==164527==    by 0x67B9850: openPMD::HDF5IOHandlerImpl::readAttribute(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)18>&) (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x67C445A: openPMD::AbstractIOHandlerImpl::flush() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x67B574D: openPMD::HDF5IOHandler::flush() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x6797021: openPMD::Attributable::readAttributes() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x6771E32: openPMD::RecordComponent::readBase() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x67A4FF4: openPMD::MeshRecordComponent::read() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x6765D57: openPMD::Mesh::read() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x67617BD: openPMD::Iteration::read() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x6778D2A: openPMD::Series::read() (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x677D550: openPMD::Series::readGroupBased(bool) (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x67804E6: openPMD::Series::init(std::shared_ptr<openPMD::AbstractIOHandler>, std::unique_ptr<openPMD::Series::ParsedInput, std::default_delete<openPMD::Series::ParsedInput> >) (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)

The final crash comes from an HDF5 sanity check:

==164527== Process terminating with default action of signal 6 (SIGABRT)
==164527==    at 0x4A718CB: raise (raise.c:50)
==164527==    by 0x4A56863: abort (abort.c:79)
==164527==    by 0x4A56748: __assert_fail_base.cold (assert.c:92)
==164527==    by 0x4A68A95: __assert_fail (assert.c:101)
==164527==    by 0x6F9FE16: H5MM_final_sanity_check (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)
==164527==    by 0x6E107F7: H5_term_library (in /home/franz/.local/lib/python3.8/site-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so)

Is there a reason why Valgrind gives me 10km of Python errors? Thought that openpmd-ls was C++ only? My locally built version of openpmd-ls runs without error under Valgrind.

What is the best way to share a reproducing file?

affects latest release HDF5 bug

Source

franzpoeschel

👍1

Most helpful comment

What is the best way to share a reproducing file?

Right, zipping is.
dump_750.zip

franzpoeschel on 10 Mar 2021

😄1 👍1

All 12 comments

What is the best way to share a reproducing file?

Right, zipping is.
dump_750.zip

franzpoeschel on 10 Mar 2021

😄1 👍1

Note to self: PIConGPU parameters for reproducing: -d 1 1 1 -g 28 28 28 --hdf5.period 50 --hdf5.file dump --hdf5.source fields_all.

franzpoeschel on 10 Mar 2021

👍1

Hm, cannot reproduce locally with latest dev.

Using latest pip:

docker run -it ubuntu:20.04

apt update
apt install -y python3 python3-pip wget unzip
python3 -m pip install openpmd-api

wget https://github.com/openPMD/openPMD-api/files/6115268/dump_750.zip
unzip dump_750.zip

openpmd-ls dump_%T.h5
# segfault



md5-ef8812c1eabd8ce248eeda514ced7851



python3 -m pip install openpmd-api==0.13.1
openpmd-ls dump_750.h5
# segfault

# 0.13.0: segfault
# 0.12.0a0: works
$ openpmd-ls dump_%T.h5
openPMD series: dump_%T
openPMD standard: 1.0.0
openPMD extensions: 1

data author: franz
data created: 2021-03-05 10:48:53 +0000
data backend: HDF5
generating machine: unknown
generating software: PIConGPU (version: 0.5.0-dev)
generating software dependencies: unknown

number of iterations: 1 (fileBased)
  all iterations: 750 

number of meshes: 8
  all meshes:
    B
    E
    e_chargeDensity
    e_energyDensity
    e_particleMomentumComponent
    i_chargeDensity
    i_energyDensity
    i_particleMomentumComponent

number of particle species: 0

ax3l on 8 Apr 2021

Looking at the core dump via:

gdb /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so /core

leads to

#0  0x00007ff8418e28a2 in __GI___libc_free (mem=<optimized out>)
    at malloc.c:3124
3124    malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ff8418e28a2 in __GI___libc_free (mem=<optimized out>)
    at malloc.c:3124
#1  0x00007ff84032f851 in openPMD::HDF5IOHandlerImpl::readAttribute(openPMD::Writable*, openPMD::Parameter<(openPMD::Operation)18>&) ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#2  0x00007ff84033a45b in openPMD::AbstractIOHandlerImpl::flush() ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#3  0x00007ff84032b74e in openPMD::HDF5IOHandler::flush() ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#4  0x00007ff84030d022 in openPMD::Attributable::readAttributes() ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#5  0x00007ff8402e7e33 in openPMD::RecordComponent::readBase() ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#6  0x00007ff84031aff5 in openPMD::MeshRecordComponent::read() ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#7  0x00007ff8402dbd58 in openPMD::Mesh::read() ()
--Type <RET> for more, q to quit, c to continue without paging--
on-38-x86_64-linux-gnu.so
#8  0x00007ff8402d77be in openPMD::Iteration::read() () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#9  0x00007ff8402eed2b in openPMD::Series::read() () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#10 0x00007ff8402f3551 in openPMD::Series::readGroupBased(bool) () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#11 0x00007ff8402f64e7 in openPMD::Series::init(std::shared_ptr<openPMD::AbstractIOHandler>, std::unique_ptr<openPMD::Series::ParsedInput, std::default_delete<openPMD::Series::ParsedInput> >) ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#12 0x00007ff8402f6859 in openPMD::Series::Series(std::string const&, openPMD::Access, std::string const&) ()
   from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#13 0x00007ff84026e47b in ?? () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#14 0x00007ff84026f4cc in ?? () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#15 0x00007ff8402ce3e7 in ?? () from /usr/local/lib/python3.8/dist-packages/openpmd_api/openpmd_api_cxx.cpython-38-x86_64-linux-gnu.so
#16 0x00000000005f2fb9 in int adios2::helper::NdCopy<signed char>(char const*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, bool, bool, char*, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, bool, bool, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, bool) ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

This seems to be the same as your valgrind analysis.

ax3l on 8 Apr 2021

Since this file has no particles, we could rework this section:
https://github.com/openPMD/openPMD-api/blob/6fd39365ff4bf6e350faec66a5ebac2f3804e8dd/src/Iteration.cpp#L387-L402

and even for 1.0.0 and 1.0.1 make a check (and warn). That would make things a bit more robust.

But the crash itself seems to be in mesh reads...

ax3l on 8 Apr 2021

This is the pure C++ version of openpmd-ls:

#include <openPMD/openPMD.hpp>
#include <iostream>

using namespace openPMD;


int main(){
    Series s = Series(
        "dump_%T.h5",
        Access::READ_ONLY
    );

    helper::listSeries(s, true, std::cout);
}

We could run this again with sanitizers enabled.

For pip installs, we ship a Python wrapper for it (because .local/bin is not in PATH on all distributions + various limitations of setuptools/pip)

ax3l on 8 Apr 2021

pip builds run quay.io/pypa/manylinux2010_x86_64 containers using GNU 8.3.1 and a relatively old glibc, IIRC. The HDF5 version is 1.12.0. For our 0.12.0 release, we were still using HDF5 1.10.5 (update in 81d5ca712c88c4bb6221da43c9ea983c19510e53).

https://github.com/openPMD/openPMD-api/runs/1817947928

ax3l on 8 Apr 2021

Local test with pure C++ version above and HDF5 1.12.0 does not fail.

Compiled again with UBSAN & ASAN, also no fail.

Release 0.13.0+ is also the release where we switched to C++14...

ax3l on 8 Apr 2021

Let's try a debug build in the exact same environment:

docker run -it quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de

yum install -y wget git glibc-static tar gdb
# yum install yum-utils
# debuginfo-install glibc

cd /opt
wget https://raw.githubusercontent.com/openPMD/openPMD-api/wheels/library_builders.sh
git clone https://github.com/openPMD/openPMD-api.git

export PATH=/opt/python/cp38-cp38/bin:$PATH

# vi library_builders.sh: fix ZFP link to
#   https://github.com/LLNL/zfp/releases/download/0.5.5/zfp-0.5.5.tar.gz
bash library_builders.sh

cd openPMD-api
# vi CMakeLists.txt: remove LTO for Python bindings
#   pybind11::lto

cat <<EOT >> setup.cfg
[build_ext]
debug = 1
EOT

cat <<EOT > src/cli/ls.cpp
#include "openPMD/openPMD.hpp"
#include <iostream>

using namespace openPMD;


int main(){
    Series s = Series(
        "dump_%T.h5",
        Access::READ_ONLY
    );

    helper::listSeries(s, true, std::cout);
}
EOT

cmake -S . -B build -DopenPMD_USE_MPI=OFF -DopenPMD_USE_PYTHON=OFF -DCMAKE_BUILD_TYPE=Debug
cmake --build build -j 6
cd build/bin

#python -m pip wheel -v .
#python -m pip install *whl
#cd ..

wget https://github.com/openPMD/openPMD-api/files/6115268/dump_750.zip
unzip dump_750.zip

# gdb -ex r --args python -m openpmd_api.ls dump_%T.h5
gdb -ex r --args openpmd-ls dump_%T.h5
# bt



md5-85169c7908089a3a9df1ff9eae16d6b9



#0  0x00007f9d9520595c in free () from /lib64/libc.so.6
#1  0x00007f9d96c270b2 in openPMD::HDF5IOHandlerImpl::readAttribute (this=0x1e1d650, writable=0x1ebaa68, parameters=...) at /opt/openPMD-api/src/IO/HDF5/HDF5IOHandler.cpp:1468
#2  0x00007f9d96c2c164 in openPMD::AbstractIOHandlerImpl::flush (this=0x1e1d650) at /opt/openPMD-api/include/openPMD/IO/AbstractIOHandlerImpl.hpp:105
#3  0x00007f9d96c2b9f6 in openPMD::HDF5IOHandler::flush (this=0x1e1d500) at /opt/openPMD-api/src/IO/HDF5/HDF5IOHandler.cpp:1836
#4  0x00007f9d96bd1ac7 in openPMD::AttributableImpl::readAttributes (this=0x1ebb0e8) at /opt/openPMD-api/src/backend/Attributable.cpp:181
#5  0x00007f9d96b4bc32 in openPMD::RecordComponent::readBase (this=0x1ebb0e8) at /opt/openPMD-api/src/RecordComponent.cpp:370
#6  0x00007f9d96c1492e in openPMD::MeshRecordComponent::read (this=0x1ebb0e8) at /opt/openPMD-api/src/backend/MeshRecordComponent.cpp:62
#7  0x00007f9d96b2bc25 in openPMD::Mesh::read (this=0x1eb2038) at /opt/openPMD-api/src/Mesh.cpp:364
#8  0x00007f9d96ac56fb in openPMD::Iteration::read_impl (this=0x1e8b2e8, groupPath="750") at /opt/openPMD-api/src/Iteration.cpp:441
#9  0x00007f9d96ac4635 in openPMD::Iteration::readFileBased (this=0x1e8b2e8, filePath="dump_750.h5", groupPath="750") at /opt/openPMD-api/src/Iteration.cpp:333
#10 0x00007f9d96ac4549 in openPMD::Iteration::read (this=0x1e8b2e8) at /opt/openPMD-api/src/Iteration.cpp:316
#11 0x00007f9d96ac698a in openPMD::Iteration::runDeferredParseAccess (this=0x1e8b2e8) at /opt/openPMD-api/src/Iteration.cpp:635
#12 0x00007f9d96b7d0bf in openPMD::SeriesImpl::<lambda(openPMD::Iteration&)>::operator()(openPMD::Iteration &) const (__closure=0x7ffdb5dcee9f, iteration=...) at /opt/openPMD-api/src/Series.cpp:792
#13 0x00007f9d96b7d71d in openPMD::SeriesImpl::readFileBased (this=0x1e1d240) at /opt/openPMD-api/src/Series.cpp:815
#14 0x00007f9d96b7b3d3 in openPMD::SeriesImpl::init (this=0x1e1d240, ioHandler=std::shared_ptr (count 9) 0x1e1d500, input=...) at /opt/openPMD-api/src/Series.cpp:438
#15 0x00007f9d96b81058 in openPMD::internal::SeriesInternal::SeriesInternal (this=0x1e1d120, filepath="dump_%T.h5", at=openPMD::READ_ONLY, options="{}") at /opt/openPMD-api/src/Series.cpp:1301
#16 0x00007f9d96b95dfb in __gnu_cxx::new_allocator<openPMD::internal::SeriesInternal>::construct<openPMD::internal::SeriesInternal, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (this=0x7ffdb5dcf2df, __p=0x1e1d120, __args#0="dump_%T.h5", __args#1=@0x7ffdb5dcf4ac, __args#2="{}")
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/ext/new_allocator.h:136
#17 0x00007f9d96b92e1a in std::allocator_traits<std::allocator<openPMD::internal::SeriesInternal> >::construct<openPMD::internal::SeriesInternal, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (__a=..., __p=0x1e1d120, __args#0="dump_%T.h5", __args#1=@0x7ffdb5dcf4ac, __args#2="{}")
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/alloc_traits.h:475
#18 0x00007f9d96b90a04 in std::_Sp_counted_ptr_inplace<openPMD::internal::SeriesInternal, std::allocator<openPMD::internal::SeriesInternal>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (this=0x1e1d110, __a=...)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr_base.h:545
#19 0x00007f9d96b8ebb1 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<openPMD::internal::SeriesInternal, std::allocator<openPMD::internal::SeriesInternal>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (this=0x7ffdb5dcf500, __p=@0x7ffdb5dcf4f8, __a=...)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr_base.h:677
#20 0x00007f9d96b8d246 in std::__shared_ptr<openPMD::internal::SeriesInternal, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<openPMD::internal::SeriesInternal>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (this=0x7ffdb5dcf4f8, __tag=...)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr_base.h:1342
#21 0x00007f9d96b8b46d in std::shared_ptr<openPMD::internal::SeriesInternal>::shared_ptr<std::allocator<openPMD::internal::SeriesInternal>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (this=0x7ffdb5dcf4f8, __tag=...)
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr.h:359
#22 0x00007f9d96b8920b in std::allocate_shared<openPMD::internal::SeriesInternal, std::allocator<openPMD::internal::SeriesInternal>, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (__a=..., __args#0="dump_%T.h5", __args#1=@0x7ffdb5dcf4ac, __args#2="{}")
    at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr.h:706
#23 0x00007f9d96b8665a in std::make_shared<openPMD::internal::SeriesInternal, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, openPMD::Access&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&> (__args#0="dump_%T.h5", __args#1=@0x7ffdb5dcf4ac, __args#2="{}") at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/shared_ptr.h:722
#24 0x00007f9d96b81392 in openPMD::Series::Series (this=0x7ffdb5dcf4e0, filepath="dump_%T.h5", at=openPMD::READ_ONLY, options="{}") at /opt/openPMD-api/src/Series.cpp:1350
#25 0x000000000059dfa5 in main () at /opt/openPMD-api/src/cli/ls.cpp:11

ax3l on 8 Apr 2021