Openpmd-api: Clang linker error: "missing vtable"

[LDAmorim]

Hi!
Sorry to bother you, but I can't understand the error that I am getting very well.
It might be that this is a duplicate, please feel free to delete this issue if it is, but I couldn't understand if the answer was elsewhere written.

I have a CPP small code that I am running in which I wish to store the results (for example my particles energy) in an OpenPMD file format.

When I compile that code, I get the long message that is in the file: error.txt. This message contains some parts that I think are the errors, such as:

"vtable for openPMD::PatchRecord", referenced from:
      openPMD::Container<openPMD::PatchRecord, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, openPMD::PatchRecord> > > >::operator[](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in functions.cc.o
      openPMD::Container<openPMD::PatchRecord, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, openPMD::PatchRecord> > > >::operator[](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) in functions.cc.o
      std::__1::unique_ptr<std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord>, void*>, std::__1::__tree_node_destructor<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord>, void*> > > > std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord>, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, openPMD::PatchRecord> > >::__construct_node<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, openPMD::PatchRecord> >(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, openPMD::PatchRecord>&&) in functions.cc.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.

From what I am reading in the internet this kind of issue suggests something off with my header / macro / destructor information. But I don't understand how to solve it.

If you want to reproduce this issue you can just use the function (the only part of my code that contains OpenPMD) where I am trying to store the vector (energy) into a file.
It is basically a copy paste from the serial write example in the OpenPMD-api repo.
func_opmd.txt

The associated header file is:
func_opmd_hh.txt

I am probably making a very basic mistake, but I have tested different things and can't understand the source of the error.
Do you have any suggestions or information you can point me to to test this?
Thanks,
Diana

Software Environment

• openPMD-api: 0.10.1%clang
• installed openPMD-api via: spack (with -python flag)
• operating system: OS 10.15 (Catalina)
• machine: Macbook (clang 11.0.0)
• name and version of Python implementation: python 2.7.16 (in-built)
• version of HDF5: 1.10.5%clang
• version of ADIOS1: (I think there is none... but I installed everything with spack)
• name and version of MPI: openmpi 3.1.4 %clang
  
  
  
  • cmake 3.15.5 (installed with spack)
  
  

[LDAmorim]

ps: This is probably just a bug on the code I wrote to use OpenPMD and not on OpenPMD itself!

[ax3l]

Oh wow, C++ is hard.
I think it's a problem on the openPMD-api side and will try to double check the declarations of virtual once more.
What version of XCode are you using? I'll add a newer one to the CI as well.

[LDAmorim]

Thanks for the super quick reply!
Yes, I agree! I am "a bit" lost with C++.
I am trying to see if it is a problem with HDF5 or with other parts that OpenPMD-api might use.
My computer's Xcode version is 11.2.1 (11B500).

[ax3l]

the thing that makes it a bit hard here is that we use runtime polymorphism - which is a bit tricky in C++ and we probably forgot some qualifiers somewhere.
I'll investigate :)

[LDAmorim]

Oh, yes that I do not understand.
But don't worry about this, it is not urgent at all, so we can try to look at it and test later!

[ax3l]

Just a little more details: how exactly do you build this? Are you coupling this into the Geant4 build system?

[LDAmorim]

Yes, and that is probably the problem. I am making my own CMakeList.txt. I will add somethings there to check if that is what is causing the issue.

[ax3l]

Your own CMakeList.txt is good, it's very likely an issue on my side.

Just in case you haven't seen it yet, here are the few lines you need to add in CMake:
https://github.com/openPMD/openPMD-api#cmake

[LDAmorim]

Oh yes, I started writing it from there.
Now, actually, I think that my CMakeList.txt is probably not working well and the source of the issue. Right now (I simplified it to test) it looks like:

cmake_minimum_required(VERSION 2.6 FATAL_ERROR)
project(g4-positrons)

find_package(Geant4 REQUIRED)
find_package(HDF5 REQUIRED)
find_package(openPMD REQUIRED)

include(${Geant4_USE_FILE})
include_directories(${PROJECT_SOURCE_DIR}/include)
include_directories(/usr/local/include)

file(GLOB sources ${PROJECT_SOURCE_DIR}/src/*.cc)
file(GLOB headers ${PROJECT_SOURCE_DIR}/include/*.hh)

add_executable(g4-positrons src/g4-positrons.cc ${sources} ${headers})

target_link_libraries(g4-positrons ${Geant4_LIBRARIES} ${HDF5_C_LIBRARIES} openPMD::openPMD)

install(TARGETS g4-positrons DESTINATION bin)

I have openPMD-api installed via spack, but also have the source code. But I get the following error message when I load all the spack modules and run the cmake command:

dyld: Library not loaded: @rpath/libopenPMD.dylib
  Referenced from: /Users/LDianaAmorim/Documents/opt/g4-positrons/build/./g4-positrons
  Reason: image not found

This still seems like a mistake I am doing in my configuration.

[ax3l]

You could add the the keyword PRIVATE to

target_link_libraries(g4-positrons PRIVATE ${Geant4_LIBRARIES})
target_link_libraries(g4-positrons PRIVATE ${HDF5_C_LIBRARIES})
target_link_libraries(g4-positrons PRIVATE openPMD::openPMD)

which is more modern syntax, but I guess that's not the issue.

If you search in spack location -i openpmd-api is there a libopenPD.dylib file?

[ax3l]

Hm, I tried to reproduce your error message on a cloud instance with near-identical setup and cannot trigger the same errors :(

Probably have to check with you next week.

[ax3l]

Coming back to your original error message in the PR description, I did merge some fixes that might improve the situation (blindly) in #632
You can try them via

spack install openpmd-api@develop
spack load -r openpmd-api@develop

[LDAmorim]

Hi Dr Huebl!
Thank you so much for your help with this.
And sorry for taking so much of your time with this.

I did spack install openpmd-api@develop -python, as in my case we were finding lots of issues when compiling with python (in-built version of OD Catalina).

I am probably still doing something wrong with my CMakeList.txt file or where I call the OpenPMD functions... Because I still get the same error messages with that branch.

On a side note, I had a similar error trying to use the library boost, so it might be a problem specific to the way I installed things in my machine.

I will keep trying to reduce my code and trying to see where this is coming from. It is probably something very basic of CPP or CMAKE that I am not doing correctly.
Thank you!

PS:

Here is my current CMakeList.txt file:

SET(CMAKE_SKIP_BUILD_RPATH  FALSE)
SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
SET(CMAKE_INSTALL_RPATH "./install")
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)

cmake_minimum_required(VERSION 2.6 FATAL_ERROR)
project(g4-positrons)

find_package(Geant4 REQUIRED)
find_package(HDF5 REQUIRED)
find_package(openPMD REQUIRED)

include(${Geant4_USE_FILE})
include_directories(${PROJECT_SOURCE_DIR}/include)
include_directories(/usr/local/include)

file(GLOB sources ${PROJECT_SOURCE_DIR}/src/*.cc)
file(GLOB headers ${PROJECT_SOURCE_DIR}/include/*.hh)

add_executable(g4-positrons src/g4-positrons.cc ${sources} ${headers})

target_link_libraries(g4-positrons PRIVATE ${Geant4_LIBRARIES})
target_link_libraries(g4-positrons PRIVATE ${HDF5_C_LIBRARIES})
target_link_libraries(g4-positrons PRIVATE openPMD::openPMD)

install(TARGETS g4-positrons DESTINATION bin)

And my bash script to run the code:

G4DIR=~/Documents/opt/geant4/geant4-install/lib/Geant4-10.5.1
G4PINSTALL=~/Documents/opt/g4-positrons/install

cd ~/Documents/opt/geant4/geant4-install/bin/
source geant4.sh
cd -

. ~/Documents/opt/spack/share/spack/setup-env.sh
spack load cmake
spack load [email protected]%clang
spack load [email protected]%clang
spack load -r openpmd-api@develop

mkdir install
mkdir build
cd build
rm -rf *
cmake -DGeant4_DIR=$G4DIR -DCMAKE_INSTALL_PREFIX=$G4PINSTALL ..
make
make install
cd ..

Then in the CPP code I have a header file including:

#include <openPMD/openPMD.hpp>
using namespace openPMD;

That is included in the function file:

void save_outOPMD(MyUserSteppingAction::part p, G4String name){
    //# particles
    G4int np=p.get_size();
    size_t size = 1;//(argc == 2 ? atoi(argv[1]) : 3);

    //# final kinetic energy
    vector<G4double> energy=p.p_energy;

    Series series = Series("../data/"+name+"_data.h5", AccessType::CREATE);
    G4cout << "Created an empty " << series.iterationEncoding() << " Series\n";

    MeshRecordComponent En=
      series
        .iterations[1]
        .meshes["En"][MeshRecordComponent::SCALAR];
    G4cout << "Created a scalar mesh Record with required openPMD attributes\n";

    series.setAuthor(
        "LDAmorim <[email protected]>");
    series.setAttribute(
        "#parts", np);

    Datatype datatype = determineDatatype(shareRaw(energy));
    Extent extent = {size};
    Dataset dataset = Dataset(datatype, extent);

    G4cout << "Created a Dataset of size " << dataset.extent[0]
          << " and Datatype " << dataset.dtype << '\n';

    En.resetDataset(dataset);
    G4cout << "Set dataset properties for the scalar field En iteration 1\n";
/*
    En.setUnitDimension({
        {UnitDimension::M,  1},
        {UnitDimension::L, -2},
        {UnitDimension::T, -2}
    });
    En.setUnitSI(1.6021773e-13); // conversion to SI (Joule) from MeV*/

    series.flush();
    G4cout << "File structure and required attributes have been written\n";

    Offset offset = {0};
    En.storeChunk(shareRaw(energy), offset, extent);
    G4cout << "Stored the whole Dataset contents as a single chunk, "
            "ready to write content\n";

    series.flush();
    G4cout << "Dataset content has been fully written\n";
 return;
}

[ax3l]

When you load, maybe also add %clang so it reads spack load -r openpmd-api@develop %clang.

Can you take one step back and compile a minimal example without Geant4?

# CMakeLists.txt
cmake_minimum_required(VERSION 3.11)
project(openPMD_test)

add_executable(mytest main.cpp)

target_compile_features(mytest PRIVATE cxx_std_11)

find_package(openPMD REQUIRED CONFIG)
target_link_libraries(mytest PRIVATE openPMD::openPMD)

main.cpp with this content.

spack load cmake
spack load -r openpmd-api@develop %clang

mkdir -p simpleTest
cd simpleTests
# main.cpp and CMakeLists.txt go here

mkdir -p build
cd build
rm -rf ../build/*
cmake ..
make
./mytest

Does this show the same problem?

[ax3l]

If you don't want to pre-build openPMD-api you can just include it on the fly in CMake via:

find_package(openPMD REQUIRED CONFIG)

-> replace with ->

include(FetchContent)
set(CMAKE_POLICY_DEFAULT_CMP0077 NEW)
set(BUILD_TESTING OFF)
set(BUILD_EXAMPLES OFF)
set(openPMD_USE_PYTHON OFF)
FetchContent_Declare(openPMD
  GIT_REPOSITORY "https://github.com/openPMD/openPMD-api.git"
  GIT_TAG        "dev")
FetchContent_MakeAvailable(openPMD)

So the full CMakeLists.txt would read then:

# CMakeLists.txt
cmake_minimum_required(VERSION 3.11)
project(openPMD_test)

add_executable(mytest main.cpp)

target_compile_features(mytest PRIVATE cxx_std_11)

include(FetchContent)
set(CMAKE_POLICY_DEFAULT_CMP0077 NEW)
set(BUILD_TESTING OFF)
set(BUILD_EXAMPLES OFF)
set(openPMD_USE_PYTHON OFF)
FetchContent_Declare(openPMD
  GIT_REPOSITORY "https://github.com/openPMD/openPMD-api.git"
  GIT_TAG        "dev")
FetchContent_MakeAvailable(openPMD)

target_link_libraries(mytest PRIVATE openPMD::openPMD)

[LDAmorim]

Thanks! I will try to run with that CMakeLists.txt

[LDAmorim]

Thank you!
The last test you suggested (plus the line set(openPMD_USE_ADIOS2 OFF)) worked on my machine.
I am now testing a similar CMakeList.txt for my own code.

[LDAmorim]

The added lines to the CMakeLists.txt file are a successful workaround to build my code too. But when I run make, I find the output error message below:

Scanning dependencies of target openPMD
[  2%] Building CXX object _deps/openpmd-build/CMakeFiles/openPMD.dir/src/Dataset.cpp.o
[  5%] Building CXX object _deps/openpmd-build/CMakeFiles/openPMD.dir/src/Datatype.cpp.o
[  7%] Building CXX object _deps/openpmd-build/CMakeFiles/openPMD.dir/src/Iteration.cpp.o
/Users/LDianaAmorim/Documents/opt/g4-positrons/build/_deps/openpmd-src/src/Iteration.cpp:53:23: error: definition of implicitly declared copy assignment
      operator
Iteration& Iteration::operator=(Iteration const& i)
                      ^
1 error generated.
make[2]: *** [_deps/openpmd-build/CMakeFiles/openPMD.dir/src/Iteration.cpp.o] Error 1
make[1]: *** [_deps/openpmd-build/CMakeFiles/openPMD.dir/all] Error 2
make: *** [all] Error 2

I will try to see what is happening in the culprit file (I changed it's extension to copy it here):

Iteration.txt

Thanks!

[ax3l]

Interesting and quite unusual.

Can you change this Iteration.cpp to replace its #include lines with

#include "openPMD/Iteration.hpp"
#include "openPMD/Dataset.hpp"
#include "openPMD/Datatype.hpp"
#include "openPMD/Series.hpp"
#include "openPMD/auxiliary/StringManip.hpp"
#include "openPMD/backend/Writable.hpp"

please? Does that help? #640

[LDAmorim]

Thanks @ax3l ! The problem was caused by having a second old installation of openPMD-api in /usr/local. This was partly picked up before PR #640 was merged. Sorry for this.

[ax3l]

No need to be sorry, thanks again for testing and reporting. With #640 merged we make this even more robust now :)

[LDAmorim]

Thank you so much @ax3l !! 🎉 🌟 👏
The current openPMD-api@develop works well!

So now even the initial CMakeList.txt, where only the lines:

find_package(openPMD CONFIG REQUIRED)
target_link_libraries(my code PRIVATE openPMD::openPMD)

are included, can successfully compile the code with OpenPMD.