Pybind11: Exposing different globals via std::unique_ptr<S, py::nodelete> > can lead to pybind11_object_dealloc(): Tried to deallocate unregistered instance!

[heiner]

First of all, many thanks for this excellent library! <3

Issue description

I want to expose the values of global structs defined in a library I'm using. As I didn't get py::return_value_policy::reference to work with py::init or def("__init__", ...), I'm trying with a holder type of std::unique_ptr<S, py::nodelete> >. This appears to work at first, and in many situations, but breaks in a situation where different objects of this type override each other.

Reproducible example code

#include <pybind11/pybind11.h>

namespace py = pybind11;

struct S {
  const char name[80];
};

S sarray[2] = {
    {"hello"},
    {"world"},
};

PYBIND11_MODULE(skel, m) {
  py::class_<S, std::unique_ptr<S, py::nodelete> >(m, "S")
      .def(py::init([](int index) -> S* { return &sarray[index]; }))
      .def_readonly("name", &S::name);
}

Together with

from skel import skel


def test_skel():
    for _ in range(2):  # range(1) works.
        s = skel.S(0)
        name0 = s.name


if __name__ == "__main__":
    test_skel()

On Ubuntu 18.04 with GCC 7.4.0, this fails with:

$ python main.py
terminate called after throwing an instance of 'std::runtime_error'
  what():  pybind11_object_dealloc(): Tried to deallocate unregistered instance!
Aborted

Interestingly, it works on MacOS 10.15.5 with Apple clang version 11.0.0 (clang-1100.0.33.12).

[heiner]

Update: The following code appears to work, but I'm not certain I'm holding this right.

  py::class_<S>(m, "S")
      .def(
          "__init__",
          [](py::detail::value_and_holder& v_h, int index) {
            v_h.value_ptr() = &sarray[index];
            v_h.inst->owned = false;
            v_h.set_holder_constructed(true);
          },
          py::detail::is_new_style_constructor())
      .def_readonly("i", &S::i)
      .def_readonly("name", &S::name);

[bstaletic]

Interestingly, it works on MacOS 10.15.5 with Apple clang version 11.0.0 (clang-1100.0.33.12).

It also works for me, on linux with gcc 10 or clang 10.

[bstaletic]

Actually, scratch that. It just happens at exit. I managed to reproduce eventually.

EDIT: At least I've seen the error happen last night, but now I cannot repro...

EDIT2: Here's how to repro.

>>> import foo
>>> s1 = skel.S(0)
>>> del s1
>>> s1 = skel.S(0)
>>> s1 = skel.S(0)
>>> del s1
terminate called after throwing an instance of 'std::runtime_error'
  what():  pybind11_object_dealloc(): Tried to deallocate unregistered instance!
zsh: abort      python

[YannickJadoul]

Is this related to #2252, #1592, and #1568? Let me try if that proposed fix helps.

Update: Good news! I could reproduce with @bstaletic's code, but not anymore with the fix from #2252. Let's now see if I can understand what goes wrong.

[YannickJadoul]

See #2252 for further updates, @heiner & @bstaletic :-)

[bstaletic]

Fixed in 2.6.0. Thanks to @YannickJadoul and @henryiii