Pybind11: [Fail to expose `shared_ptr<...>` data member of a class]

[ecolss]

I created a custom class with a shared_ptr data member,

class Child
{
public:
    int x;
};

class Parent
{
public:
    Parent() : child(std::make_shared<Child>()) {}
    Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
public:
    std::shared_ptr<Child> child;
    std::shared_ptr<std::string> dummy;  // <- this is the issue
};

PYBIND11_MODULE(example, m)
{
    py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
        .def(py::init<>())
        .def_readwrite("child", &Parent::child)   // <- compiles OK
        .def_readonly("dummy", &Parent::dummy)   // compile error
        .def("get_child", &Parent::get_child);   // <- compiles OK
}

The issue is that I cannot expose dummy, which really confuses me, since I can expose child.
And I cannot understand what's the root cause, some snippets of the error messages are as follows,

anaconda3/lib/python3.8/site-packages/pybind11/include/pybind11/cast.h:1483:5: error: static_assert failed due to
      requirement 'std::is_base_of<pybind11::detail::type_caster_base<std::__1::basic_string<char> >,
      pybind11::detail::type_caster<std::__1::basic_string<char>, void> >::value' "Holder classes are only supported for custom types"
    static_assert(std::is_base_of<base, type_caster<type>>::value,

anaconda3/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:159:39: error: no member named 'cast' in
      'pybind11::detail::type_caster<std::__1::shared_ptr<std::__1::basic_string<char> >, void>'
            handle result = cast_out::cast(

anaconda3/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:172:96: error: no member named 'name' in
      'pybind11::detail::type_caster<std::__1::shared_ptr<std::__1::basic_string<char> >, void>'
        static constexpr auto signature = _("(") + cast_in::arg_names + _(") -> ") + cast_out::name;

[ecolss]

As @bstaletic suggested in chat room, this seems to be a bug of pybind11, and we shall not return share_ptr directly, instead use for example,

def_property_readonly("exposeThis", [](const ClassA& self) { return *self.exposeThis; })

[YannickJadoul]

@ecolss std::string does get converted to a Python str, so I'm not sure this is an error. The compilation error is quite horrible, but it's a compilation error, telling you you can't do that, rather than some crash at runtime.

We could make the custom std::string caster accept different holder types, I guess, but it would give a wrong impression of what's happening: you're not returning a shared_ptr, since it will be copied and converted to a str.

[ecolss]

@ecolss std::string does get converted to a Python str, so I'm not sure this is an error. The compilation error is quite horrible, but it's a compilation error, telling you you can't do that, rather than some crash at runtime.

We could make the custom std::string caster accept different holder types, I guess, but it would give a wrong impression of what's happening: you're not returning a shared_ptr, since it will be copied and converted to a str.

Hi Yannick, thanks so much for your comment.

So in my above comment, I'm returning a copy of that string object, not a reference?
So far, that seems to be the only way to make my code compile, do you have any other ideas?

[YannickJadoul]

So in my above comment, I'm returning a copy of that string object, not a reference?

Yes, indeed. It's described here: https://pybind11.readthedocs.io/en/stable/advanced/cast/strings.html

The reason is: std::string and Python's str are not compatible (already because they both manage their own memory). Also, Python strings are for example immutable, while C++ strings aren't.

So far, that seems to be the only way to make my code compile, do you have any other ideas?

What do you want, then? If you're using this std::string-str conversion, there's no point in returning a std::shared_ptr, right?

You might be able to make std::string a normal, custom type, by making it "opaque" (https://pybind11.readthedocs.io/en/stable/advanced/cast/stl.html). But then of course, you need to add methods to this class yourself, and you won't get a str on the Python side anymore.

[ecolss]

@YannickJadoul Thanks for the doc pointers!

Yes, I want to use std::string - str conversion for my case, and I don't want to make a custom type for it (not willing to add extra methods for it then).

The only concern I have now is the std::string -> str conversion would be time-consuming (at least a bit), since it'll copy the string here, not just passing a reference or pointer, so I wanted to achieve better performance on this.

[YannickJadoul]

Well, it's a copy in C++. It should be reasonably fast, especially since you're interfacing already with Python (and any string operation in Python already costs time).

Do test and measure, I'd say. There's not a lot of other things you can do when it comes to performance. If this isn't to your satisfaction, you can think about investing more time to fix it. But I don't think there's a silver bullet that'll let you have str and not perform this copy.