Pybind11: Mixing type conversions and opaque types with pybind11

[daenuprobst]

(This is a copy of a question I posted no stackoverflow, but I suspect it might be a bug / unwanted behaviour or should at least be documented)

I am making use of opaque types with pybind11. For example, I defined one for the stl container std::vector<uint32_t>, which is a argument type of the method FromVec:

void FromVec(std::vector<uint32_t> vec);

PYBIND11_MAKE_OPAQUE(std::vector<uint32_t>);

PYBIND11_MODULE(tmap, m)
{
    py::bind_vector<std::vector<uint32_t>>(m, "VectorUint", 
      "Unsigned 32-bit int vector.");

    m.def("from_vec", &FromVec, py::arg("vec")
}

With this I can now (in Python) do the following:

vec = VectorUint([2, 66, 262, 662, 26, 62])
from_vec(vec)

However, type conversion does not work anymore, as the function now expects a VectorUint and doesn't accept list anymore, e.g.:

l = [2, 66, 262, 662, 26, 62]
from_vec(l)

Is there a way to allow for both, oblique types and type conversion? That is, when a list is passed instead of a oblique bound VectorUint it is converted to a std::vector<uint32_t> rather than Python throwing "incompatible function arguments"?

[molpopgen]

This behavior is documented here, where it states that making a container opaque disables "the template-based conversion machinery of types, thus rendering them opaque."

To get the behavior you are looking for, you need to overload FromVec to accept a py::list argument (or a py::array_t<uint32_t>), and that function then needs to create a new std::vector from the input data itself.

[daenuprobst]

That much for my reading comprehension skills. Thanks!

[elliott-imhoff]

Would it be possible to see an example of converting from a py::list to an std::vector while maintaining the references to the objects in the list? I've tried this:

void FromVec(py::list& py_vec)
{
    std::vector<myObj> cpp_vec;
    for (auto& handle : py_vec)
    {
        cpp_vec.push_back(handle.cast<myObj>());
    }

    FromVec(cpp_vec);
}

However, it seems the cast() function ends up copying the object, which removes the whole purpose of using PYBIND11_MAKE_OPAQUE() in the first place.

[elliott-imhoff]

Taking the original example code given, but tweaking it just slightly to use objects (with references I want to maintain) instead of uint32_t's (where myObj is a c++ object).

void FromVec(std::vector<myObj>& vec);

PYBIND11_MAKE_OPAQUE(std::vector<myObj>);

PYBIND11_MODULE(tmap, m)
{
    py::bind_vector<std::vector<myObj>>(m, "VectorMyObj", 
      "Vector of myObj instances");

    m.def("from_vec", &FromVec, py::arg("vec")
}

Suppose (in python) I have two old VectorMyObj lists (let's call them myobj_old_list1 and myobj_old_list2) that I need to concatenate together and pass into from_vec at once, which then modifies the objects. I need the changes to the objects that happen in from_vec to appear in myobj_old_list1 and myobj_old_list2. I thought I found a solution by declaring an empty VectorMyObj (let's call it myobj_new_list), then adding the instances of the myObj objects one by one to the new vector, hoping this would copy the references and myobj_old_list1/myold_old_list2 and myobj_new_list would point to the same underlying objects.

myobj_new_list = tmap.VectorMyObj()
for obj in myobj_old_list1:
    myobj_new_list.append(obj)

for obj in myobj_old_list2:
    myobj_new_list.append(obj)

tmap.from_vec(myobj_new_list)

However, when I check the objects in myobj_old_list1 and myobj_old_list2 after from_vec has been called, those objects haven't been modified. The objects in myobj_new_list, on the other hand, have been modified. From this, I assume that appending the objects to a VectorMyObj makes a copy of the objects themselves, instead of the references. So now I'm stuck again. Please correct me if I'm wrong.

Any suggestions to get the behavior I'm looking for would be greatly appreciated. Thanks!

[YannickJadoul]

@elliott-imhoff You're creating a vector of myObj, so C++ will copy these. This hasn't really got anything to do with pybind11, has it? If you want to reference the same object, you'll need pointers or references. Or maybe shared_ptr would be useful, in such a case, if you want to share ownership of the myObj objects.

[elliott-imhoff]

Ok thank you for the reply. I wasn't aware that std::vector's copied objects when appending them to the list. Guess I'll have to use shared_ptr's :(

[YannickJadoul]

@elliott-imhoff It's also possible to store pointers or std::refs to objects in a vector. But then you need to be sure about who owns the objects, and sure that the objects you refer or point to will be kept alive as long as you use them.

[beatmax]

Calling py::implicitly_convertible<py::list, std::vector<uint32_t>>() does the trick
(https://stackoverflow.com/a/60744217/7604145).