Pybind11: Using python C API with pybind11

[neutralid]

We have some old code that worked directly with Python.h and the python C API. Lots of methods pass around PyObjects, which are probably pretty close to py::object in terms of implementation. If I attempt to bind a function which returns a PyObject, I get type errors since it doesn't know what PyObject is.
I can create a dummy class binding to stop these errors, but then python just treats the returned value as a generic object.

It appears that py::object makes some use of PyObject under the hood, but it's unclear if there's an easy way to cast/convert to the former. Can anyone provide guidance?

[neutralid]

I apparently was on the verge of a breakthrough just needed to keep chugging along. For future people like me, py::object does indeed wrap a PyObject*, so all we had to do in our bindings was define a lambda to properly get a pointer/convert a PyObject to a py::object.

.def("setValue", { c.setValue(o.ptr()); })
.def("getValue", { return py::reinterpret_steal(c.getValue());

[jagerman]

Just a note on this to elaborate a bit:

One of the big deficiencies in the Python C API is that you can't tell whether a returned PyObject * is a borrowed or stolen reference (except by reading the documentation). Pybind tries to correct this by having py::object, which manages an inc_ref() on creation and a dec_ref() on destruction.

When you need to work with borrowed or stolen references you can use auto o = py::reinterpret_steal<py::object>(pyobject_ptr); or auto o = py::reinterpret_borrow<py::object>(pyobject_ptr); to steal or borrow a reference, respectively. Borrowing is often the right thing, but some C API functions (particularly ones that create new objects) return new references that need to be stolen to get the reference counts right.

[artiit]

Hi, @jagerman , My code is something like bellow:

PyObject* GetRet(Ret* ret) {
  npy_intp shape[1];
  shape[0] = 1024;
  PyArray_Descr* descr = PyArray_DescrFromType(NPY_OBJECT);
  PyObject* obj = PyArray_Empty(1, shape, descr, 0);
  PyArrayObject* np_array = reinterpret_cast<PyArrayObject*>(obj);
  PyObject** out = reinterpret_cast<PyObject**>(PyArray_DATA(np_array));
  for (int i = 0; i < ret->size(); ++i) {
      out[i] = PyBytes_FromStringAndSize(ret[i]->data(), ret[i]->size());
    }
  return obj;
}

pybind11 wrapper like:

m.def("get_ret", [](Ret* ret){
//...
return py::interpret_steal<py::object>(GetRet(ret));
});

In this case, should I use py::reinterpret_steal instead of py::reinterpret_borrow and what py::return_value_policy should I take?
I found that the return_value_policy does not matter cause both reference and take_ownership just worked well.

Another question is external call function is much more time consuming than internal function execution. For example, I have two py file.
test.py

import my_pybind11_wrapper_moudule as mm
def foo():
  t1 = time.time()
  res = mm.get_ret(Ret) # wrapper of GetRet
  t2 = time.time()
  time1 = t2-t1
  return res

main.py

t1 = time.time()
res = foo()
t2 = time.time()
time2 = t2-t1

When I use steal, time2 is 30ms when time1 is 25ms, when I use borrow, both are 30ms.

Any suggestions will be appreciate.

Thanks.