Pybind11: Let C++ function return bytes without extra copy

[ppwwyyxx]

Make sure you've completed the following steps before submitting your issue -- thank you!

Issue description

According to the documentation, to let a C++ function returns bytes instead of unicode string in Python, one needs to use py::bytes: http://pybind11.readthedocs.io/en/stable/advanced/cast/strings.html?highlight=string#return-c-strings-without-conversion.

However I found that py::bytes calls PyBytes_FromString function, which copies the string according to https://docs.python.org/3/c-api/bytes.html#c.PyBytes_FromString.

Is there anyway to construct a bytes object directly in C++ without an extra copy?

[Xfel]

If you look at the memory model of bytes, you can see that the data array must be located right after the object header in memory. So no, there is no way to return a bytes object without a copy.

[podgorskiy]

As @Xfel said, there is no way to return bytes without copying, unless you have created a valid bytes object with object header in the first place.

So, if you really need to return bytes without copying, you might consider this hacky way:

.def("some_function", []()->py::object
{
    PyBytesObject* bytesObject = nullptr;

    // this lambda will create bytes object of the given size and 
    // will return pointer to the allocated memory, like malloc would do
    auto alloc = [&bytesObject](size_t size)
    {
        bytesObject = (PyBytesObject*) PyObject_Malloc(
                offsetof(PyBytesObject, ob_sval) + size + 1);

        PyObject_INIT_VAR(bytesObject, &PyBytes_Type, size);
        bytesObject->ob_shash = -1;
        bytesObject->ob_sval[size] = '\0';
        return bytesObject->ob_sval;
    };

    // In your function, you will need to use `alloc` function to allocate 
    // memory for your string that you want to return.
    your_function(args, alloc);

    return py::reinterpret_steal<py::object>((PyObject*)bytesObject);
}

[bstaletic]

This issue seems to have been answered.