Pybind11: [QUESTION] How to directly modify bytes passed by Python?

[Xeite]

I have a function in CPP like this

void encode(uint8_t* buffer) {
      // Directly modify specified buffer.
      buffer[0] = 1;
      buffer[1] = 2;
}

And in Python, I used the function like this

buffer = bytes(10)
encode(buffer)
# Expect buffer has been modified.

1. First I tried to use likes this

void encode(uint8_t* buffer) {
    buffer[0] = 0xff;
    buffer[1] = 0xff;
}

m.def("encode", &encode);

But I failed to run Python because uint8_t* was interpreted to int

TypeError: encode(): incompatible function arguments. The following argument types are supported:
    1. (arg0: int) -> None

1. I read CPP can accept python bytes in the form of py::bytes, so I changed the signature naively.

m.def("encode", [](py::bytes buffer) 
     encode(buffer);
});

But it did not work of course because py::bytes could be converted uint8_t

1. I guess if cpp accepts python bytes in the form of py::bytes which can be converted to char* from doc, it might be possible to define char* for argument. So tried

void encode(uint8_t* buffer) {
    buffer[0] = 0xff;
    buffer[1] = 0xff;
}

m.def("encode", [](char* c) 
     encode((uint8_t*)c);
});

I succeed to pass Python bytes to binding function now, but the buffer modified in the binding function could not shown in the python.

buffer = bytes(2)
print(buffer)

# output: b'\x00\x00

encode(buffer)

print(buffer)
# output: b'\x00\x00
# Note that buffer still does not changed.

How to see the modified buffer from Python in this case? I read about return_value_policy and add the def with small hope, but it did not work as well (I think it's syntax for return value)

Thanks.

[bstaletic]

Strings are immutable in python. Any mutation is a copy and no in-place changes are allowed. You'll have to do something like buffer = encode(buffer).

[YannickJadoul]

No, bytes objects are not mutable:

>>> x = b'abc'
>>> x[1] = 'B'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'bytes' object does not support item assignment

https://docs.python.org/3/library/stdtypes.html#bytes-objects:

Bytes objects are immutable sequences of single bytes. Since many major binary protocols are based on the ASCII text encoding, bytes objects offer several methods that are only valid when working with ASCII compatible data and are closely related to string objects in a variety of other ways.

So you couldn't write that in Python either, so I don't think pybind11 is to blame here.

[Xeite]

Thanks, I missed basic concept of the data structure. Thank you for answering me.
ps. I don't blame pybind11. Rather it's awesome project, and thank for the project. :)

I closed the question.

[virtuald]

bytebuffer is mutable IIRC, use py::buffer as your pybind11 argument and you can use the buffer protocol to mutate it.