Cython: [Solved] What is the proper way to convert C <char *> byte stream into Python's <bytes> object with 0x00 bytes in it

Created on 17 Oct 2019  ·  4Comments  ·  Source: cython/cython

Hi,

I'm trying to pass some bytes from a C function into a Python function, but it's always terminated by the 0x00 byte in it.

I know that the 0x00 byte is supposed to be the terminator of strings in C. But in pure C, it's not difficult to ignore the 0x00 byte by some manual manipulation. For example:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main()
{
    char a[9];
    char *b = malloc(9);

    a[0] = 0x01;
    a[1] = 0x02;
    a[2] = 0x03;
    a[3] = 0x04;
    a[4] = 0x00;  // if we truely treat a as a string, then this will be the terminator in most case
    a[5] = 0x06;
    a[6] = 0x07;
    a[7] = 0x08;
    a[8] = 0x00;

    memcpy(b, a, 9); 

    for (int i=0; i<9; i++)
    {
        // prints all 9 bytes properly
        printf("%x\n", b[i]);
    }

    return 0;
}

But in Cython, the 0x00 byte seems to be inevitable since I'm trying to auto-convert some bytes from C \ 

cdef class Test():

    cdef unsigned char buf[8]

    def __cinit__(self):
        self.buf[0] = 0x01
        self.buf[1] = 0x02
        self.buf[2] = 0x03
        self.buf[3] = 0x04
        self.buf[4] = 0x00
        self.buf[5] = 0x06
        self.buf[6] = 0x07
        self.buf[7] = 0x08

    def main(self):
        # It's always terminated by the 0x00 byte in buf[4]
        # and only prints b'\x01\x02\x03\x04'
        print(self.buf)

But normally, 0x00 is always a normal byte in Python's \

Well, I know that there is a way to construct a Python object in the C code, so manually constructing a Python's \

So, question: What is the proper way to convert a C \

Additional info:

  • Platform: GNU/Linux
  • Python version: 3.7.0
  • Cython version: 0.29.7
  • language_level: 3
support question
Source
picture Mr-indescribable

All 4 comments

If you know the length, you can return buf[:size] to get the correct
conversion.

On Wed, Oct 16, 2019, 8:29 PM 不可描述先生 notifications@github.com wrote:

Hi,

I'm trying to pass some bytes from a C function into a Python function,
but it's always terminated by the 0x00 byte in it.

I know that the 0x00 byte is supposed to be the terminator of strings in
C. But in pure C, it's not difficult to ignore the 0x00 byte by some manual
manipulation. For exmaple:

include

include

include

int main()
{
char a[9];
char *b = malloc(9);

a[0] = 0x01;
a[1] = 0x02;
a[2] = 0x03;
a[3] = 0x04;
a[4] = 0x00;  // if we truely treat a as a string, then this will be the terminator in most case
a[5] = 0x06;
a[6] = 0x07;
a[7] = 0x08;
a[8] = 0x00;

memcpy(b, a, 9);

for (int i=0; i<9; i++)
{
    // prints all 9 bytes properly
    printf("%x\n", b[i]);
}

return 0;

}

But in Cython, the 0x00 byte seems to be inevitable since I'm trying to
auto-convert some bytes from C type into Python type:

cdef class Test():

cdef unsigned char buf[8]

def __cinit__(self):
    self.buf[0] = 0x01
    self.buf[1] = 0x02
    self.buf[2] = 0x03
    self.buf[3] = 0x04
    self.buf[4] = 0x00
    self.buf[5] = 0x06
    self.buf[6] = 0x07
    self.buf[7] = 0x08

def main(self):
    # It's always terminated by the 0x00 byte in buf[4]
    # and only prints b'\x01\x02\x03\x04'
    print(self.buf)

But normally, 0x00 is always a normal byte in Python's object.

Well, I know that there is a way to construct a Python object in the C
code, so manually constructing a Python's object should be doable but seems
it's not the proper way to do that.

So, question: What is the proper way to convert a C byte stream
into a Python object with 0x00 bytes in it. (With Cython's automatic type

conversion if possible)

Additional info:

  • Platform: GNU/Linux
  • Python version: 3.7.0
  • Cython version: 0.29.7
  • language_level: 3

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cython/cython/issues/3189?email_source=notifications&email_token=AADWVAOWJJO4WQKGIOY2VR3QO7LZDA5CNFSM4JBTVVJKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HSKWHXQ,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADWVAM43QR77JO7KSDSA4DQO7LZDANCNFSM4JBTVVJA
.

robertwb picture robertwb on 17 Oct 2019
👍1

@robertwb

WOW, thanks, that's what I'm seeking for.

But why?

Now the Cython code is updated to:

cdef class Test():

    cdef unsigned char buf[8]

    def __cinit__(self):
        self.buf[0] = 0x01
        self.buf[1] = 0x02
        self.buf[2] = 0x03
        self.buf[3] = 0x04
        self.buf[4] = 0x00
        self.buf[5] = 0x06
        self.buf[6] = 0x07
        self.buf[7] = 0x08

    def get_buf(self):
        return self.buf[:8]

    def main(self):
        # This converts the bytes properly
        print(self.get_buf())

In my understanding, when I renturn buf[:8] in the get_buf() method, it will do the same thing to convert a \

But the truth is I was wrong, there must be something different between these two processes. Could you please to tell me a bit more about this? Thanks a bunch :heart:

Mr-indescribable picture Mr-indescribable on 17 Oct 2019

Cython treats char[] and char[n] the same as char, and char has no
inherent length but is so commonly used for null-terminated data that it is
converted that way. (For that matter, char arrays are also often used for
null-terminated data with an (to often unchecked) upper bound, rather than
actual bound, of their length. Welcome to the world of stack overflows...)

Writing buf[:size], or even buf[m:n] is an explicit way of telling Cython
exactly what characters belong in the resulting object.

On Wed, Oct 16, 2019 at 9:24 PM 不可描述先生 notifications@github.com wrote:

@robertwb https://github.com/robertwb

WOW, thanks, that's what I'm seeking for.

But why?

Now the Cython code is updated to:

cdef class Test():

cdef unsigned char buf[8]



def __cinit__(self):

    self.buf[0] = 0x01

    self.buf[1] = 0x02

    self.buf[2] = 0x03

    self.buf[3] = 0x04

    self.buf[4] = 0x00

    self.buf[5] = 0x06

    self.buf[6] = 0x07

    self.buf[7] = 0x08



def get_buf(self):

    return self.buf[:8]



def main(self):

    # This converts the bytes properly

    print(self.get_buf())

In my understanding, when I renturn buf[:8] in the get_buf() method, it
will do the same thing to convert a byte stream into a
Python object. So the same process was meant be invoked just like what it
did when it converts the buf into a object in the original code.

But the truth is I was wrong, there must be something different between
these two processes. Could you please to tell me a bit more about this?
Thanks a bunch ❤️

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cython/cython/issues/3189?email_source=notifications&email_token=AADWVAJMAYAZOHHTP6USRVDQO7SGDA5CNFSM4JBTVVJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBOW4ZQ#issuecomment-542994022,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADWVAIS3KQWO5TNY4V44ZDQO7SGDANCNFSM4JBTVVJA
.

robertwb picture robertwb on 17 Oct 2019
👍1

@robertwb

Uh, I get it, the thing actually matters is whether Cython knows the actual length.

Thanks for the answer.

Mr-indescribable picture Mr-indescribable on 17 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Barry1 picture Barry1  ·  4Comments
branko623 picture branko623  ·  5Comments
WillAyd picture WillAyd  ·  4Comments
scoder picture scoder  ·  5Comments
hroncok picture hroncok  ·  4Comments