Flatbuffers: Byte data from flatbuffer vector into a Python NumPy array

[SkyToGround]

I have a flatbuffer structure which holds some binary data. A simple example illustrating this shown below:

table NDArray {
   name:string;
  dataType:DType;
  pData:[ubyte];
}

The data in pData might actually be an array of int32 or some other type and I want to feed this data into the numpy-function fromstring() to get that array without having to read every byte individually. E.g.:

my_array = numpy.fromstring(fb_raw_data, dtype=numpy.int32)

Is this possible?

[ghost]

That should generally be possible, but I doubt the Python implementation has an accessor for that. Since Python has no raw pointers, I am guessing this would have to take the form of "copy vector data to string", which could be added. @rw?

[SkyToGround]

In the current implementation, I have to copy the data byte by byte:

nr_of_bytes = fb_arr.PDataLength()
np_buffer = np.empty([nr_of_bytes, ], dtype = np.uint8)
for i in range(nr_of_bytes):
    np_buffer[i] = fb_arr.PData(i)
data_arr = np.fromstring(np_buffer.tobytes(), dtype = np.int32)

Thus any improvement to this would be welcome.

[ghost]

Anyone want to contribute a function for this?

[SkyToGround]

I might look into solving this problem myself eventually, if no one has implemented this feature in a few months time.

[rw]

1) I think you'd be better off using numpy.frombuffer.
2) There's no way to do what you want with the current Python interface. Keep in mind that we are endian-safe, so if flatbuffers gives you access to the underlying memory, then we'll violate that promise. For that reason, we can't always give you 'memcpy'-like performance.
3) That said, there should be adequate ways to integrate ctypes or numpy types so that we can have accessors that provide direct, correct views into the data. Would you like to try adding that feature? I think what it would take is a) generating numpy-style types for any fixed-length (int, struct) flatbuffers type, and b) adding that to the code generator and adding some tests.

[ghost]

On 2, I think it is ok to give people buffer access through an obnoxious name like getbuffer_little_endian or whatever.. then it is obvious that it is the clients responsibility to ensure they can deal with little endian data.

Alternatively, only generate buffer access for [byte] / [ubyte].

[maxweisel]

Piggybacking on this issue, would it by possible to get byte vectors as a whole object in python? The C# API does this beautifully. I understand C# makes it slightly easier. However, at the moment it looks like I'll need to create an empty bytearray and append each byte to it by looping through the vector.

[ghost]

Yup, such an accessor would be great. My Python is lacking, though.. what is the quickest way to memcpy something into another array? Anyone want to make a PR?

[maxweisel]

This is what I'm doing at the moment:

dataBytes = bytearray()
dataLength = flatbufferMessage.DataLength()
for i in range(dataLength):
dataBytes.append(flatbufferMessage.Data(i))

I'm fairly new to python so this might not be ideal, but this seems to be the best approach I can take in python 2.

[joireman]

Using the following schema storing int16 data like so:

union MyDataUnion {
  Int8Data,
  Int16Data,
  ...
}

table MyData {
  id: ubyte;
  readings: MyDataUnion;
}

where the typed data is defined similar to the following.

table Int16Data {
  data:[short];
}

If I have a flatbuffer reference to the MyData table my_data_fb as an object: Looking into the generated flatbuffers code for the Int16Channel class Data method I'm able to define a method
that gets the offset to the data which I can then use along with the data length and Bytes method to send to np.frombuffer

import flatbuffers
import numpy as np

def get_data_byte_offset(my_data):
    o = flatbuffers.number_types.UOffsetTFlags.py_type(my_data._tab.Offset(4))
    offset = my_data._tab.Vector(o) if o != 0 else 0
    return offset

my_int_data = Int16Data.Int16Data()
my_int_data.Init(my_data_fb.Readings().Bytes, my_data_fb.Readings().Pos)
num_pts = my_data.DataLength()

# Get offset to start of data 
offset = get_data_byte_offset(my_int_data)

# Unpack data to a numpy array
data_np = np.frombuffer(my_data_fb.Readings().Bytes,
                        offset=offset,
                        dtype=np.dtype('i2'),
                        count=num_pts)

This works and is very fast, relative to calling:

data = np.array([my_data.Data(i) for i in range(num_pts)])

but is this universally applicable?

[ghost]

This has been implemented.