Flatbuffers: Faster Python Serialization

[tokotchd]

Hello,

I see that there is great support for reading numpy arrays from flatbuffer vectors. I'm ecstatic that this is supported, but I could not find the reverse functionality.

In fact, It seems that the only way to write data to flatbuffer vectors in python is with builder.prependByte(). As you image, this is unusably slow for large data.

It seems like there could be a fairly easy and fast method of implementation that involves something akin to: builder.CreateByteVector(.tobytes()). Unfortunately, without hacking apart the library, there's assertions in place to prevent using this function from Python within the scope of another table or struct (nested assert), which is where it would need to be as I understand it.

@kbrose created the original functionality, is there a chance he'd be willing to work on this?

[aardappel]

@rw author of the Python port

[kbrose]

Hi @tokotchd! My work was solely about the reading of flatbuffers (I had no need to write in my use case), so unfortunately I will be unable to provide support on this work, but I wish you luck!

I will say that there may be a way to add the bytes in bulk with something like

builder.Bytes[builder.head : (builder.head + my_array.nbytes)] = my_array.tobytes()

making sure everything is in the correct byte order (little endian) and the builder has been correctly initialized (with StartVector I think?).

It would be great if this works out and that could be incorporated directly into the python API!

[tokotchd]

I haven't had much luck directly accessing builder.Bytes, when doing so I end up only writing len(my_array.tobytes()) bytes to the builder.

When I write with prependByte() and CreateByteVector() I end up writing about 30% more than expected bytes (8388608 instead of 6220804). I'm not sure where or why this overhead is coming from, even looking directly at source.

Can anyone elaborate?

[tokotchd]

The above overhead problem persists, but I have a working workaround thanks to @kbrose

bytesOfImage = testImage.tobytes()
Image.ImageStartDataVector(builder, len(bytesOfImage))
builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage
data = builder.EndVector(len(bytesOfImage))

[kbrose]

Glad to hear it @tokotchd !

Do you think you could add an example of that to the documentation? @aardappel is that something you think would be useful?

[aardappel]

@kbrose sounds useful to me! Maybe as part of some special tips section in the python document?

[Amit072]

@tokotchd I think direct copy of data to Bytes buffer using below approach is not safe.

[1] builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage

Prependbyte() internally calls Prep() which may take decision to enlarge the Bytes buffer with padding etc, and will update head if it decide to regrow the Byte buffer.

I think it is problematic to directly use approach [1] if we have followed up write operation to builder using Prependbyte.
And also, what if your image size is large than allotted Bytes array, you may overwrite the allotted byte buffer in builder without taking care of alignment, head etc.

I'm also looking for this optimization, PrependByte is too slow for large buffers.

[Amit072]

Thanks @tokotchd, I am able to make it working with below change. We just need to seek the header to correct location before writing to Bytes array.

 bytesOfImage = testImage.tobytes()
 Image.ImageStartDataVector(builder, len(bytesOfImage))
 #We need to seek the header to correct place before writing into Bytes array.
 builder.head = builder.head - len(bytesOfImage)

 builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage
 data = builder.EndVector(len(bytesOfImage))

You can check my answer @stackoverflow
https://stackoverflow.com/questions/49239293/fast-python-serialize-to-flatbuffer?answertab=active#tab-top

[kbrose]

@Amit072 Do you think you could add an example of that to the documentation?

[gyenesvi]

This addition to the builder seems useful, however, one thing is not clear. If I have a data member in a table defined with forced alignment as:

table Buffer {
  data:[ubyte] (force_align: 16);
}

and I want to create the data array from numpy using data = builder.CreateNumpyVector(numpy_array), and then add the data with the generated method BufferAddData(builder, data) does it properly align the data? Isn't it a problem that we are bypassing the generated method BufferStartDataVector because of the use of builder.CreateNumpyVector? I guess it depends on where the forced alignment is handled actually.

[felixfrank]

@gyenesvi, correct me if I'm wrong, but the way I read the flatbuffer docs and interpret the underlying idea of force alignment, is that it is an attribute of structs and cannot be used for arrays. As far as I know, if you add the attribute to a table field (like in your example) it is just ignored.

Also I'm curious, why would you actually want to add additional padding between individual array members of the same type?

[gyenesvi]

It's been a while since I worked on this, but as far as I remember, the point of the question was that if the schema contains force_align (which I have seen that way I put in my simplified example), and I write out the data using the above workaround (which does not seem to handle alignment), will that be a valid document, i.e. will it match the schema?

I was assuming that in the above case, the force_align is for the data member of the table, and not for the array itself (not between array members, just the beginning of the array, which seemed reasonable to me). From the documentation it seems you are right @felixfrank, that force_align is defined for structs (so the point is that it is ignored for _tables_, and not that it cannot be applied for arrays), so it is ignored in this case anyway, so the above workaround produces proper buffers, right?

[felixfrank]

I see. Let me elaborate on my previous statement that force_align cannot be used for arrays. This basically follows from:

1. force_align is ignored for tables
2. flatbuffer structs cannot hold arrays (because structs are compile-time fixed size objects, whereas arrays in flatbuffer schemas are variable length).

That being said, you are absolutely right in that the builder.CreateNumpyVector only takes care of the alignment according to the datatype of the numpy.array members and the additional uint32 field for the array size (which is what the Start...Vector function in the generated code does as well). In that regard it does produce proper buffers.

If you require a different alignment it get's tricky. I guess you can use the builder.StartVector function and pass the required alignment there directly.

[gyenesvi]

I believe that the alignment that builder.CreateNumpyVector takes care of is fine for me. I was only confused whether that produces valid documents, but it seems now that yes, since force_align does not matter here.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity for 1 year. It will be automatically closed if no further activity occurs. To keep it open, simply post a new comment. Maintainers will re-open on new activity. Thank you for your contributions.