Flatbuffers: How to use flexbuffers for python

[guispor7]

hi
I want to know how to use flexbuffers for python. I only find usage of flexbuffers for c++.
I can't find any files like flexbuffers.py.
I have a fbs file like this :

enum CustomOptionsFormat : byte {
  FLEXBUFFERS = 0,
}
table Operator {

  opcode_index:uint;


  inputs:[int];
  outputs:[int];

  builtin_options:BuiltinOptions;
  custom_options:[ubyte];
  custom_options_format:CustomOptionsFormat;

  mutating_variable_inputs:[bool];
}

How to use python to write data to custom_options?
Thank you very much

[aardappel]

FlexBuffers currently doesn't have a Python implementation, sorry.

[yeralin]

Then we need to throw an error when a user is trying to compile a schema to --python with (flexbuffer) tags.

[aardappel]

A Python FlexBuffers implementation is actually close to being finished :)
@dmitriykovalev

[dmitriykovalev]

https://github.com/google/flatbuffers/pull/5880 is merged!

[gyenesvi]

When can we expect this to make its way to an official release that we can pip install?

[aardappel]

@gyenesvi whenever 1.13 happens, which may be many months away.

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

[william-silversmith]

Is 1.13.0 happening anytime soon? I think flexbuffers would be helpful for my use case where I have a map of integers to variable length binary data.

[william-silversmith]

Ah, I saw the info #6076 I'll try including a copy of flexbuffers in the application until 1.13.0 comes out and see how that goes.

[william-silversmith]

I gave it a try but unfortunately it was too slow for my use case (~4x slower than pickle.loads accessing only a small fraction of items whereas the former method accessed all of them). The culprit turned out to be the binary search, whose running time was dominated by calling .AsKeyBytes. I tried creating a python script with a numpy array and running binary search using mostly the same code and it became orders of magnitude faster. Some specializations would probably make this approach much more practical.

To explain, in my use case, using ~400 cores for many hours a day, this computation is taking two weeks, which is ~60% unpickling. There might be smarter ways to structure the data, but it seems within reach to essentially double performance by using faster maps. I'll probably end up rolling my own data structure, but figured it could be helpful to leave some info.

[aardappel]

1.13 (which may become 2.0) is still not done, there's some bigger features we want to merge first. Hard to say when.

@william-silversmith maybe @dmitriykovalev can shed light on why its slow.

Then again, if your workload is so big that you are throwing 400 cores at it, I don't follow why it is happening in Python.. we also have FlexBuffers implementations for C++ and Rust that could easily be 100x faster.

[william-silversmith]

It's pretty common in scientific computing to use Python/Numpy and then write a C++ wrapper if need be. I'm in the process of optimizing a pipeline, so it's still doing things that worked at a smaller scale but not so much anymore. I may end up writing a Cython wrapper.

Not sure if you guys already did this in flatbuffers C++ edition, but while researching this topic, I found a way to make binary search faster: https://algorithmica.org/en/eytzinger

[william-silversmith]

I ended up writing my own, but maybe some of the ideas will be helpful to share. https://github.com/seung-lab/mapbuffer