Flatbuffers: Existential questions and insights

[maingoh]

In my company we were using flatbuffers for few months and after we realized we would need grpc sooner than we thought, we had to switch to protocol buffer.
With these months of use, we faced some difficulties with flatbuffers and I wanted to share my questions and insights with you.

From a high level aspect & ease of use:

• I feel this is a really interesting project with an innovative design and a lot of potential. I often check the progress but it doesn't seem to be as pushed by google than protocol buffers. Is there a reason why there is only one google maintainer ? (it seems)
• Would it be totally impossible to "merge" the efficiency of flatbuffers and the ease of protocol buffers ?
• The Python binding is working but doesn't seem to be often updated and with as much ease as C++. The C++ compiler improve really quickly but others languages don't (GRPC for example).
  Which can be a problem if your stack use multiple languages.
• Will the object api only be for c++ or can it be extended to other langages ?
• In protocol buffer there is a DebugString function which is quite useful to debug, it would be nice to have one in flatbuffers, without having to parse the .fbs and serialize to json.
• It's probably us but we didn't find it explicit that adding objects to vector must be in reverse order. It should probably be more highlighted in the doc (at that time, the doc wasn't as complete though). Also could it be possible to add them in normal order ? New developers can find this
  disconcerting.

About versionning:

• It is mentioned that we can add a field without any problem. What about a required field ?
• Also is it possible to make a required field deprecated ?
• There are times when we wish to remove deprecated field from the .fbs, for readability. I wonder if we could find a better way of deleting fields without using ids.

About performance:

• It seems that Java binding and Go binding are not as fast as protobuf ones. Is it because they are not optimized enough for now or is there no room for improvement ?
  https://discuss.dgraph.io/t/comparison-of-protocol-buffers-v-s-flatbuffers/1085/5
  https://blog.dsl-platform.com/flatbuffers-performance-in-java/
• In average (and approximately), I was wondering how much memory does a flatbuffer payload take more than a protocol buffer one?

Thank you very much,

[aardappel]

Thanks for your extensive feedback!

You had to switch to Protobuf because we don't have GRPC bindings for all the languages that you use?

Comments on your bullets in the same order:

• You have to understand that until recently, all of Google's massive code-base was using Protobuf uniformly. Data exchange formats have kind of a viral effect. So Protobuf is and always will be hugely important to Google, regardless of how awesome FlatBuffers is :) We have quite a few internal users, but adoption is slow going because of that reason. We have more internal contributors than just me, but commits may appear authored by me because of the way we move things from our non-git source control. Action is mostly in C++ since that is what we use, other languages are mostly external contributions.
• We made a conscious decision to sacrifice simplicity of the API where needed for the sake of speed. We have an API that is as easy to use as Protobuf, it is the object API, and just like Protobuf is inefficient. We'd recommend to only use it in non-speed sensitive cases.
• The Python binding was made by an external contributor, and since we don't use it internally, it doesn't get as much love. A GRPC binding for Python was started but never completed.
• The object API can totally work for other languages, in fact, it is more appropriate for other languages. In slow languages like Python, an object API may even be faster, since the ratio between the cost of object allocation vs the cost of computation is different.
• To have a DebugString style function, we would need access to the schema, so it would pretty much be a wrapper around the current JSON generator. Protobuf messages can be parsed (without field names) even when you don't know the schema (since it contains types). FlatBuffers doesn't contain types so this is generally not possible.
• Most people would use CreateVector which adds all elements in one go. If doing it by element is not document clearly, this is an oversight, please file an issue.

Versioning:

• A new required field can fail the C++ verifier on old data. In all other use cases it fine. We should probably document that too, yes.
• Yes, you can deprecate a required field, that will just stop requiring it.
• Yeah, we could add a dedicated syntax for deprecated fields. For now, you could rename the field into something more obviously deprecated.

Performance:

• FlatBuffers was definitely designed for languages like C++, so Java and Go may not be as good a fit. A big issue in Java for example is that it can't read UTF-8 strings in-place. FlatBuffers replaces object allocation with a small amount of work during field access, which may not always be the right trade-off. Of course, It is hard to tell without looking at these benchmarks if they're not doing something silly, you should always do tests for your own use-case.
• In our testing, FlatBuffer binaries are typically 30% bigger, but YMMV.

[ghost]

Made some documentation fixes based on the above here: https://github.com/google/flatbuffers/commit/4b27c92910ebe57c1d21932df2a85c142516c7b4
Other issues have been noted (Object API for more languages, Java performance, removal of deprecated fields) but will take longer to materialize :)

[maingoh]

Thank you for your detailed answer ! Yes, we needed GRPC support for Python and C++ (which I saw has been reworked few weeks ago).

Here are my answers/new questions:

High level :

• Thank you for the clarification, hope it will get adopted more and more internally :)
• I was wondering if there was a way to "cast" the buffer into a class/struct, but apparently the constructors of the STL types always copy the data (probably the same problem for other languages). Is it why it is inefficient ? If those constructors existed we would probably have something easy to use and efficient (like a wrapper around the buffer).
• I guess I was expecting other languages to improve as fast as C++, but I understand it is a lot of work :)
• Good news !
• The DebugString method in protobuf doesn't required anything like loading the .proto (and the field names are printed). For debug purpose it is quite frustrating to have to load a .fbs file (you have to know were they are). I don't know if something more straightforward could be possible. What about generating the needed information inside the .h so that the DebugString function knows the schema ? And eventually make it optional.
• Thank you for adding it to the documentation ! I saw in another issue that the reverse order was more an historical decision, probably a support for the normal order could be nice ? To avoid new users to focus on this.

Versionning:

• So If an old program that does not contain the required field in the .fbs and does not use the verifier, receives the new required field in the buffer it will just ignore it ?
• Same question, if an old program doesn't have the "deprecated" attribute on the required field and does not receive it because it has been deprecated in the new version ? I guess it will crash no ?
• I was expecting to completely remove the field from the .fbs file, not just renaming it. Or i guess we need to first deprecate it everywhere and then remove it ? What would be the best strategy ?

Performance:

• I was just looking for comparison, I didn't check much the benchmarks, and we don't use java so it is fine. I was just wondering why it is a slower than protobuf in some languages. Could it be more efficient with C/C++ binding library ? Like Cython/JNI
• It seems negligible. What takes more memory ? The header for each field ? Aproximately how many bytes per fields ?

Thank you for taking notes of my insights, hope this get materialized soon :)

[ghost]

High level:

• Yup, no way to get an STL container to point to existing memory.
• Protos can be parsed without knowing the schema (though the printout won't contain field names).. which you can't do with FlatBuffers. Yes, we could solve it using generated code, which would be nice.. I'll look into it.

[ghost]

• I don't think we'd want to support both forward and reverse order, as that would create too much complexity in the code base. We'd have to switch entirely to forward building, but this would be an API breaking change. Worth considering, but it's a pretty big step at this point.

Versioning:

• Yes, new fields are just ignored by old code, required or not. Even if the old uses the verifier, it was compiled from an older schema that didn't have the field, so it won't require it to be present.
• Unless the deprecated field was required, all fields are implicitly optional, so any old code should have been testing for their presence before access (if it was a non-scalar.. scalars will just be a default value).
• You currently can't remove it from the .fbs. The best I can do is allow you to write something like deprecated_field where right now you must write some_name:some_type (deprecated). The schema compiler needs to know there used to be a field there.

Performance:

• It is likely other languages can still be made faster, we just haven't done much work on them personally. A binding through C could be a speedup for really slow languages like Python, but for decently fast languages like Java, going through JNI is likely slower than just doing it in Java, since the most basic field lookup is just using 2 offsets. Also, doing it all in Java is friendlier since Java users likely do not want to have to deal with having platform specific C++ code to compile and distribute.
• It really is a very different encoding. We always store scalars as the declared type, so a long will always take 8 bytes, whereas Protobuf uses varints. Protobuf needs one or two bytes preceding each value for type/id, whereas we have vtables (which may be amortized or not), alignment etc. So it strongly depends on your schema and your actual data how efficient that will be.

[maingoh]

Thanks, everything is clear for me now.
Hope this issue gave you some areas of improvement ! I let you close it :)

[zejal]

Hello, sorry for jumping in this conversation, but I am not clear why we need the deprecated field. I mean if we have for example: table { c:int (id: 2); b:int (id: 1); } where a field has been removed entirely, should not we generate a vtable with entriies 0, 1 and 2 and entry 0 is always 0 offset (always empty). Is not this what does "deprecated" property ?
Of course it's entirely possible I missed something.
Thanks a lot

[aardappel]

@zejal yes, when using ids, we could simply leave out fields entirely.

However, we still need a way to do that when not using ids.

Also, even when using ids, because it is quite important that ids are assigned sequentially for performance, it is nice to be able to check that with an error. For example, if you add a new field and skip an id, there's no way to tell the difference between that being an error or an intentional deprecated field.

[zejal]

I see thanks. Would it be safe then to simply put some "placeholder" with arbitrary type ? ie go from
table { a:int; b:int; c:int } to: table { deprecated0 : empty (deprecated); b:int; c:int } where "empty" is some type (an empty table for example). The question comes from the fact we might generate FlatBuffers schémas from our own idl format and we'd like to be able to remove fields complelely, at the cost of having explicit ids. So we'd lose name and type.

[aardappel]

Yes, a placeholder would be fine. In fact that was one of the suggestions above: the ability to have placeholders without a name or a type. Using some kind of type specifically for this purpose sounds like an ok alternative.

[aardappel]

@maingoh You asked for a DebugString that works without loading a schema, it just arrived here: https://github.com/google/flatbuffers/commit/72a99abfb7db64dc49720b28b41f382b5ec7cde0

[maingoh]

Wow, I didn't expect it to arrive as fast. Thanks for your great work! Also does it handle serialization to json?

[aardappel]

@maingoh : this new code converts a FlatBuffer to a string which as close as possible to JSON (though at the moment, FlatBuffers' more permissive JSON, i.e. there are no string quotes around field names).

There is no parsing of JSON to FlatBuffers using this mini-reflection yet, which would be possible, but a fair bit more complicated. For the moment, best to use the existing Parser for that purpose.

[maingoh]

Alright I was just asking for an export to json as in some cases we need to give our user an human readable result (from a buffer) which would be easier to parse for them if it was strict json.
Just giving you my use cases, don't feel forced to implement those features only because of me ;)

[zejal]

If I remember correctly there is already a facility to convert back and forth from JSON and FlatBuffer binary format ? Certainly from JSON to FB (use it routinely). The only thing that JSON output does not handle as far as I know is the eventual multiple references to same table in the structure.

Best.

De : Hugo [e-mail]
Envoyé le :jeudi 28 septembre 2017 22:33
À : google/[e-mail]
Cc : [e-mail]; [e-mail]
Objet :Re: [google/flatbuffers] Existential questions and insights (#4385)

Alright I was just asking for an export to json as in some cases we need to give our user an human readable result (from a buffer) which would be easier to parse for them if it was strict json.
Just giving you my use cases, don't feel forced to implement those features only because of me ;)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/google/flatbuffers/issues/4385#issuecomment-332955547, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AZSv1mlnSMjBgeYYMJJFkiwI-8yVx48mks5snAJ6gaJpZM4OZa2h.

[aardappel]

@zejal yes, there's been bi-directional FlatBuffers <-> JSON right from the start. But it involves parsing a schema at run-time.

This new functionality does FlatBuffers -> JSON without loading or parsing a schema.

[mfarrugi]

Just wanted to chime in and say we choose protobuf for a project that wasn't performance sensitive because --gen-object-api isn't supported for java/python (ie. https://github.com/google/flatbuffers/issues/4769).

In general, I think having the simpler API would make flatbuffers very close to being strictly better than protobuf.

[aardappel]

@mfarrugi I agree we should have it for every language, it just hasn't happened yet. Contributions welcome.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity for 1 year. It will be automatically closed if no further activity occurs. To keep it open, simply post a new comment. Maintainers will re-open on new activity. Thank you for your contributions.

[backnotprop]

Most people would use CreateVector which adds all elements in one go. If doing it by element is not document clearly, this is an oversight, please file an issue.

@aardappel Hey there, does CreateVector add them in reverse order?