Flatbuffers: [Request] Allow size and offset type to be parameterized

[dleslie]

As of writing this issue it appears that FlatBuffers encodes _most_ lengths, sizes and locations as Integers; there are some locations where Offsets are store as Shorts. This is a waste of bytes for many applications where the encoded data barely extends beyond the maximum value represented by a byte, let alone a 32-bit integer.

As a naive test, I took some time to replace FlatBuffers' numeric representations that use Int32 with Int16, and saved around 20 bytes on-average for every object encoded in our application. This amounts to approximately 5 to 20 percent of the total size, depending on what is being encoded.

If I may be so bold, I would suggest allowing all sizes and offsets to have their encoded type parameterized by flatc. Some applications can get away with only using _byte_ to represent such values, whereas others may need a 32-bit integer.

[aardappel]

The reason it is hard-coded to 1 size is for speed: when you access these buffers, it is doing lots of quick offsets calculations behind the scenes. If there were lots of conditionals in there determining the size of the offset, or a varint say, that would significantly slow down access.

So any such changes would have to be at compile time, but would have the disadvantage of "forking" the format, a 16-bit FlatBuffers would need all tools and all code reading/writing to be in a agreement. We so far haven't done this, as it doesn't seem to be worth it.

The top 2 items here discuss a possible 64-bit FlatBuffers: https://github.com/google/flatbuffers/projects/10

Related: https://github.com/google/flatbuffers/issues/4240

[dleslie]

That's interesting; I did imagine it to be a compile time change, in the manner of n-byte FlatBuffers.

-------- Original Message --------
On Aug. 5, 2019, 11:56 a.m., Wouter van Oortmerssen wrote:

The reason it is hard-coded to 1 size is for speed: when you access these buffers, it is doing lots of quick offsets calculations behind the scenes. If there were lots of conditionals in there determining the size of the offset, or a varint say, that would significantly slow down access.

So any such changes would have to be at compile time, but would have the disadvantage of "forking" the format, a 16-bit FlatBuffers would need all tools and all code reading/writing to be in a agreement. We so far haven't done this, as it doesn't seem to be worth it.

The top 2 items here discuss a possible 64-bit FlatBuffers: https://github.com/google/flatbuffers/projects/10

Related: #4240

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[llchan]

Random idea: I think in the existing 32-bit-offsets world the root table is guaranteed to be at a 4-byte-aligned offset, which means the 2 lowest bits in the first byte are always zero in existing flatbuffers. If we wanted to do a 16-bit mode, we could potentially use the lowest bit to mark it as 16-bit mode, and mask that off when jumping to the root table (pretty cheap one-time mask per root access). That way a 16-bit mode reader could detect if it's being fed in a 32-bit message and fail fast. Newly-compiled 32-bit readers could also detect the converse. However, legacy 32-bit readers would not be protected from future 16-bit messages, but maybe that's a reasonable risk as long as the user is aware of it. As long as it's a conscious decision to opt into 16-bit mode I think it's reasonable to assume the developer has taken the necessary precautions to rebuild readers/writers.

This strategy doesnt help if we _also_ want to do a 64-bit mode (or 8-bit mode), but I think 16-bit mode is the most useful of these. Someone who needs gigantic messages can add their own framing, but it's not currently possible to make flatbuffers more compact.

[aardappel]

@llchan I like the idea of using lower bits, since in addition to having new readers being able to detect it, the offset can maybe be set such that on older readers its an illegal buffer, e.g. it fails the verifier. Current users of FlatBuffers must already ensure that either they only consume buffers from a known (local) source, or that they use the verifier.

That said, if we're going to allow other bit-sizes, we should really also have a 64-bit strategy at the same time. I've actually been asked for that more often than 16-bit. The amount of people that would like to access in-memory data stores of some kind >2GB is not insignificant, it appears.

Possibly, 16-bit support could be done in the same way as 64-bit support is suggested in the above link: by allowing it to be specified per table/vector/field. That has the advantage the format is not being forked, and since schemas with new annotations need a recompile to take effect, it would potentially not have backwards compatible issues (if you'd declare it to be illegal on existing decls). It could also allow the same efficient situation as for 64-bit: you could have a buffer where all tables consist entirely out of 16-bit offsets, and only the root vector being 32-bit offsets, getting most of the data compression with none of the size limitation.

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

[dleslie]

This is still important to me. I'm maintaining a half-baked internal 8-bit offset fork, and would prefer to see this behaviour in mainline.

-------- Original Message --------
On Jun 12, 2020, 13:33, github-actions[bot] wrote:

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

[dleslie]

Still important to me; still maintaining a branch with 8-bit offsets, lengths and etc.

[aardappel]

@dleslie I don't expect 8-bit offsets to be added to the main repo any time soon. You could try making a PR with support for 8/16/64? bit offsets, and we could see how invasive that is. uoffset_t is used in a LOT of places though, so I can't imagine it being minimal, seeing how many things it interacts with.