Flatbuffers: Is it possible to simply convert javascript object and flatbuffers?

[darklinden]

I think it is too complex for converting between js object and flatbuffers' buffer.
I think there maybe a friendly interface like below.

MyTableObj = { ... };
flatHelper = { ... };

var jsDataModuleObj = flatHelper.createObj(MyTableObj);

... // data has simple getter and setter, so you can use it like a simple object.  
... // do sth

// convert between buff and js obj
var buff = flatHelper.js2flat(jsDataModuleObj);
var obj = flatHelper.flat2js(buff);

I tried to using obj.constructor.name and typeof or anything else to mark up the data 's relationship, but it dose less help.

Is there any solutions?

Thanks.

[louisremi]

You can build such helpers to convert from a JS object to a flat buffer, but this buffer will have to be specific to your flatbuffer schema.

[ghost]

FlatBuffers is all about maximum performance serialization, and the API is as it is to allow in-place construction of serialized data. Your solution would create lots of temporary objects, causing unnecessary GC churn.

[darklinden]

Thanks for reply. I'm sorry I was sticked by some other things in several days.

@louisremi Thanks, now I 'm trying to make a code generator to do this. The only cost is every root js object may need a class name string property.
@gwvo I think there may not so many unexpected js objects. Imagining the logic like below.

logicMgr = { ... };
MyWorkingObj = { ... };
flatHelper = { ... };

flatHelper.js2flat = function(obj) {
    var builder = new flatbuffers.Builder(0);

    sth.startSth(builder);
    ...
}

sth.onMsgFromServer = function() {
    logicMgr.workObj = flatHelper.flat2js(buff);
};

sth.workingOn = function() {
    ... // do sth     
    funcA(logicMgr.workObj.x);
    funcB(logicMgr.workObj.y);
    logicMgr.workObj.n = funC( ... );
    ...

    var buff = flatHelper.js2flat(jsDataModuleObj);
    logicMgr.sendBack(buff)
}

I thought a lot. Everything that js object is used for their logic uses. The js object would been created even if you didn't use flatbuffers. Is me correct? And if I do so, may be RequireJs do help for memory costs?

PS:
My work mates says the js data object may be too large because there 's some buider function calls inside. He is about to create a new simple data object with the data copy to pass and deal with logic. Is he correct?

Thanks.

[ghost]

I'm not sure if I follow what you're suggesting, my familiarity with js is low. @evanw?

[evanw]

It's totally possible to convert from js objects to flatbuffers and back because flatbuffers can be converted to JSON and all JSON is valid JavaScript. However, right now the only way to generate JSON is using the command-line app.

@gwvo, do you have any plans for adding JSON serialization as an option to the code generators for different language targets? Right now the only way to do this is by cross-compiling the command-line app using emscripten. I assume that works but I haven't tried it.

The other option would be to create a script that could consume a schema and convert back and forth to and from JSON. I just saw that flatbuffers now has a reflection schema so this seems pretty easy to do.

[ghost]

@evanw: supporting it in other languages natively (not thru C++) would mean duplicating a ton of the code in idl_parser.cpp and idl_gen_text.cpp. Given how much duplicated code we already have in FlatBuffers, that does not sound like a good direction.

In JS you don't really need a parser for JSON of course, but you still need to load a schema (like the binary schema format that can be loaded using FlatBuffers itself), which you could then use to traverse a JS object generically and generate a FlatBuffer on the fly. More elegant than porting parsing code, but still a lot of work.

I see no easy way to get this done for most languages.

[evanw]

Yup that's what I meant by "create a script that could consume a schema and convert back and forth to and from JSON". I also just realized that since JavaScript uses a JIT, you could write a JavaScript library that generates efficient FlatBuffer code on the fly given a schema. Then everything could be done in JavaScript and there doesn't have to be a dependency on C++ at all. This would also help us with some stuff we're doing so I'll try to write a prototype tonight as an experiment.

[ghost]

Hah, so you'd generate JS code from a binary schema that knows how to convert a JS object into a FlatBuffer? Sounds pretty cool.

[evanw]

@gwvo I almost got something working but then I ran into a problem. The format in reflection.fbs doesn't encode the per-field padding value but that value is required to be able to generate buffers containing structs. Is there a reason why the value was omitted? Or is it something that should be added?

[evanw]

Actually never mind, I can hack around it: fields[i].padding = fields[i + 1].offset - fields[i].offset - fields[i].inline_size.

[evanw]

I have an initial version of the library I wanted: https://github.com/evanw/node-flatbuffers/. It converts between FlatBuffers and plain JavaScript objects. I made it because it's useful for what I'm currently working on. We're experimenting with lots of schema changes and writing all of that serialization code by hand every time was a huge pain. Other people have brought up the verbosity of the FlatBuffers API and this library can help with that too.

Although it trades off performance for ease of use, it's probably still reasonably efficient. Building JSON on the fly is slow in other languages but in JavaScript the JIT often uses hidden class transitions to generate very efficient code. There also isn't always extra GC overhead (your problem may require you to construct objects in memory anyway, this just does it for you).

@darklinden I'm not sure what you're asking. Does this library solve what you were trying to do? Note: the library I linked to is brand new and while it appears to be working, there may still be an issue or two that I haven't encountered yet.

[ghost]

@evanw: yes, I guess I assumed that value would be calculated.

This definitely looks like an awesome API, and the code (in index.js) looks simple enough.

I can imagine it is still pretty quick, as the only overhead is object allocation. I could see a lot of people willing to pay that cost.

I'd be for adding this an an optional way to do things to official API. What is missing is tests and some documentation (in JavaScriptUsage.md, since this API style is unique to JS).

[darklinden]

@evanw Great, that's maybe the thing I expect for.
I 'd do some research on this several days ago and I thought there may a lot of problems.

I'm new for javascript, so I just change the cpp code to work.

There's a function to export the default json, and functions to convert from json to flat and back.
I thought I could store the parser to make the parse faster.
Here the code, it works for me, but has not been tested much.

https://github.com/darklinden/flat2js

@gwvo It's so pleasure to get the news. Thanks for your attention.

Thanks.

[ghost]

@evanw: is there still interest in contributing this alternative API?

[evanw]

@gwvo We're actually moving off of flatbuffers so I'm not interested in spending the time to contribute a JavaScript-only API at this time. I think https://github.com/evanw/node-flatbuffers/ is a great start and it's been working fine for us (we've been using it at Figma). Right now that package is using the "flatbuffers" npm name but I'm happy to give control of that name to you if you'd like. Just let me know.

[ghost]

Aww.. what are you replacing it with and why?

Yes, renaming that package something others than flatbuffers would avoid people blindly getting that and thinking they go the official package.

[evanw]

We're replacing it with something I wrote called Kiwi. We recently added support for realtime editing of our file format so we needed a format that was good for both small delta messages and large rollup messages. It's very convenient to have that code implemented once instead of twice and flatbuffers wasn't a good fit for our delta message use case, so we're switching away from flatbuffers.

We didn't want to pay for the overhead of the v-table feature in flatbuffers or for mandatory full precision for numeric values that are usually small (most of our messages consist of these values). It's also inconvenient to work around not being able to detect field presence (we have been abusing the default value feature for this purpose but it's not ideal). I looked at using Protocol Buffers for a while since it's good for small messages but it was kind of hard to work with (the code is over 200kloc and reimplements part of the standard library), the available libraries don't let you check for field presence, and protocol buffers ended up having several inefficiencies for large messages.

Anyway, flatbuffers is great and you shouldn't feel bad that we're switching away from it! It was a good fit for our use case but our use case changed and now it's not as good a fit. It's been fantastic to work with and I really appreciate the design that went into it and how lightweight the implementation is.

I don't think I can rename the package because of the way npm works but I can give you control of the package name so you can publish the official version of the JavaScript-only package under that name once you have one.

[ghost]

Makes sense.. its not perfect for every use case. The overhead for a minimal message is quite large (20 bytes or so). I've thought about adding varints as an option, no reason we can't have them.

Is there a doc that shows the binary layout of Kiwi?

[cloutiertyler]

Evan, it looks like your Kiwi link no longer works.

[evanw]

Fixed, sorry about that. I didn't put "https://" before it and GitHub thought it was a relative URL for some reason. The correct URL is of course https://github.com/evanw/kiwi.

[evanw]

The binary layout hasn't been documented yet unfortunately. I've been meaning to do that since you asked but I haven't found the time so far. There are only a few primitive types (byte, int, float, and string) and the serialization routines are very simple so it should be pretty easy to understand the details of the format from the code. Here's the simple version in the meantime:

• Variable-length integers use LEB128 just like Protocol Buffers, including the part where the sign bit is the least significant bit for signed integers. 32-bit integers can take up to 5 bytes to encode this way. Kiwi doesn't support 64-bit integers because JavaScript doesn't support them.
• Floats are stored as 32-bit values but their bits are rotated so the 8-bit exponent comes first. If the exponent is zero (+0, -0, and denormalized numbers), the other 24 bits (the sign and mantissa) aren't encoded. This means all floats encode to 4 bytes except for 0, which takes up 1 byte.
• Strings are stored using a null-terminated UTF-8 encoding.
• Arrays are stored as a single variable-length unsigned integer length followed by that many elements.
• Structs are stored by writing out the value for each field in order, with no other metadata.
• Messages are stored as a sequence of (field ID, value) pairs terminated by a single null byte. Each field ID is stored as a variable-length unsigned integer.

There's a live demo at http://evanw.github.io/kiwi/ where you can see which bytes get produced for different data types.

[ghost]

Nice, that's pretty compact. Though it is close enough to the design of protobuf that I wonder why you didn't use that? Is this just a slightly smaller/simpler protobuf?

[kevzettler]

I am in the process of deciding on a Serialization library and bummed to discover this discussion thread. Glad to see this issue is still open because I feel it is a huge pain point. I had committed to flatbuffers and started instrumenting when I was very disappointed to see:

At https://google.github.io/flatbuffers/flatbuffers_support.html

No JSON parsing support in Javascript 'the JSON language'?

At this time I am considering using @evanw 's old package at https://github.com/evanw/node-flatbuffers/ because the API is significantly less work to integrate into my project (a js webrtc game). Would this be a bad idea?

Additionally, after reading this thread I'm now confused and debating whether I should re-evaluate Protobufs, consider Evan's Kiwi project, or maybe roll my own serialization as it may be less overhead.

[aardappel]

@kevzettler this thread is about converting a JS object to FlatBuffers, akin to the "object API" that now exists in C++. This would make a lot of sense for JS.

Parsing JSON is a different matter (even if JS they are closely related). Implementing a parser and a binary generator is no simple matter, certainly a lot more complicated than an object API (in JS, the latter would be more generally useful, since it already has a JSON parser).

[kevzettler]

this thread is about converting a JS object to FlatBuffers, akin to the "object API" that now exists in C++. This would make a lot of sense for JS.

Yes. Sorry to conflate JSON parsing with JS object-to-flatbuffer "object API " idea.

What I'm really looking for is an API like flattBuffer.fromJS(JsObject)

Is this type of "object API" currently available?

I conflated the two ideas because In my mind "JSON parsing" is: flattBuffer.fromJS(JSON.parse(JsObjectString));

[aardappel]

No it isn't available yet. If you see @evanw 's comments above he already had something like that going, but it never made it into the official FlatBuffers project. It be great for someone to pick up his work and make a PR.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity for 1 year. It will be automatically closed if no further activity occurs. To keep it open, simply post a new comment. Maintainers will re-open on new activity. Thank you for your contributions.