Flatbuffers: FR: Support github.com/Kotlin/kotlinx.serialization

[salamanders]

I'd love to try flatbuffers in Kotlin on Android. Kotlin already has a framework for serialization, and FlatBuffers supports Java... so it should be easy, right?

Mirror image of this issue at https://github.com/Kotlin/kotlinx.serialization/issues/57

It would help introduce FlatBuffers to those learning Kotlin on and off Android.

[aardappel]

As Kotlin can call any Java, yes, this should be straightforward.

Not sure about using a serialization framework, as most of those assume there is an unpacking step into objects, which FlatBuffers doesn't have. You're better off using the existing API as-is if you care about efficiency.

There was an effort to write a native Kotlin FlatBuffers implementation at some point, but that stalled.

[salamanders]

There was an effort to write a native Kotlin FlatBuffers implementation at some point, but that stalled.

bummer! I got as far in the instructions as "write a schema file" instead of class annotations, and realized it took some setup and configuration to use. Before I dove in, I was hoping a Kotlin implementation would be a beginner-friendly onramp to using flatbuffers without the cognitive overhead - the FlatBufferStream(FileOutputStream(file)).use{it.writeObject(myAnnotatedObjectInstance)} hello-world version.

[salamanders]

In theory could a Kotlin Data Class represent the schema, allowing users to completely skip the entire "compile" step?

[rockerhieu]

Now that kotlin-native and kotlin-multiplatform are becoming more popular, please support this.

[aardappel]

@rockerhieu do you know who could make a Kotlin port?

[Lakedaemon]

I ported flatbufffer to kotlin (0.9) in mai 2016, see there https://github.com/google/flatbuffers/pull/1190
(closed for lack of activity). Feel free to use it.

I tried to get it merged, but there were way too many obstacles for that so I lost interest. And I won't spend anymore time trying to get it merged but, If a kotlin port gets eventually merged, I'll gladly contribute

The kotlin port is fully working and also more type safe (avoiding integers for flatbuffer construction) than the java branch of flatbuffers. Also, it is much simpler to use as it has a kotlin friendly api.
A kotlin user should definetely NOT use the java branch of flatbuffers (but you can if you like pain)

With the release of kotlin 1.0 and lately 1.3 (typealiases, inline classes), there is still some room to make the port even better.

Also, If I'm not wrong, some of my merged commits in the past allow supporting different versions of kotlin for flatbuffer (so you can have fb for kotlin 1.0, fb for kotlin 1.3, ..)

Lastly, the last commit of my port might not be the best as I introduced a Writer class which (retroactively) was probably unneeded. But the commits before that implemented kotlin for flatbuffer without a Writer in a clean way. So, don't hesitate to look at all commits to see what you can salvage and what you can't.

Best regards
Olivier binda

[aardappel]

Yup, if someone can pick up @Lakedaemon's work, that be cool :)

[Lakedaemon]

I had a quick look at the flatbuffer codebase and it looked like they warmed up to the idea of having (language specific) code writers (there is a CodeWriter class now that wasn't there 3 years ago).
So, my last commit should be good too with minor modifications.

I would really like to see kotlin supported for flatbuffer as it would benefit kotlin and the fast expending kotlin community (3 years ago, kotlin was a new language so maybee it was not a prioriry to the flatbuffer guys to get it supported. Now that google has endorsed kotlin and uses flatbuffer in tenserflow lite, maybee it will be easier to get it merged as flatbuffer looks like a google backed project (though it may not be)).

With (really) minor changes my fork should be up and running in no time and support the latest features of Kotlin. I'm available and willing by mail to discuss technical details/changes

I forgot that Kotlin got support for unsigned values in version 1.3.
As it wasn't available 3 years ago, I had to implement it using the available bit operations of the time,
but now, proper unsigned value support (powered by inline classes underneath) should be implemented for kotlin in flatbuffer

Also, another reminder : DO NOT use the java branch of flatbuffer inside kotlin software :
1) it is not null safe (the kotlin port is)
2) it will prevent you kotlin code to be compiled natively or for the browser (the kotlin port enables that)
3) the Java api is gruesome (it will still take a few java iterations to get java's api on par with kotlin, now that they are on the right track), look at the documentation part of the kotlin branch to look at how a complicated flatbuffer should sanely be constructed

Also, there are some really bright ideas inside flatbuffers :
1) no parsing, mmap and everything is ready for computations
2) a flatbuffer may (somewhat) implement a type system
3) schema are flatbuffer (this one is mindboggingly awesome)
There are lots of things that may be possible with flatbuffer as an underlying format

But there are a few caveats though :
1) Unions suffer from a bad design and we all have to work around that now :/
2) strings : flatbuffers want to be an universal format for all languages, but languages have very varying features and don't always use the same format (Java use modified utf16, C++ use utf8). As a result, the flatbuffer api for strings will be useless for many languages. You'll have to work around that with a buffer of other types (shorts for java, kotlin, groovy, scala and friends) and a cast (to chars and CharSequence)
3) There might be a performance issue with enums but I'm not sure (it was 3 years ago, I don't remember, I may be wrong)
4) Flatbuffer has initially be released in 2014. One wonders why it takes so much time to support more languages and features, if it is so groundbreaking and revolutionary. Especially since implementing support only requires writing strings.

[rockerhieu]

@Lakedaemon this is a great start.

I just want to add that your work is a kotlin/jvm implementation of flatbuffers. There are still kotlin/native, kotlin/js, kotlin/objc on the list.

[Lakedaemon]

Yeah. You are right about that.
Most of the work already done should be (with non-major effort/pain) easily transformed into pure kotlin though
(as flatbuffers is mostly about turning a binary ByteBuffer (blob), into a structured Type with thje right interfaces, lots of bits operations and integer computations (for index, padding, width, complicated but luckily already done).

There should still be some work with adapting the api/structures to the various platform.
But that's the fun part (actually writing a kotlin code generator for flatbuffer is fun work)

There should be great benefits for (hopefully) no major effort.

[Lakedaemon]

I would like to amend the previous comment.

Though I understand the theory about multiplatform kotlin project and though I have played a little bit with that, I don't have enough experience with multiplatform project and I don't have the necessary knowledge and practical experience about the js, objc and native platforms to give good enough advice about those platforms.

For example, though I said that it is possible to write pure kotlin code for multi-platform fb, it won't have the best performance on all platforms. To get that, it will (obviously) be necessary to adapt the api and the classes to each target.

Other fb targets, like might be very good inspiration on how to do things on non-jvm targets like
C++ or go for kotlin/native and kotlin/objc
javascript for kotlin/js

So, it is not that bad as it is not like starting from scratch.

[aardappel]

@Lakedaemon I was already fine having Kotlin support 3 years ago, otherwise I would have not reviewed the PR back then. But the PR as-is still needed work that was never finished.

Java's API is not great because the language is really limiting in what it can represent efficiently. Efficiency is more important than API design in FlatBuffers.

Your caveats:

1. They are slightly un-ergonomic, I would agree, but that's the way they fitted best in FlatBuffers 1.0.
2. Not only do other serialization formats almost universally use UTF-8 (e.g. Protobuf), in general UTF-8 has "won" as a format. It unfortunate there are still languages/systems that use UTF-16. I do not see a way this could have been handled better. Certainly UTF-16 would have not been a good default, and supporting both would also not have been helpful, as now you need to deal with multiple formats depending on who wrote the data. If someone wants to use FlatBuffers entirely with UTF-16 languages then of course that would be better to keep it in UTF-16, but most use cases of FlatBuffers are more cross-platform or cross-language than that. And people can store UTF-16 in an [short] if they really want to.
3. Not sure how Kotlin implements them, but they should be integer constants, not classes.
4. Most work on FlatBuffers is done by the community. There are endless languages and features, they are implemented when someone wants to.

[Lakedaemon]

Your made your points. I can understand that.

On my side, my first PR to add (experimental) kotlin support was made the 28 Nov 2015.
I quit after the 19 Apr 2016, without managing to get kotlin support merged because I was burning out (the barrier to contribute to fb is too high, IMO) and also, because of

Do you know anyone in the Kotlin community that is somewhat of an expert on the language that would want to help with a code review of this PR?

(and though I did contact andrey Breslav to ask him for a review, the jet brain team never did one).

Imagine if you had to go and ask Bjarne Stroustrup for a review, to contribute to some project on the internet...

It could have gone differently :
1) merge the code with an experimental tag (warning that it may not be ready for production use), that way other kotlin users would have used it and sent feedback, improvements, bug reports, code reviews from experts like you wanted
2) IF the code was good enough and battle tested, it could then be promoted to "stable"
3) by now, you could have had 3 years of test and feedback for the kotlin support

But it is your right to lead fb differently. This guy https://github.com/google/flatbuffers/pull/5082 and his pr won't probably come back though ;)

[Lakedaemon]

Also, the Go and Kotlin "revolutions" are mostly about api design, productivity, safety, ...

If all we wanted was absolute performance, we would still be developping in C++.

[aardappel]

We have so far merged support for 10 or so languages, and all of them have had to meet certain quality standards before they got merged. Nothing goes into FlatBuffers without review being completed, we cannot merge unfinished code that still has issues, because a) people will start using it, and if the code still has bugs or the API will still change, that has to be avoided as much as possible, experimental tag or not, and b) because many PRs are made by volunteers and we can't count on them finishing the job. They have to have a minimum bar of finished-ness when merged.

So I am sorry to hear the long review burned you out, but there is no other way to maintain quality and direction in a project like this. We cannot be managing levels of stability for each different feature, and we only have one branch.

A long review is also a sign that the PR submitter wants to do things his own way rather than follow the direction of the project. For pretty much all PRs that were abandoned after long review that is the case, and it certainly is for the one you linked.

Asking for somehow who is "somewhat of an expert" in a new language is very different from expecting Stroustrup to review. I am not a Kotlin expert so getting a second opinion before we commit to a certain style of API is important, especially since we didn't seem to agree on some of it.

And it is cool that Go and Kotlin are about productivity etc, but FlatBuffers isn't. The whole point of FlatBuffers is efficiency. If I wouldn't need efficiency, I would not be using FlatBuffers, there are probably "easier" serialization systems out there.

There are cases where a feature can be both easy to use and also fast. That's great. But if the two are at odds in FlatBuffers, the latter has priority.

[Lakedaemon]

1) The Kotlin code produced by the generator was never reviewed for the reasons you mentioned (there is no knowledge of kotlin among the mergers of fb). Yet, what preventing merging and made reviews so long was apparently the kotlin side. Isn't something is wrong here ?

Among all the people working at google, it was not possible to ask to a googler that knows a bit of kotlin to do a proper review ? for a google supported project ? that serves the interest of google ?
Or maybee the google-employed fb mergers cannot ask other googlers for help but contributors should ?

When you pay an item in a store, you come with a banker that vouchs for you or you let the credit card device call the bank to make sure there is money on the card ?

2) With no knowledge nor experience of a target langage, arguing that a long time user of the target language wants to make things his way (maybee the way the target language is meant to be used and not the C++ way) doesn't feel of ?

3) performance and allocations :
If the target language doesn't have primitives (like kotlin) but only objects allocated on the heap (like Kotlin), you can't avoid allocations, even for ints because ints are objects in kotlin.

What you want to avoid is BIG allocations (there is a lot of reuse in the short allocations, they are fast)

And so, allocating memory to have an int, or allocating memory to have an (Type)-> Int function whose code is only "return int" is the same performancewise but the later makes things type safer for fb construction especially since lambdas can be inlined in kotlin (if you use them the right way, like when building a complete fb in one line). So you get the same performance as int but with added type safety, and nice completion from the ide (you don't get that with ints)

4) 10 target languages barely makes fb a cross language serialization/deserialization library, like it is marketted. If you have code that isn't written in these 10 langages you hit a wall.
Also, some of those code gererators are old : java 11, go 2, etc.. are coming.
Does fb support language versioning ?

The efficiency of flatbuffer comes within the limits of the target language.

5) But everything comes down to

a) people will start using it, and if the code still has bugs or the API will still change, that has to be avoided as much as possible, experimental tag or not, and b) because many PRs are made by volunteers and we can't count on them finishing the job. They have to have a minimum bar of finished-ness when merged.

I would argue that a working code generator that is used for production has a minimum bar of finished-ness.

Most code generators are written by volunteers, most would maintain them (because they use them) and keep on improving them (because they use them in projects) if given a (real) chance. Just don't drive them away.

If fb can't count on volunters, fb is screwed because the fb people cannot write code generators nor maintain them (fb people don't use the other weird languages) nor improve them.

I bet that, among the 10, the only decent fb code generator is the C++ one (because it has a contributor, with deep C++ knowledge and merging powers).

6) and

And it is cool that Go and Kotlin are about productivity etc, but FlatBuffers isn't. The whole point of FlatBuffers is efficiency.

The whole point of fb is efficiency WITHIN the limits of the target language.
The flatbuffer api for the target language MUST satisfy the necessary language-friendliness to be usable in the language (or we would just usea C code generator and link it's foreign generated code)

C++ or Java verbosity and 10 lines to do something that is not safe and ARGUABLY more efficient is a big no no (and using UNSAFE methods like it was suggested in the go thread is not acceptable for a lot of business out there).

[Lakedaemon]

There are cases where a feature can be both easy to use and also fast. That's great. But if the two are at odds in FlatBuffers, the latter has priority.

But you could have the best of both world.

You could have a safe and usable code generator 1 for language A
And a mega fast, unsafe code generator 2 for language A

It is not duplication of code (some code can be shared) as it brings 2 different functionnalities
And you wouldn't have double the work, because you would not be the one maintaining it

[Lakedaemon]

Anyway, good luck to fb. (end of rant)

[aardappel]

@Lakedaemon
Nope, if you check my previous review comments, almost none were about Kotlin, but instead about how you were implementing the generator (in C++), and you didn't appear to want to address.

At the time, I knew of 0 people at Google involved with Kotlin.

Again, you want to contribute to a project that is about maximum efficiency, and then go your own direction, and ignore code review comments. That won't work. I may not be a Kotlin user, but I know a fair bit about programming language implementation, to say the least. Until you can prove to me with a benchmark that Kotlin does magic, unnecessary dynamic allocations are not welcome in FlatBuffers :)

Your generator does not have "a minimum bar of finished-ness" just because you're using it, if it ignores the direction of the FlatBuffers project. We've had probably over a 100 individual contributors, and usually code review is very friendly and it is easy to reach agreement.

While the C++ implementation is always the benchmark, many of the other language ports are actually pretty solid. They have been used by large projects without ever any serious bugs (ones that would affect serialized data integrity).

And no, we don't have to adhere to what is considered a friendly API in a given language. We don't do so in any of the other implementations either, they look quite foreign to how a normal object based API in those languages would look like. That is by design. FlatBuffers is not going to change that direction for Kotlin's sake.

[aardappel]

Since this issue became mostly a rant about the past PR, if someone wants to kickstart the Kotlin port, please open a new issue and/or PR.

[aardappel]

New effort to add Kotlin support to FlatBuffers: https://github.com/google/flatbuffers/pull/5409