Kotlinx.serialization: Performance issue?

[riclage]

I'm trying to compare the performance of kotlinx.serialization with Gson and Moshi and I'm finding it significantly slower. Here is a PR with the three being compared using jmh.

The original project comparing just Gson and Moshi is by Zac Sweers. In my PR, I converted the models to Kotlin and added the benchmark for kotlinx.serialization using JSON.parse() and JSON.stringify().

I'm wondering why the performance of kotlinx.serialization is so much lower than the other two. Is there anything I am missing? I would like to investigate this further but not sure where to start. Would appreciate any pointers on where to start looking.

Thanks!

[elizarov]

Thanks a lot for sharing. We have not looked at the performance yet (as the project was mostly focused on feature-wise exploration of potential design approaches) and it is worth looking indeed.

[wizzardo]

I also did some benchmarks, and kotlinx.serialization doesn't look good
https://github.com/wizzardo/json-benchmarks

[elizarov]

I've investigated toJson performance on that benchmark. Most of CPU time goes to PrintWriter.print method. There are no visible problems in the serialization framework itself. On the other hand, moshi (the fastest) uses okio.Buffer to write UTF8 strings, which seems to have the fastest routine for that purpose. We will not do anything specific about it until we have pure Kotlin IO library with the correspondingly optimized code (it is work in progress)

[elizarov]

On the other hand, fromJson performance is limited by the cross-platform nature of the JSON parse. We don't have StringBuilder.setLength in common stdlib, so we cannot get rid of extra string allocations in common code. See https://youtrack.jetbrains.com/issue/KT-18910

[elizarov]

I've improved JSON.parse performance (see #102). Now it is better than moshi, but still slightly slower than gson on the above benchmark. I'm getting the following results on my machine:

Benchmark                                   Mode  Cnt     Score    Error  Units
SpeedTest.gson_reflective_fromJson         thrpt   20  1206.518 ± 16.611  ops/s
SpeedTest.gson_streaming_fromJson          thrpt   20  1155.199 ± 47.725  ops/s
SpeedTest.gson_streaming_fromJson_buffer   thrpt   20  1066.076 ± 21.565  ops/s
SpeedTest.kserializer_fromJson             thrpt   20   996.613 ± 18.420  ops/s
SpeedTest.moshi_reflective_fromJson        thrpt   20   680.343 ± 11.271  ops/s
SpeedTest.moshi_streaming_fromJson         thrpt   20   757.944 ± 25.096  ops/s
SpeedTest.moshi_streaming_fromJson_buffer  thrpt   20   861.485 ± 20.341  ops/s

The only reasonable explanation for gson's better performance seems to be in the fact that gson converts input data to CharArray and then reads chars directly from there, while we are keeping it in String and reading chars with charAt and that requires an extra dereference and, maybe, extra range checks that HotSpot cannot eliminate.

[JakeWharton]

As a user of this library I welcome improvements to its performance!

But for what it's worth, this is a case that neither Gson and especially not Moshi are really optimizing for. We don't consider serialization to/from a String or a character array to be their common usage. Moshi is optimized for streaming to and from bytes and especially so with other libraries that use Okio to move bytes around (i.e., Retrofit and OkHttp).

[elizarov]

I've also optimized JSON.stringify (see #102, too) by getting rid of StringWriter. Now I'm getting:

Benchmark                                 Mode  Cnt     Score    Error  Units
SpeedTest.gson_reflective_toJson         thrpt   20   852.561 ± 15.620  ops/s
SpeedTest.gson_streaming_toJson          thrpt   20  1022.266 ± 17.730  ops/s
SpeedTest.gson_streaming_toJson_buffer   thrpt   20   522.309 ± 10.330  ops/s
SpeedTest.kserializer_toJson             thrpt   20   985.452 ± 17.961  ops/s
SpeedTest.moshi_reflective_toJson        thrpt   20  1204.622 ± 26.366  ops/s
SpeedTest.moshi_streaming_toJson         thrpt   20  1248.495 ± 22.924  ops/s
SpeedTest.moshi_streaming_toJson_buffer  thrpt   20  1508.646 ± 27.443  ops/s

It is still slower than moshi precisely because this benchmark does not actually measure converting JSON to string with moshi, but uses moshi's ability to write to byte array with okio. The corresponding support of JSON-to-bytes in kotlinx.serialization has to wait until we have cross-platform byte buffer with fast utf8-writing routines in kotlinx.io.

[sandwwraith]

As of 0.5.0, these fixes among with other improvements are included in build. Please feel free to share updated evaluations here.

[wizzardo]

I updated my benchmarks, kotlinx.serialization looks much better now https://github.com/wizzardo/json-benchmarks

[ZacSweers]

Updated benchmarks here as well: https://github.com/hzsweers/json-serialization-benchmarking/pull/4

kotlinx serialization has improved, but so has gson (drastically) and moshi. Moshi now also has native kotlin support for both reflection and code gen

[sandwwraith]

I think this can be closed for now, since kotlinx.serialization is not at the bottom of benchmark :). Of course, improving performance further would still be our concern in future work.