Messagepack-csharp: JSON Deserialize

[MohMehrnia]

how do json deserialize and serialize?

[AArnott]

This question is answered in the project README.

[MohMehrnia]

In README says used MessagePackSerializer.SerializeToJson for Serialize but for deserializing is not?

[AArnott]

Ah, I thought you were talking about _converting_ JSON to msgpack, which is documented in our README.

If you want to deserialize JSON, this isn't the library for it. This is a messagepack serialization library. Although it supports _serializing_ to JSON (for debugging purposes) it does not deserialize from JSON.

You could use our ConvertFromJson API to convert JSON to msgpack and then deserialize msgpack if you really wanted to, but ... why?

[MohMehrnia]

Oh... OK Thanks I wish to find a JSON Serialialize and deserialize fastest of newtown JSON

[AArnott]

Maybe check out https://github.com/neuecc/Utf8Json

[MohMehrnia]

yes I see this component but isn't upgrade a long time(3 years ago), are safe for production application?

[AArnott]

@neuecc would be best to comment on it. System.Text.Json is another option, which is supported by Microsoft.

[MohMehrnia]

utf8json is the best performance vs System.Text.Json(), I need a JSON process library fastest than newton.json or system.text.json

[neuecc]

Unfortunately, System.Text.Json's serializer design of performance is very poor.
It will be impossible to get high performance unless fully refactored.
But maybe it will not come.
However, it is certainly maintained by Microsoft.
I want to restart Utf8Json, but I don't know when.

[Haihg]

I want to restart Utf8Json, but I don't know when.

Thumb up if if you can, please! Companies like where I'm working, security policy don't allow using open source libraries of a few years old, so it will be skipped even after evaluation comes back with good result.

utf8json is the best performance vs System.Text.Json(), I need a JSON process library fastest newton.json or system.text.json

A very good summary of serializers for .net core / full are here
https://aloiskraus.wordpress.com/2019/09/29/net-serialization-benchmark-2019-roundup/

I've done a bit of research on our actual data (which are much more complicated, on a real corporate application, using our method):
1. Most libraries are easy to implement, except for libraries coming from google and ...
- Libraries from google (protobuf, flatbuffers) are very time-consuming to set up, lack of type support, usage are also different - if you work directly with these, new entities are generated, so you can't re-use existing entities in the DB layer, unless if you rewrite your db access layer to work with the generated entities. So, good for a brand new project (but still, expect the complication.) In return, these are cross-platform (Java, C, C#, Ruby, etc.) e.g. suitable for micro services, gRPC and applications that communicate from multiple platform. A big no for us as this is not our situation.
- MessagePack/Utf8Json are at a medium level of difficulty to setup due to full and strict contract requirement. We currently use Json.Net and it allows writing a custom resolvers for properties, even if not with this Json.Net is contractless (same as Apex.Serialization) so these saved a lot of time. When we started testing out MessagePack and Utf8Json, it was kind of pain to start with.
2. In terms of performance:
- Json.Text and Json.Net are very similar (in some cases, one is better than the other, in other case the opposite)
- Apex.Serialization is 12 times faster than Json.Net (note that Apex.Serialization doesn't work cross-platform, even for different .net versions, so it's best for doing a binary deep clone or being used from the same application/server)
- Utf8Json is consistently 2 times faster than Json.Net
- MessagePack is 3 - 6 times faster than Json.Net (depending on how you set up the keys)
3. In terms of data size:
- Json.Net was tested with optimized data size (no default values, no null values) - note if default values are excluded it may have issue with javascript in such a case like an integer field now becomes undefined (instead of 0 within C#)
- Json.Text is 1.6 times larger than Json.Net (probably because it doesn't have an option to exclude fields with default values)
- Utf8Json is 2 times larger than Json.Net
- MessagePack size is 1.5 times larger than Json.Net
- Apex is 12 times smaller than Json.Net (coincidentally it is 12 times faster as well)
- After compression, all libraries gain a 4-10 time smaller size. Worth noting MessagePack has built-in Lz4 compression and performance is better than using a separate Lz4 library (Lz4 doesn't have much overhead compared with serializing so this difference becomes marginal)
4. Other notes:
- Test was done with both small and large data
- Performance is calculated as average of (serliase + deserialise) / 2 (some other tests measure serialize and de-serialize performance separately, some do benchmark on de-serialize time but we do average of these because our data often involve 1 write (to cache server) and then one read)
- Most libraries target .net standard 2, except Json.Net will run on almost all .net framework
Apex and MessagePack (probably all other binary libraries) will crash the application if there are circular references in the data, e.g. leaving application no option to log error or recover from exception. This should not be an issue if the data is handled for circular references carefully.
- Except Json.Net, no other libraries accept circular references, so an extra step is needed to handle this where the data structure has circular references.
- Feature-wise, Json.Net has everything by default.
- As for serializable member, Json.Net are flexible (fields, properties, both, or custom resolver), MessagePack and Utf8Json deals with properties only (private or public) and MessagePack only deals with fields (e.g. including backing fields for properties) - Haven't been able to figure out which is actually the best way, but so far everything in all libraries are doing their job well.

[MohMehrnia]

Apex.Serialization only binary serialization and I need JSON serialization

[AArnott]

MessagePack (probably all other binary libraries) will crash the application if there are circular references in the data

This can be avoided nowadays by setting a max object graph height setting in MessagePackSecurity.

[Haihg]

MessagePack (probably all other binary libraries) will crash the application if there are circular references in the data

This can be avoided nowadays by setting a max object graph height setting in MessagePackSecurity.

Thanks. I'll try. Can I please also confirm if MessagePack uses direct (unsafe) memory access at all?

[AArnott]

The distinction between safe and unsafe memory is blurred somewhat when it comes to reusable buffers, which MessagePack makes heavy use of, since memory corruption is possible when you share buffers via a pool. We do use pointers in a few places too.