Fable: JSON serialization of Fable types

[alfonsogarciacaro]

Thanks to the work of @davidtme, we're about to modify the JSON serialization in v0.7 so the $type field is not needed any more. This is achieved by creating a type schema at compile time and passing it to ofJson (see this PR if you're interested). To wrap up things we have to decide how to serialize a couple of things that have a different representation in Fable and F# as understood by Json.NET: options an tuples.

• In Fable, options are erased (think of nullable types) and tuples are JS arrays.
• In Json.NET, options are like any other union types and tuples are serialized as {"Item1": "A", "Item2": "B"}

So the question is, should we represent options and tuples (when using Fable.Core.JsInterop.toJson/ofJson) as they're in runtime or as Json.NET expects. I'd love to hear the opinion of the community on this. This could be a summary of the pros and cons of each approach:

A là Json.NET:

• Pros: No need of custom converters when using Json.NET on server side.

• Cons: The JSON and the schema become bigger (making the JSON lighter is one of the reasons to remove the $type field).

  A là Fable:

• Pros: The JSON is lighter and the same as when just using the JS native JSON.stringify.

• Cons: A custom converter is needed for Json.NET (but we could write a Json.NET FableConverter and distribute it through Nuget).

As a final note, I don't like very much having to chase 3rd-party specs (mainly because of my experience with TypeScript). If we try to adapt to json.net, we then have a _hard_ dependency with the library: if json.net changes the way it represents options and tuples, Fable's implementation will automatically _break_ which poses a maintenance burden.

[alfonsogarciacaro]

Another final note ;) At the moment, ofJson doesn't do validation so we are planning to add a validate method that will check if all the type fields are actually present in the JSON. For that option types could represent optional fields (validation will succeed even if the JSON doesn't contain the field). However, this would be more difficult (or at least less natural) to do if options are represented as other union types.

[et1975]

A là Fable (+1)
Another point of consideration is integration with 3rd parties, which are unlikely to follow Json.NET conventions.

[davidtme]

One thought, would a replacement plugin have enough access to the fable type info to be able to override the default ofjson/tojson. Could such a plugin be shipped with fable and switch on if need?

[7sharp9]

Is there an a example of how each looks in use?

[davidtme]

@7sharp9 this is from the unit tests (sorry on my phone atm)

A là Fable:
Tuple: {"a":[1,2]}
Option some: {"a":1 }
Option none: {"a":null }

Json.net:
Tuple: {"a":{"Item1":1,"Item2":2}}
Option some: {"a":{"Case":"Some","Fields":[1]}}
Option none - I don't think the test for this is correct but {"a":null } does work but should be something like {"a":{"Case":"None"}}

[srdjan]

A là Fable +1

[rhozee]

+1 for the Fable way...

[michaelsg]

The deciding factor in going with Fable (besides dying to use F# for front-end work) for my current project was that I could achieve type fidelity between the server and client. My design relies on passing messages thus type-checking is required to dispatch the messages on both the client and server. I'm happy to report it works beautifully after I made some changes (as @davidtme discovered) to work around NewtosoftJson and Fable's type encoding.

You do, indeed, need a TupleArray converter, an Option converter and a DU converter. You also need a custom binder on the server to resolve the type names. Fable always encodes type names with a period between elements, but some types (nested modules and types) use a plus-sign between the levels of the full typename in .NET. Fable also doesn't include the assembly name in the $type field.
The following could be bit smarter but works for now and proved I could make it work.

let private typeCache = new Dictionary<string, Type>()
let private getAssemblies() = 
    System.AppDomain.CurrentDomain.GetAssemblies()
let private findType name =
    getAssemblies()
    |> Seq.tryPick(fun a -> 
        a.GetTypes()
        |> Seq.tryPick(fun t -> if t.FullName.Replace("+", ".") = name then Some t else None))
let private getType name =
    match typeCache.TryGetValue name with
    | true, t -> t
    | _ ->
        match findType name with
        | Some t ->
            typeCache.Add(name, t)
            t
        | None ->
            null
type private FableSerializationBinder() =
    inherit SerializationBinder()
    override x.BindToType(assemblyName:string, typeName:string) = 
        if not <| isNull assemblyName
        then base.BindToType(assemblyName, typeName)
        else getType typeName
    override x.BindToName(typ:Type, assemblyName:byref<string>, typeName:byref<string>) = 
        assemblyName <- null
        typeName <- typ.FullName.Replace("+", ".")

I prefer Tuples as arrays - this also matches Typescript's notion of Tuples.
Options being either null or a value works perfectly in the generated Javascript. I don't think serialization should change that.

I don't yet understand how we can get rid of $type by passing in a compiler-generated schema. What if the type of the object being deserialized is unknown to the deserializer? My current work depends on this ability. Also, not all my types are in one Fable project.

Or is this an optimization in cases where a target deserialization type is known? I could understand optimizing in cases where the type is known to ofJson. All type information could be omitted from the serialization.
How would toJson know if it should include $type (because the receiving deserializer isn't expecting any single type) or omit it because it is?

My position seems to be the direct opposite of @davidtme.

• types are shared between server and client.
• use toJson with obj/any and produce JSON that the server can deserialize without knowing the expected type. .NET runtime type checking :? works on resulting object.
• use ofJson on some generic JSON object from the server without knowing what that type is and still end up with a proper JavaScript instanceof the same type on the client.

I don't mind paying the cost of having $type in all my serialized objects in exchange for full type equality front- to back-end. David wants a more size-optimized JSON with looser type-coupling. Hopefully, we can both get what we want.

[alfonsogarciacaro]

@michaelsg Thank you very much for your useful comments. Actually, we were thinking of replacing the current JSON serialization, but you're absolutely right it'd be better to have both available so we'll do that. About type names, I was doing some simplifications, but it's possible to include the assembly name and the + sign from Fable. I guess we should do this to make type exchange easier in your case.

@davidtme I actually thought of that but forgot to include it in the issue description, sorry. You're right a plugin should be available to modify the behavior of ofJson if needed. This should be already possible with a replacement plugin but I'll check.

[davidtme]

I think when I started the change it was to completely get rid of the $type but it became clear that in some cases you need. If you provided the $type it’s used: (see test https://github.com/fable-compiler/Fable/pull/426/commits/fd70e4b9cf532161862be712fbdd5c764b860571#diff-74539a3c408918a75789131a9040dab9R281)

I to share my client and server f# model passing message back and forth but the “it’s after I made some changes to work around NewtosoftJson and Fable's type encoding.” then a list of caveats.

Don’t get me wrong, I don’t like the json.net tuple and options style of json - in an ideal world json.net would change. When I started I tried to use the typle-array and option type converters but this quickly became a pain so I started using the defaults - now I’m using fable I’m having to add them back in. We not have lot's of json.net settings versions

The internal structure of tuples and options won’t be changed, once you have the values in fable tuples will be array and options will be null.

I think if there is a plugin(s) (maybe a flag to turn it on?) then this question will be answered – just a bit more work on the edge cases of $type

@alfonsogarciacaro Random thought - could the default tojson/ofjson included js code live in a plugin (on by default) so only added when needed, if you switch to a different mode you get a different js include and schema?

[michaelsg]

I'm not sure this is an either/or situation. I see it as 2 ways of serializing and deserializing and I think you will know which is needed in each case. I can easily see using both techniques in a single application.

Example: I'm currently passing objects loosely and relying on their runtime type on the server and client to distinguish them - requires $type in both directions. Let's say I have a message to send that has a really big payload and I can reduce it significantly by omitting $type info. The outer message object contains an embedded typeless JSON string. Once this message/object arrives at the recipient, the recipient - knowing what type to expect in the payload - uses the untyped/schema deserialization on the outer message's payload.

I would not be offended by 4 separate functions:

• toUntypedJson - omits $type
• toTypedJson - includes $type
• ofUntypedJson<'a> (requires target type 'a <> obj, compiler emits schema for deserializing 'a)
• ofTypedJson - uses $type - as now

This is code just out of my head....

//... some implementation using a serializer on the server to serialize and omit typing info
let toUntypedJson o = //.....
// ... server impl including $type info
let toTypedJson o = // ....

type Message = { Payload: string }

let innerPayloadObj = Object2() // big object
let messageString = toTypedJson { Payload = toUntypedJson innerPayloadObj }
sendToClient messageString

// meanwhile on the client...
let messageString = getNextIncoming()
let o = ofTypedJson messageString // uses $type to create object

match o with
| :? Message as m -> // btw, this requires UnboxFast or something like that and won't compile in Fable.
   // let m = o :?> Message // required for Fable   
   let innerPayload = ofUntypedJson<Object2> m.Payload // uses schema

[wizzard0]

$type may be useful for inheritance, but in most other places I would really prefer the A là Fable approach.