Runtime: Ensure JSON serializer/deserializer can handle common F# types

[terrajobst]

We should make sure that the JSON serializer can handle types / structures that commonly occur in F#, such as

• Discriminated unions
• Option types

/cc @steveharter @ahsonkhan

[cartermp]

Should be:

• Discriminated Unions
• Record types
• Anonymous Record types (should be identical but we emit some slight special names)

[ahsonkhan]

This requires careful design to make sure it is handled correctly to meet consumer expectations. We should flesh out the requirements of what the behavior should be, especially for cases like round-tripping, etc.

[bartelink]

Suggest mining https://github.com/jet/Jet.JsonNet.Converters and https://github.com/microsoft/fsharplu for commonly shimmed features - will be monitoring this issue as I would love rebase the test codebase for the former on top of this and/or dogfood it here. Related issue: https://github.com/jet/equinox/issues/79

[AArnott]

Would the discriminated unions support be helpful in non F# cases as well? I'm particularly interested in cases of deserializing protocols such as JSON-RPC, where the incoming message type may be any of a few schemas and we need to distinguish them based on a couple of properties.

[bartelink]

Good question @AArnott - supporting management of a catchall case would also ideally be possible.
To call out one specific need - the json.net intrinsic F# support emits the fields as a sub-object alongside the case name; Jet.JsonNet.Converters and many common converters including Java ones can and do operate by representing the object as a single object node [including any marker/discriminator fields].

[Tarmil]

While this is being decided, I've implemented an F# serializer as a separate library: https://github.com/Tarmil/FSharp.SystemTextJson

[hravnx]

While I understand that the F# specific types like DUs and Records are not supported yet, I've run into strange behaviour serializing a simple Map.

Given the following F# program (running in .Net Core 3 preview 9):

open System
open System.Text.Json


[<EntryPoint>]
let main _argv =

    let m1 = Map.empty<string, string> |> Map.add "answer" "42"
    let json1 = JsonSerializer.Serialize(m1)
    printfn "JSON1 %s" json1

    let m2 = Map.empty<string, Version> |> Map.add "version" (Version("10.0.1.1"))
    let json2 = JsonSerializer.Serialize(m2)
    printfn "JSON2 %s" json2

    0 

I expect the following output:

JSON1 {"answer":"42"}
JSON2 {"version":"Major":10,"Minor":0,"Build":1,"Revision":1,"MajorRevision":0,"MinorRevision":1}}

But instead I get this:

JSON1 {"answer":"42"}
Unhandled exception. System.InvalidCastException: Unable to cast object of type 'mkIEnumerator@431[System.String,System.Version]' to type 'System.Collections.IDictionaryEnumerator'.
 ...

So the first case worked, but not the second. Is this a known limitation or a bug?

[xperiandri]

@hravnx, I think it is related to https://github.com/dotnet/corefx/issues/40949

[bartelink]

Version requires a converter even for for Json.net; suggest using a simple C# defined type to validate (I believe records don't work so be careful not to try that). Then extend to using non-record syntax in F# to define equivalent of the C# class etc. - going straight to a Map of versions is trying to do too many things at once. Also can we not pollute this thread with specific cases please - i.e. can this be moved to another issue? Another thing to simplify is to use Dictionary or |> dict in lieu of Map (thought the first case pretty much proves Map is fine, as one might expect as it implements IEnumerable)

[hravnx]

Map works. But I'll stop "polluting" now. Bye.

[ScottHutchinson]

Here is an example where the JsonSerializer.Deserialize function fails to properly deserialize the value of an enum option. The value is correctly serialized as 1 (B), but is deserialized as A. The Newtonsoft.Json.JsonConvert.DeserializeObject function works properly (using the commented-out code). Also, Newtonsoft.Json does not require the [<CLIMutable>] attribute, but JsonSerializer does.

open System.IO

type TestType =
    | A = 0
    | B = 1

[<CLIMutable>]
type TestObjectB =
     {
         test : TestType option
     }

//let jsonSerializeToString obj = 
//    use writer = new StringWriter()
//    let ser =  new Newtonsoft.Json.JsonSerializer()
//    ser.Formatting <- Newtonsoft.Json.Formatting.Indented
//    ser.Serialize(writer, obj)
//    writer.ToString()

//let jsonDeserializeFromString str =
//    Newtonsoft.Json.JsonConvert.DeserializeObject<TestObjectB>(str)

open System.Text.Json

let jsonSerializeToString obj = 
    let mutable options = JsonSerializerOptions()
    options.WriteIndented <- true
    JsonSerializer.Serialize(obj, options)

let jsonDeserializeFromString (str: string) =
    JsonSerializer.Deserialize<TestObjectB>(str)

let Test obj = 
    let str = jsonSerializeToString obj
    let obj' = jsonDeserializeFromString str
    obj'

[<EntryPoint>]
let main argv =
    { test = Some TestType.B } |> Test |> ignore
    { test = None } |> Test |> ignore
    0

[ScottHutchinson]

^^
C:Usersscott>dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 3.0.100
Commit: 04339c3a26

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:Program Filesdotnetsdk3.0.100

Host (useful for support):
Version: 3.0.0
Commit: 7d57652f33

[cartermp]

Just as a quick update, various things do work.

This example now works in .NET 5:

open System
open System.Text.Json


[<EntryPoint>]
let main _argv =

    let m1 = Map.empty<string, string> |> Map.add "answer" "42"
    let json1 = JsonSerializer.Serialize(m1)
    printfn "JSON1 %s" json1

    let m2 = Map.empty<string, Version> |> Map.add "version" (Version("10.0.1.1"))
    let json2 = JsonSerializer.Serialize(m2)
    printfn "JSON2 %s" json2

    0 

F# records and anonymous records now appear to work (nested records example):

open System
open System.Text.Json

type Person = { Name: string; Age: int }
type Gaggle = { People: Person list; Name: string }

[<EntryPoint>]
let main _argv =

    let r1 =
        {
            People =
                [ { Name = "Phillip"; Age = Int32.MaxValue }
                  { Name = "Ryan Nowak"; Age = Int32.MinValue } ]
            Name = "Meme team"
        }

    let r2 =
        {|
            r1 with
                FavoriteMusic = "Dan Roth's Banjo Bonanza"
        |}

    printfn $"{JsonSerializer.Serialize r1}"
    printfn $"{JsonSerializer.Serialize r2}"

    0 

Yields:

{"People":[{"Name":"Phillip","Age":2147483647},{"Name":"Ryan Nowak","Age":-2147483648}],"Name":"Meme team"}
{"FavoriteMusic":"Dan Roth\u0027s Banjo Bonanza","Name":"Meme team","People":[{"Name":"Phillip","Age":2147483647},{"Name":"Ryan Nowak","Age":-2147483648}]}

But DUs are still missing. FSharp.SystemTextJson is still the way to serialize F# union types.

[cartermp]

@terrajobst @layomia happy to chat about what all would be the ideal format and what kind of data needs to be dealt with

[layomia]

@cartermp, thanks - I'll reach out to discuss.