Runtime: Proposal: Add mechanism to handle circular references when serializing

[Jozkee]

See initial proposal with extended comments here:
https://github.com/dotnet/runtime/pull/354/files

See proposal extension for ReferenceResolver on https://github.com/dotnet/runtime/issues/30820#issuecomment-623752644.

Rationale and Usage

Currently there is no mechanism to prevent infinite looping in circular objects nor to preserve references when using System.Text.Json.

Community is heavily requesting this feature since is consider by many as a very common scenario, specially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though JSON specifiacation does not support reference loops by default. Therefore this will be shipped as an opt-in feature.

The current solution to deal with reference loops is to rely in MaxDepth and throw a JsonException after it is exceeded. Now, this is a decent and cheap solution but we will also offer other not-so-cheap options to deal with this problem while keeping the current one in order to not affect the out-of-the-box performance.

Proposed API

namespace System.Text.Json
{
    public class JsonSerializerOptions
    {
        public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default;
    }
}

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// This enumeration defines the various ways the <see cref="JsonSerializer"/> 
    /// can deal with references on serialization and deserialization.
    /// </summary>
    public enum ReferenceHandling
    {
        Default,
        Preserve
    }
}

EDIT:

We considered having ReferenceHandlig.Ignore but it was cut out of the final design due the lack of scenarios where you would really need Ignore over Preserve.

Although is not part of the shipping API, the samples and definitions of Ignore remain in this description for their informative value.

In depth

• Default:

  • On Serialize: Throw a JsonException when MaxDepth is exceeded, this may occur by either a Reference Loop or by passing a very deep object. This option will not affect the performance of the serializer.
  • On Deserialize: No effect.

• Ignore:

  • On Serialize: Ignores (skips writing) the property/element where the reference loop is detected.
  • On Deserialize: No effect.

• Preserve:

  • On Serialize: When writing complex types, the serializer also writes them metadata ($id, $values and $ref) properties in order re-use them by writing a reference to the object or array.
  • On Deserialize: While the other options show no effect on Deserialization, Preserve does affect its behavior with the following: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it.

Feature Parity (Examples of System.Text.Json vs Newtonsoft's Json.Net)

Having the following class:

class Employee 
{ 
    string Name { get; set; }
    Employee Manager { get; set; }
    List<Employee> Subordinates { get; set; }
}

Using Ignore on Serialize

On System.Text.Json:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Ignore
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WriteIgnoringReferenceLoops()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        ReferenceLoopHandling = ReferenceLoopHandling.Ignore
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "Name": "Angela",
    "Manager": {
        "Name": "Bob",
        "Subordinates": [] //Note how subordinates is empty due Angela is being ignored.
    }
}

Using Preserve on Serialize

On System.Text.Json:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
        WriteIndented = true,
    };

    string json = JsonSerializer.Serialize(angela, options);
    Console.Write(json);
}

On Newtonsoft's Json.Net:

public static void WritePreservingReference()
{
    var bob = new Employee { Name = "Bob" };
    var angela = new Employee { Name = "Angela" };

    angela.Manager = bob;
    bob.Subordinates = new List<Employee>{ angela };

    var settings = new JsonSerializerSettings
    {
        PreserveReferencesHandling = PreserveReferencesHandling.All
        Formatting = Formatting.Indented 
    };

    string json = JsonConvert.SerializeObject(angela, settings);
    Console.Write(json);
}

Output:

{
    "$id": "1",
    "Name": "Angela",
    "Manager": {
        "$id": "2",
        "Name": "Bob",
        "Subordinates": { //Note how the Subordinates' square braces are replaced with curly braces in order to include $id and $values properties, $values will now hold whatever value was meant for the Subordinates list.
            "$id": "3",
            "$values": [
                {  //Note how this object denotes reference to Angela that was previously serialized.
                    "$ref": "1"
                }
            ]
        }            
    }
}

Using Preserve on Deserialize

On System.Text.Json:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerOptions
    {
        ReferenceHandling = ReferenceHandling.Preserve
    };

    Employee angela = JsonSerializer.Deserialize<Employee>(json, options);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

On Newtonsoft's Json.Net:

public static void ReadJsonWithPreservedReferences(){
    string json = 
    @"{
        ""$id"": ""1"",
        ""Name"": ""Angela"",
        ""Manager"": {
            ""$id"": ""2"",
            ""Name"": ""Bob"",
            ""Subordinates"": {
                ""$id"": ""3"",
                ""$values"": [
                    { 
                        ""$ref"": ""1"" 
                    }
                ]
            }            
        }
    }";

    var options = new JsonSerializerSettings
    {
        MetadataPropertyHanding = MetadataPropertyHandling.Default //Json.Net reads metadata by default, just setting the option for ilustrative purposes.
    };

    Employee angela = JsonConvert.DeserializeObject<Employee>(json, settings);
    Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}

Notes:

1. MaxDepth validation will not be affected by ReferenceHandling.Ignore or ReferenceHandling.Preserve.
2. We are merging the Json.Net types ReferenceLoopHandling and PreserveReferencesHandling (we are also not including the granularity on this one) into one single enum; ReferenceHandling.
3. While Immutable types and System.Arrays can be Serialized with Preserve semantics, they will not be supported when trying to Deserialize them as a reference.
4. Value types, such as structs, will not be supported when Deserializing as well.
5. Additional features, such as Converter support, ReferenceResolver, JsonPropertyAttribute.IsReference and JsonPropertyAttribute.ReferenceLoopHandling, that build on top of ReferenceLoopHandling and PreserveReferencesHandling were considered but they will not be included in this first effort.
6. We are still looking for evidence that backs up supporting ReferenceHandling.Ignore.

Issues related:

• https://github.com/dotnet/corefx/issues/38579
• https://github.com/dotnet/corefx/issues/37786
• https://github.com/dotnet/corefx/issues/40045
• https://github.com/dotnet/corefx/issues/41139
• https://github.com/dotnet/corefx/issues/41288
• https://github.com/aspnet/AspNetCore/issues/14497

[Jozkee]

@ahsonkhan @JamesNK thoughts?

[JamesNK]

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

{
  "$id": "1",
  "Name": "Angela",
  "Subordinates": {
    "$id": "2",
    "$values": [
      {
        "$id": "3",
        "Name": "Bob",
        "Manager": {
            "$ref": "1"
        }
      }
    ]
  }
}

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

[JsonReferenceHandling(ReferenceHandling.Preserve)]

// to

[JsonProperty(ReferenceHandling = ReferenceHandling.Preserve)]

[Jozkee]

Consider changing "values" in the array metadata object to "$values" to indicate it isn't a real property.

Agree.

Instead of a dedicated property, what about adding ReferenceHandling as a property to JsonPropertyAttribute (is that the right name? I don't recall off the top of my head).

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

[ahsonkhan]

JsonPropertyAttribute does not exist in System.Text.Json as far as I know.

Correct, we have JsonPropertyNameAttribute
https://github.com/dotnet/corefx/blob/b41b09eadd5eb3f5575845a982f2c177c37f7ce7/src/System.Text.Json/ref/System.Text.Json.cs#L748-L753

cc @steveharter

[scalablecory]

Is this proposal compatible with Json.Net?

[scalablecory]

ReferenceHandling: byte

Why byte?

[JamesNK]

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Is there a way to override equality? Newtonsoft.Json has https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializerSettings_EqualityComparer.htm

[JamesNK]

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

What happens if $id isn't the first property?

[Jozkee]

Is this proposal compatible with Json.Net?

Yes for the metadata semantics that I am trying to follow; no for a feature parity, however we are not tied to follow the first one, just come up with what is already there since I have not found value in going on a different direction.

Why byte?

I was following existing enums in the namespace that also inherits from byte, thing that is made to reduce their size in the Utf8JsonReader struct. but since the Serializer is a class, you are right, there is no much value in changing the enum's default value, I will fix that. Thanks.

[Jozkee]

What determines if objects are equal? object.ReferenceEquals or object.Equals? Newtonsoft.Json uses object.Equals.

Currently the implementation relies on default equality of Hashset/Dictionary, which uses the Object.GetHashCode() and Object.Equals(Object obj).
If someone wants to provide its own equals mechanism, he can override these methods and/or implement IEquatable on its class.

What happens if there are additional properties in a JSON reference object? (has $ref) What about if $ref isn't the first property in an object?

Json.Net throws when finds that there are properties other than $ref in a JSON reference object, I was planning on sticking to that behavior.
I see no point in allowing other properties to co-exist with a $ref, unless the user relies on the $ref property for another purpose; for that maybe in a future we could add a mechanism to disable this functionality similar to MetadataPropertyHandling.Ignore

[steveharter]

@JamesNK on "Ignore" - is this mode necessary and what scenarios would prefer that over "Preserve"?

"Ignore" has potnential inconsistentcy w.r.t. what is serialized vs. what is ignored which would make me question using it in certain scenarios including:

• Serialization order currently depends on reflection order which is not deterministic, so if property A and B both serialize the same reference, sometimes A will be serialized and sometimes B will be serialized.
• The root of the graph may change causing a given type to sometimes serialize a reference, and other times not.

In many scenarios the [JsonIgnore] could be used instead to prevent cycles which has the advantage of being deterministic.

However if:

• "Throws" isn't desired
• and "Preserve" isn't desired because the metadata ($id $ref) or other reasons (performance?)
• and [JsonIgnore] can't be used (perhaps attributes can't be used)
• and potential non-deterministic behavior is OK

then "Ignore" makes sense.

[terrajobst]

Video

We seem to lean towards not having Ignore -- it results in payloads that nobody can reason about. The right fix for those scenarios to modify the C# types to exclude back pointers from serialization.

We'd like to see more detailed spec for the behavior, specifically error cases.

[JamesNK]

Ok, I don't feel strongly that ignore is necessary. And it should never be the default setting so you can always add it in the future if desired.

[paulovila]

Hi @Jozkee , is there a way to try PreserveReferencesHandling, like in a nuget preview?

[Jozkee]

@paulovila this feature is still a work in progress, once is finished and merged into our master branch you can try it out by installing the latest bits of Net core.

[los93sol]

This is extremely disappointing, it was called out by numerous people while 3.0 was in preview as a complete blocker for a lot of people to use the new serializer and benefit from it. It’s even more disappointing this still isn’t being prioritized properly. It seems like by the time this gets addressed people will have just abandoned the new serializer altogether and likely won’t return to it.

[ahsonkhan]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

[los93sol]

IMO this is a major breaking change and it actually roadblocks a lot of people from following the recommended serializer setup from the docs. I’m frustrated because despite this being called out in preview there’s no plan to address in 3.0 or even 3.1 that I’ve seen. Waiting for 5.0 is way too far out on the roadmap and why I said it still isn’t being prioritized properly.

[mythrz]

Edit: Deleted the other comments. Are meaningless!! I was following the wrong crumbles...
In my case, I had an additional Method (garbage/test method...) inside the class that I wanted to serialize... mb, the essential is working.

[doogdeb]

I also experience an issue when trying to serialize the following.

 public class CacheModel
 {
    public Type DtoModelType { get; set; }

    public string CacheKey { get; set; }
  }

DtoModelType is never of type CacheModel but I still get an "JsonException: A possible object cycle was detected which is not supported" error.

[doogdeb]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

[mythrz]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

[doogdeb]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

I understand that, but you mention that this change is earmarked for 5.0, which I'm assuming is .Net 5.0, which is a year away.

[ahsonkhan]

I also experience an issue when trying to serialize the following.

 public class CacheModel
 {
    public Type DtoModelType { get; set; }

    public string CacheKey { get; set; }
  }

DtoModelType is never of type CacheModel but I still get an "JsonException: A possible object cycle was detected which is not supported" error.

The issue you are seeing is specifically about System.Type support and unrelated to cycles/referenceloophandling (marking that comment and this one as off-topic).

I have filed an issue for this with a potential workaround using converters: https://github.com/dotnet/corefx/issues/42712
Let's move the discussion there.

[nickcoad]

It’s even more disappointing this still isn’t being prioritized properly.

What gave you that impression? We are actively working on adding this feature for 5.0.

Isn't 5.0 a year away?

No. We are currently in 3.0/3.1! Next year, around this time, .NET 5 will be released. They are gonna skip a number (the number 4) so we don't get confused with .NET Framework 4.*

Next year, there will be one .NET to "rule us all"

"Next year, around this time" is just another way of saying "a year away"...

[ahsonkhan]

Isn't 5.0 a year away?

I understand that, but you mention that this change is earmarked for 5.0, which I'm assuming is .Net 5.0, which is a year away.

"Next year, around this time" is just another way of saying "a year away"...

Yes, this feature is for 5.0 (i.e. master), which is planned to ship near end of next year (hence the milestone). 3.1 is already done. That said, S.T.Json also ships as a NuGet package so once the feature is done, you could consider using a preview package on 3.1 before 5.0 officially ships.

[paulovila]

It's complicated to depend on the newtonsoft serialisation patch because It is not being updated at the same pace as the SignalR patch is, and that causes that we can’t upgrade to the intermediate versions that you provide. 🤷‍♂️

[ahsonkhan]

@paulovila - I am sorry, but I don't understand what you mean. What target framework are you using in your project?

• What do you mean by "the newtonsoft serialisation patch"? Why is the versioning/shipping of Newtonsoft affect your use of System.Text.Json?
• What do you mean by "SignalR patch"? SignalR generally ships at the same time as .net core, asp.net core, as far as I know.

and that causes that we can’t upgrade to the intermediate versions that you provide.

Let's say you have a .NET Core/Asp.Net Core 3.1 application. The built-in System.Text.Json library that shipped with 3.1 doesn't contain this reference handling feature. However, if you are fine with referencing preview packages, you can add a PackageReference to the latest 5.0 nightly build of the System.Text.Json NuGet package and get that feature (once its implemented/merged into master).

[paulovila]

@ahsonkhan the aforementioned patches are:

<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="3.0.0" />
<PackageReference Include="Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson" Version="3.0.0" />

with that you can circumvent the circular references as follows:

using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
...
services.AddMvc()
    .AddNewtonsoftJson(options =>
    {
        options.SerializerSettings.NullValueHandling = NullValueHandling.Ignore;
        options.SerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
    })
    .SetCompatibilityVersion(CompatibilityVersion.Version_3_0);

services.AddSignalR().AddNewtonsoftJsonProtocol(options =>
{
    options.PayloadSerializerSettings.NullValueHandling = NullValueHandling.Ignore;
    options.PayloadSerializerSettings.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
});

Is this patch strategy the right way to go? I can see now that there has been some alignment in the nuget preview versions since I tried version 3.0.0.0.

[Allann]

Also affected by this issue. Waiting for a year to be able to use EF in an API/Web app is not an option, will be in production then and a complete change in serialization methods at that stage would carry too much risk. Therefore the only option is to user NewtonSoft and not the System.Text.Json.

I can't fin how to circumvent the CR as shown above in an Api controller since it doesn't use AddMvc anymore.

Is there any other options? Maybe the way navigation properties are declared? Time for some additional testing.

[paulovila]

@Allann maybe take a look at this https://github.com/paulovila/UnoPlatformBravent/blob/master/src/UnoWebApiSwagger.WebApi/Startup.cs

[Allann]

@paulovila thanks for the link, but try upgrading that to 3.1 and see how you go ;) AddMvc is not part of the recommended templated code for a WebAPI anymore. Instead they recommend AddControllers.
For example the ConfigureServices is this:

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddDbContext<CountryContext>(opt =>
               opt.UseInMemoryDatabase("Countries"));
            services.AddControllers();
        }

This works fine and produces an very fast and responsive RESTful service. I have managed to get the required request working by changing the navigation properties, and the way in which I created the resultset. Removing the Include was the first step, but a pretty crappy work-around, as I now need 3 different models to provide the same functionality, just because Include get's "fixed" when I don't want it in the first place.

[paulovila]

This is breaking the continuity of netcore 3.0 to 3.1. In order to work 3.1, I think you guys should fix the "Microsoft.AspNetCore.Mvc.NewtonsoftJson" and "Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson" nuget patches for the time being

[BrennanConroy]

should fix the "Microsoft.AspNetCore.Mvc.NewtonsoftJson" and "Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson"

What needs to be fixed with these packages? If you have found a problem in those packages please file an issue at https://github.com/aspnet/AspNetCore/issues/new/choose

[paulovila]

@Allann, @BrennanConroy I can see now that patches are working with NetCore 3.1, I think there was some intermediate version in Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson that had a refence colliding Microsoft.AspNetCore.Mvc.NewtonsoftJson, so no need to report anything :)

[Arash-Sabet]

We had to change the code below:

var domainObject = payload.GetMyObject();
parentObject.DomainObject = domainObject;
... // ommitted code here
DoSomething(parentObject.DomainObject);
updatedPayload = JsonSerializer.Serialize(parentObject.DomainObject);

to the following one to avoid circular reference exception:

var domainObject = payload.GetMyObject();
... // ommitted code here
DoSomething(parentObject.DomainObject);
updatedPayload = JsonSerializer.Serialize(domainObject);

Is it related to the case that @Jozkee is suggesting to fix/improve? Not sure why the 2nd code snippet does not throw that exception while both domainObject and parentObject.DomainObject are pointing to the same object?

[Jozkee]

@Arash-Sabet I can see how that can cause an infinite loop if domainObject contains a reference to parentObject somewhere.

On your 2nd snippet you are not referencing domainObject in your parentObject.DomainObject so that might be why it does not throw in that case.

[Arash-Sabet]

@Jozkee For sure domainObject has no reference to its parentObject at all.

[ahsonkhan]

For sure domainObject has no reference to its parentObject at all.

Can you share what the type and POCO structure of domainObject and parentObject looks like? A simplified repro would also help to address your concern.

Not sure why the 2nd code snippet does not throw that exception while both domainObject and parentObject.DomainObject are pointing to the same object?

I can't tell if that's true just from the snippet provided. How do you know that domainObject and parentObject.DomainObject are pointing to the same object?

[etghanibahman]

Do you have any advice on how we can serialize DataSet with current version of System.Text.Json.JsonSerializer ?

[drdamour]

man, this thing is not ready for very simple use cases....

[Jozkee]

@drdamour Thank you for giving it a try.
Regardless of https://github.com/dotnet/runtime/pull/655 being merged, it is still a WIP.
Can you specify the cases that are causing you issues so we can fix it?

[drdamour]

simple double relationship from db office <=> employee getting either an including the other circles back to the other now on deserialization. can't mark them as do not serialize because i need to be able to get from either way.

[Jozkee]

@drdamour You mean Office and Employee enitites where Office contains a List<Employee>? and each Employee contains an Office property?

Can you provide your model classes in a comment?

[Jozkee]

Forgot to add the comment here.

The preview version for ReferenceHandling feature is here.

• Download the System.Text.Json preview package version 5.0.0-alpha.1.20071.1 from dotnet5 gallery (through Visual Studio's NuGet client):
  https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json

• How to use it:
  Set ReferenceHandling.Preserve to ReferenceHandling property in JsonSerializerOptions.

var options = new JsonSerializerOptions
{
    ReferenceHandling = ReferenceHandling.Preserve
};

string json = JsonSerializer.Serialize(objectWithLoops, options);

Here's the spec doc including samples and notes about many scenarios.
If you want to add feedback, feel free to leave a comment.

[Jozkee]

@Dzivo No, this is going to be shipped on 5.0; right now it is only available on the preview package version of System.Text.Json.

[Dzivo]

@Jozkee thx for the info switching back to NewtonsoftJson until then :)

[PillowMetal]

• Download the System.Text.Json preview package version 5.0.0-alpha.1.20071.1 from dotnet5 gallery (through Visual Studio's NuGet client):
  https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json
• How to use it:
  Set ReferenceHandling.Preserve to ReferenceHandling property in JsonSerializerOptions.

var options = new JsonSerializerOptions
{
    ReferenceHandling = ReferenceHandling.Preserve
};

string json = JsonSerializer.Serialize(objectWithLoops, options);

@Jozkee Verified this works as specified when making Serialize and Deserialize calls, but does not work when trying to apply this property globally in startup.cs with:

services.AddControllersWithViews().AddJsonOptions(options =>
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve);

Will this also eventually be added in .NET 5.0?

[BrennanConroy]

What version of the .NET Runtime are you using with your web app?

cc @pranavkm

Will this also eventually be added in .NET 5.0?

It should already be in 5.0.

[PillowMetal]

@BrennanConroy I am using the following in my project file:

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <RuntimeFrameworkVersion>3.1.2</RuntimeFrameworkVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="System.Text.Json" Version="5.0.0-preview.2.20153.7" />
  </ItemGroup>

... and if I specifically use the options parameter in the Serialize and Deserialize methods, everything works great. But whether I use no parameters at all OR use the AddJsonOptions method, I get the following exception:

JsonException: A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 0. Consider using ReferenceHanlding.Preserve on JsonSerializerOptions to support cycles.

dotnet --info provides the following:

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Filesdotnet\sdk\3.1.102\

Host (useful for support):
Version: 3.1.2
Commit: 916b5cba26

[pranavkm]

MVC has a copy constructor for JsonSerializerOptions that only knows about properties that compiled against: https://github.com/dotnet/aspnetcore/blob/master/src/Mvc/Mvc.Core/src/Infrastructure/JsonSerializerOptionsCopyConstructor.cs#L12-L33. The copy constructor is to allow for the output formatter (serialization) to use relaxed encoding: https://github.com/dotnet/aspnetcore/blob/master/src/Mvc/Mvc.Core/src/Formatters/SystemTextJsonOutputFormatter.cs#L40-L44. There is an issue tracking add a copy ctor to JsonSerializerOptions that's currently sitting in 5.0: https://github.com/dotnet/runtime/issues/30445

[terrajobst]

@Jozkee should this be marked as api-ready-for-review?

[Jozkee]

@terrajobst I just updated the issue description to reflect the suggestions form our last API review, so I think it should be ready.

Some folks were pushing against ReferenceHandling being a class and suggested that it would rather be an enum; we discussed that the reasoning for this being a class was to guard for the future in case we extend the ReferenceHandling options to a max of full parity with Newtonsoft.Json.

@bartonjs @KrzysztofCwalina

https://youtu.be/H9zrbztep4M?t=5890

Switching back to enum API since no one has provided evidence or requested further customization of the current provided options.

[BreyerW]

@Jozkee I have use for custom reference handling: preserving references across several serializations. I wrote 3d editor where i dont serialize everything in one go but instead serialize one by one, reducing maximum memory pressure. However for it to work even in the face of very complex reference types i have to store string ids somewhere and cleanest by far way for that is to write custom reference handling.

I can provide rudimentary snippet if desired

[Jozkee]

@BreyerW As far as I know, there are no features in the JsonSerializer that keep state across multiple calls and this one is not the exception.

I can provide rudimentary snippet if desired

Yes please, I am curious of your implementation and what's your exact scenario.

[BreyerW]

Edit 2: I might have jumped to conclusion sorry. It seems the change might not impede my scenario since its different class than i thought. Still i will leave the comment as-is as it may prove useful anyway for different reasons

@Jozkee yes naturally json.net doesnt have it by default and thats why i had to write custom reference handling. My scenario is already explained in previous post and i will edit this post to provide example soon

Edit

public class ThreadsafeReferenceResolver : IReferenceResolver
    {
        private static IDictionary<string, object> stringToReference;

        private static IDictionary<object, string> referenceToString;

        public static ThreadsafeReferenceResolver()
        {

            stringToReference = new Dictionary<string, object>(EqualityComparer<string>.Default);
            referenceToString = new Dictionary<object, string>(EqualityComparer<object>.Default);
        }

        public void AddReference(
            object context,
            string reference,
            object value)
        {
            if (value.GetType().IsValueType) return;
            if (stringToReference.TryGetValue(reference, out _))
                return;
            referenceToString.Add(value, reference);
            stringToReference.Add(reference, value);
        }

        public string GetReference(
            object context,
            object value)
        {
            if (value.GetType().IsValueType) return null;
            if (!referenceToString.TryGetValue(value, out string result))
            {
                result = value.GetInstanceID().ToString();
                AddReference(context, result, value);
            }

            return result;
        }

        public bool IsReferenced(
            object context,
            object value)
        {
            return referenceToString.TryGetValue(value, out _);
        }

        public object ResolveReference(
            object context,
            string reference)
        {
            stringToReference.TryGetValue(reference, out var obj);
            return obj;
        }
    }

Ah and let me add that i use this indirectly to discover state changes (then compute objects delta without holding entire reference structure because every reference is just guid)

Biggest difference compared to default reference handling is using guid instead of simple int/long and not clearing dictionaries when done with single instance of serialization (take note of static on dictionaries they should be cleaned on end of frame instead)

And guid instead of long is one example used by json.net to explain custom IReferenceResolver in other words custom IReferenceResolver would let you design your own unique key which can be valuable

[terrajobst]

Video

It sounds like you need to design an extension point by making it class instead. That sounds reasonable, so we bumped it back to api-needs-work. Once you have a proposal, we can re-review.

[KerosenoDev]

Hi. In ConfigureServices I have added:

services.AddControllers().AddJsonOptions (options => options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve);

But when I run it I keep getting this error:

System.Text.Json.JsonException: A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32. Consider using ReferenceHanlding.Preserve on JsonSerializerOptions to support cycles.

How should I configure it for Web API?

BTW, there seems to be an error in the exception text: _ReferenceHanlding_

[Jozkee]

@KerosenoDev that's a current problem in ASP.NET side which make it unaware of new features in JsonSerializerOptions; for now you can workaround it by passing an Encoder.

services.AddMvc().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
    options.JsonSerializerOptions.Encoder = JavaScriptEncoder.Default;
});

cc @pranavkm

BTW, there seems to be an error in the exception text: ReferenceHanlding

That is already fixed in https://github.com/dotnet/runtime/pull/34700.

[KerosenoDev]

Thanks you @Jozkee, it worked for me.

Now the service does not give problems, but the application that consumes it does, when deserializing.

I've added ReferenceHandling = ReferenceHandling.Preserve and Encoder = JavaScriptEncoder.Default.

But I get the following error: The '$id' and '$ref' metadata properties must be JSON strings. Current token type is 'PropertyName'...

[Jozkee]

@KerosenoDev Can you share your JSON payload and maybe a code snippet of what you are doing?

Can you also share the version of your System.Text.Json package. Are you able to repro the error with the latest prerelease 5.0.0-preview.4.20214.2?

[KerosenoDev]

@Jozkee something like this is what I expected:

{
    "location": "1",
    "warehouses": {
        { "warehouse": "1" },
        { "warehouse": "2" }
     }
}

Something like this is what the server sends:

{
    "$id": "1",
    "$values": [
        {
            "$id": "2",
            "location": "1",
            "warehouses": {
                "$id": "3",
                "$values": [
                    {
                        "$id": "4",
                        "warehouse": "1"
                    },
                    {
                        "$id": "5",
                        "warehouse": "2"
                    }
                ]
            }
        }
    ]
}

Attributes with $ were added with the ReferenceHandling and Encoder configuration.

This code consumes the API:

public async static Task<List<Locations>> GetLocations() {
    List<Locations> locations;

    using (HttpClient webApiClient = new HttpClient()) {
        JsonSerializerOptions options = GetOptions();
        HttpRequestMessage request = CreateRequest(HttpMethod.Get, string.Format("{0}GetLocations", GetBaseUrl()));

        HttpResponseMessage result = await webApiClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
        result.EnsureSuccessStatusCode();

        using (Stream content = await result.Content.ReadAsStreamAsync()) {
            locations = await JsonSerializer.DeserializeAsync<List<Locations>>(content, options);
        }
    }

    return locations;
}

private static JsonSerializerOptions GetOptions() {
    return new JsonSerializerOptions {
        PropertyNameCaseInsensitive = true,
        WriteIndented = true,
        ReferenceHandling = ReferenceHandling.Preserve,
        Encoder = JavaScriptEncoder.Default
    };
}

private static HttpRequestMessage CreateRequest(HttpMethod httpMethod, string url) {
    return new HttpRequestMessage(httpMethod, url);
}

The version of System.Text.Json is 5.0.0-preview.2.20160.6, the most recent that appears to me (NuGet).

[Jozkee]

something like this is what I expected:

@KerosenoDev You were expecting to receive a JSON without $ properties from the server but instead you received the JSON containing $id, $ref and $values. Is that why you need to enable ReferenceHandling?

I tried to repro the error that you are reporting doing a simple console app but no luck... could you please also share your Locations and Warehouses classes? It would be even better if you could share your scenario in a GitHub repository so we can make sure that I am doing the same thing that you do.

Here's what I did.
https://gist.github.com/Jozkee/7b48bd4aadad6bdb3e47f1c244f12326

[KerosenoDev]

@Jozkee

You were expecting to receive a JSON without $ properties from the server but instead you received the JSON containing $id, $ref and $values. Is that why you need to enable ReferenceHandling?

I needed to enable it because I had this error: System.Text.Json.JsonException: A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 32.

Before, in objects without cycles and System.Text.Json 4.7.1, serialization returned JSON without $ properties. Now, ReferenceHandling (and Encoder) solves the problem of objects with cycles, but add $ properties.

But now when deserializing in app, I get the following error, with all objects, have cycles or not: The '$id' and '$ref' metadata properties must be JSON strings.

could you please also share your Locations and Warehouses classes?

I put the entities wrong, it is warehouse who has the list of locations.

It would be even better if you could share your scenario in a GitHub repository

I hope this works for you, the names are in Spanish: https://gist.github.com/KerosenoDev/01c93bc28d96f40bafb0927446e8f6ef

Sorry if I express myself badly in English :sweat_smile:

[PillowMetal]

@KerosenoDev that's a current problem in ASP.NET side which make it unaware of new features in JsonSerializerOptions; for now you can workaround it by passing an Encoder.

services.AddMvc().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
    options.JsonSerializerOptions.Encoder = JavaScriptEncoder.Default;
});

@Jozkee I assume this workaround only works for ASP.NET and not .NET Core? (I tried adding an encoder and it's a no go). I understand the issue was the lack of the copy constructor being aware of the new serializer settings that was being tracked in https://github.com/dotnet/runtime/issues/30445 but that issue is now closed; it appears to have been merged into https://github.com/dotnet/runtime/pull/34725

I have the latest 5.0 preview build package of System.Text.Json but is there another package I can reference that has the new copy constructor functionality being worked on?

EDIT: I should also note that I am using AddControllersWithViews not AddMvc but I assume that is irrelevant when this preserve feature is implemented?

[Jozkee]

I assume this workaround only works for ASP.NET and not .NET Core?

@PillowMetal the workaround is only meant for ASP.NET Core, from previous comments I understand that you are using that, is that correct?

I understand the issue was the lack of the copy constructor being aware of the new serializer settings that was being tracked in #30445 but that issue is now closed

Yes, the copy .ctor is already in .NET 5.0, however it has to be implemented in dotnet/aspnetcore repo.

I have the latest 5.0 preview build package of System.Text.Json but is there another package I can reference that has the new copy constructor functionality being worked on?

As I said, it still needs to be implemented in ASP.NET side.

I should also note that I am using AddControllersWithViews not AddMvc but I assume that is irrelevant when this preserve feature is implemented?

I tried with AddControllersWithViews and seems to work as well. You can provide a sample to compare and see if there is something wrong in my repro.

services.AddControllersWithViews().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
    options.JsonSerializerOptions.Encoder = JavaScriptEncoder.Default;
});

[PillowMetal]

@PillowMetal the workaround is only meant for ASP.NET Core, from previous comments I understand that you are using that, is that correct?

@Jozkee Yes, that is correct, .NET Core 3.1.201 SDK and 3.1.3 runtime

Yes, the copy .ctor is already in .NET 5.0, however it has to be implemented in dotnet/aspnetcore repo.

As I said, it still needs to be implemented in ASP.NET side.

Do you know of an issue# I can follow to see it's progress or is that going to be part of https://github.com/dotnet/runtime/pull/34725?

I tried with AddControllersWithViews and seems to work as well. You can provide a sample to compare and see if there is something wrong in my repro.

Starting with what works, I have this in my .csproj file:

<PackageReference Include="System.Text.Json" Version="5.0.0-preview.4.20216.4" />

These two extension methods defined:

public static void Set<T>(this ISession session, string key, T value) =>
    session.SetString(key, JsonSerializer.Serialize(value, new JsonSerializerOptions
    { ReferenceHandling = ReferenceHandling.Preserve }));

public static T Get<T>(this ISession session, string key) =>
    session.GetString(key) == null 
        ? default
        : JsonSerializer.Deserialize<T>(session.GetString(key), new JsonSerializerOptions
            { ReferenceHandling = ReferenceHandling.Preserve });

I call my extension methods in the following manner:

HttpContext.Session.Set("LEA", await _repository.GetLeaAsync(districtId, schoolId, schoolNbr));
BeSchool lea = HttpContext.Session.Get<BeSchool>("LEA");

All that is needed to be known here is that a single EF model object (BeSchool) is being retrieved from a repository (DB context), serialized into a session string, and then deserialized from the session string back into it's model.

The record retrieved DOES have circular dependencies and everything works fine; I inspect the "lea" variable and all is good.

So what I want to do is remove the redundant serializer options handling references in the extension methods, and just use a global setting for the application.

So I add the following code in my Startup.cs and everything still runs fine:

_ = services.AddControllersWithViews().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
    options.JsonSerializerOptions.Encoder = JavaScriptEncoder.Default;
});

Now the problem.... I remove the serializer option arguments from my extension methods that call Serialize and Deserialize (compare to above):

public static void Set<T>(this ISession session, string key, T value) =>
    session.SetString(key, JsonSerializer.Serialize(value));

public static T Get<T>(this ISession session, string key) =>
    session.GetString(key) == null 
        ? default
        : JsonSerializer.Deserialize<T>(session.GetString(key));

... and expect my extension methods called via my session calls to use the new global serializer options set in the Startup.cs file, but instead, I receive the ole familiar exception:

JsonException: A possible object cycle was detected. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 64. Consider using ReferenceHandling.Preserve on JsonSerializerOptions to support cycles.
System.Text.Json.ThrowHelper.ThrowJsonException_SerializerCycleDetected(int maxDepth)

If this is due to the copy constructor issue, I understand and will wait for that to be implemented on the ASP.NET side, but I just thought this workaround you mentioned would solve this problem as well.

[pranavkm]

.. and expect my extension methods called via my session calls to use the new global serializer options set in the Startup.cs file,

@PillowMetal AddJsonOptions does not configure the serializer options globally and there isn't an API available to configure these options globally. You'll have to continue passing in the options at the locations you intend to call JsonSerilaizer.Serialize \ JsonSerializer.Deserialize.

[PillowMetal]

@pranavkm ok, thanks. I'll continue passing in options at the calls. So what is the purpose of the serializer options being set in the Startup file then?

My first guess would be these are used to set options for the Controller.Json method and possibly uses of new JsonResult in controllers?

[KerosenoDev]

@Jozkee was the code useful?

[Jozkee]

@KerosenoDev sorry for have kept you waiting.

I took a look at the snippets that you provided but since your code was still depending on a Db connection, I had to use mock data. I tried returning two simple Location instances that share a reference to a Warehouse and then GETting that from a console app using your snippet that does an HTTP request, but no luck either. Btw, I can read Spanish quite well so no problem understanding the code :)

Anyway, I was finally able to repro the issue that you are facing on a different way, that is indeed a bug on the case where you use ReferenceHandling.Preserve and DeserializeAsync.

public static async Task ReadMoreDataOnMetadataValueAsync()
{
    // 1..7
    string json = "{\"$id\":";

    // 8..17 - Problem happpens when we step in this portion of the JSON payload.
    json += "\"90123456\"";

    // 18..32
    json += ",\"foo\":\"value\"}";

    var stream = new MemoryStream(Encoding.UTF8.GetBytes(json));

    var opts = new JsonSerializerOptions
    {
        DefaultBufferSize = 16,
        ReferenceHandling = ReferenceHandling.Preserve
    };

    var foo = await JsonSerializer.DeserializeAsync<Foo>(stream, opts);
}

The reason why this happens is that when we step into the token "90123456" we exceed the deserializer's buffer size (16), so we have to unwind the call stack in order to read more data from the stream. When we step back into the code that should read the string token ("90123456"), we call reader.Read again and hence the reader moves to the next token which is a property ("foo"). That should not happen when stepping back where we left since we didn't complete processing the previous token in the first place.

@KerosenoDev One way to workaround this would be to increase the size of your initial buffer size by specifying a larger DefaultBufferSize in your options, but that workaround will be also flawed for large payloads.

A better way to workaround this for now is to deserialize it using one of the sync APIs in JsonSerializer but that will make you buffer the entire JSON payload into memory.

I will file an issue for that and will try to get that fixed. Thanks for bringing this up.

[KerosenoDev]

Thanks @Jozkee, specifying a larger DefaultBufferSize has worked for me.

I will file an issue for that and will try to get that fixed. Thanks for bringing this up.

I will be aware!

[ahsonkhan]

specifying a larger DefaultBufferSize has worked for me.

Did you set the DefaultBufferSize to some value when you saw the issue (and if so, what value did you pick) and what value did you change it to now so that it worked? In the gist you shared, I didn't see any place where you were overriding the DefaultBufferSize, which means the JsonSerializer was using the predefined size of 16 KB.

I would imagine that will be a flaky workaround since a different JSON payload, where the string value of $id crosses the boundary between the chunks , would still result in the issue, unless you set the DefaultBufferSize to be at least as big as the largest JSON payload size you expect. I recommend waiting for the bug to be fixed OR, as @Jozkee mentioned, use the synchronous APIs.

[KerosenoDev]

@ahsonkhan as you said, I specified the DefaultBufferSize larger than the JSON payload.

I recommend waiting for the bug to be fixed OR, as @Jozkee mentioned, use the synchronous APIs.

Yes, it is provisional. I am waiting for the bug fix. About synchronous APIs, I don't know if I have understood it well.

But I have tried to deserialize like this, without DefaultBufferSize, changing DeserializeAsync by Deserialize and ReadAsStreamAsync by ReadAsStringAsync:

string json = await resultado.Content.ReadAsStringAsync(); ubicaciones = JsonSerializer.Deserialize<List<Ubicaciones>>(json, opciones);
And it has also worked.

EDIT: Throws an System.OutOfMemoryException with a large data load.

[praiakov]

@KerosenoDev that's a current problem in ASP.NET side which make it unaware of new features in JsonSerializerOptions; for now you can workaround it by passing an Encoder.

services.AddMvc().AddJsonOptions(options =>
{
    options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
    options.JsonSerializerOptions.Encoder = JavaScriptEncoder.Default;
});

cc @pranavkm

BTW, there seems to be an error in the exception text: ReferenceHanlding

That is already fixed in #34700.

And to API?

services.AddControllers().AddJsonOptions(options =>
{
   options.JsonSerializerOptions.ReferenceHandling = ReferenceHandling.Preserve;
});

[Jozkee]

ReferenceResolver

As of now, when we opt-in to ReferenceHandling.Preserve, the ReferenceResolver that serves as a bag of objects references on serialization and deserialization is an internal type and there is no way to specify how I want to store my references nor to specify what's the format of the ID used on $id and $ref metadata.

This proposal shows how we can expose ReferenceResolver and how to use it.
This proposal also changes ReferenceHandling to ReferenceHandler, the name ReferenceHandling incorrectly indicates that the type is an enum.

Exposing the ReferenceResolver members would also be beneficial when https://github.com/dotnet/runtime/issues/1562 (Add JsonConverter signatures that can receive serialization state) is completed since it will allow users to interact with the resolver, either the default or a custom one.

Proposal

public partial class JsonSerializerOptions
{
    public ReferenceHandler ReferenceHandler { get; set; }
}

public sealed partial class ReferenceHandler
{
    public static ReferenceHandler Default { get; }
    public static ReferenceHandler Preserve { get; }

    /// <summary>
    /// Initializes a new instance of the <see cref="ReferenceHandler"/> class.
    /// When this constructor is used, the instance will implicitly specify the use of preserve semantics.
    /// </summary>
    /// <param name="resolverProvider">A function that returns the ReferenceResolver instance used for each serialization call.</param>
    public ReferenceHandler(Func<ReferenceResolver> resolverProvider) { }
}

/// <summary>
/// This class defines how the <see cref="JsonSerializer"/> deals with references on serialization and deserialization.
/// Defines the core behavior of preserving references on serialization and deserialization.
/// </summary>
public abstract class ReferenceResolver
{
    /// <summary>
    /// Adds an entry to the bag of references using the specified id and value.
    /// This method gets called when an $id metadata property from a JSON object is read.
    /// </summary>
    /// <param name="referenceId">The identifier of the respective JSON object or array.</param>
    /// <param name="value">The value of the respective CLR reference type object that results from parsing the JSON object.</param>
    public abstract void AddReference(string referenceId, object value);

    /// <summary>
    /// Gets the reference identifier of the specified value if exists; otherwise a new id is assigned.
    /// This method gets called before a CLR object is written in order to get the object's reference id.
    /// </summary>
    /// <param name="value">The value of the CLR reference type object to get an id for.</param>
    /// <returns>The reference id realated to the specified value.</returns>
    public abstract string GetReference(object value);

    /// <summary>
    /// Indicates wether the specified value has already been written and hence can be referenced by its reference id.
    /// This method gets called before a CLR object is written so we can decide whether to write $id and enumerate the rest of its properties or $ref and step into the next object graph.
    /// </summary>
    /// <param name="value">The value of the CLR reference type object to get an id for.</param>
    /// <returns>`true` if a reference to value exists; otherwise, `false`.</returns>
    public abstract bool IsReferenced(object value);

    /// <summary>
    /// Returns the CLR reference type object related to the specified reference id.
    /// This method gets called when $ref metadata property is read.
    /// </summary>
    /// <param name="referenceId">The reference id related to the returned object.</param>
    /// <returns>The reference type object related to specified reference id.</returns>
    public abstract object ResolveReference(string referenceId);
}

Prior Art

From Newtonsoft.Json:

/// <summary>
/// Used to resolve references when serializing and deserializing JSON by the <see cref="JsonSerializer"/>.
/// </summary>
public interface IReferenceResolver
{
    /// <summary>
    /// Resolves a reference to its object.
    /// </summary>
    /// <param name="context">The serialization context.</param>
    /// <param name="reference">The reference to resolve.</param>
    /// <returns>The object that was resolved from the reference.</returns>
    object ResolveReference(object context, string reference);

    /// <summary>
    /// Gets the reference for the specified object.
    /// </summary>
    /// <param name="context">The serialization context.</param>
    /// <param name="value">The object to get a reference for.</param>
    /// <returns>The reference to the object.</returns>
    string GetReference(object context, object value);

    /// <summary>
    /// Determines whether the specified object is referenced.
    /// </summary>
    /// <param name="context">The serialization context.</param>
    /// <param name="value">The object to test for a reference.</param>
    /// <returns>
    ///     <c>true</c> if the specified object is referenced; otherwise, <c>false</c>.
    /// </returns>
    bool IsReferenced(object context, object value);

    /// <summary>
    /// Adds a reference to the specified object.
    /// </summary>
    /// <param name="context">The serialization context.</param>
    /// <param name="reference">The reference.</param>
    /// <param name="value">The object to reference.</param>
    void AddReference(object context, string reference, object value);
}

NOTE: System.Text.Json ReferenceResolver considered dropping the method IsReferenced and use GetReference(object value, out bool alreadyExists) instead in order avoid one dictionary lookup.
But there could be cases where someone wants to know if the object is already referenced without accidentally adding it to the bag of refereces.
So we decided to stick with the original Newtonsoft design since the performance-functionallity trade off is not bad and it would also make it easier to port existing an IReferenceResolver from Newtonsoft.

See prototype

```cs
///

/// This class defines how the deals with references on serialization and deserialization.
/// Defines the core behavior of preserving references on serialization and deserialization.
///

public abstract class ReferenceResolver
{
///

/// Adds an entry to the bag of references using the specified id and value.
/// This method gets called when an $id metadata property from a JSON object is read.
///

/// The identifier of the respective JSON object or array.
/// The value of the respective CLR reference type object that results from parsing the JSON object.
public abstract void AddReference(string referenceId, object value);

/// <summary>
/// Gets the reference identifier of the specified value if exists; otherwise a new id is assigned.
/// This method gets called before a CLR object is written. Based on `alreadyExists` we can decide whether to write $id and enumerate the object properties or write $ref and step into the next object graph.
/// </summary>
/// <param name="value">The value of the CLR reference type object to get an id for.</param>
/// <param name="alreadyExists">`true` if a reference to value already exists; otherwise, `false`.</param>
/// <returns>The reference id related to the value.</returns>
public abstract string GetReference(object value, out bool alreadyExists);

/// <summary>
/// Returns the CLR reference type object related to the specified reference id.
/// This method gets called when $ref metadata property is read.
/// </summary>
/// <param name="referenceId">The reference id related to the returned object.</param>
/// <returns>The reference type object related to specified reference id.</returns>
public abstract object ResolveReference(string referenceId);

}
```

Sample usage

Basic use case, someone wants to write/read its own identifier, in this case a Guid:

public class GuidReferenceResolver : ReferenceResolver
{
    private readonly IDictionary<Guid, PersonReference> _people = new Dictionary<Guid, PersonReference>();

    public override object ResolveReference(string referenceId)
    {
        Guid id = new Guid(referenceId);

        PersonReference p;
        _people.TryGetValue(id, out p);

        return p;
    }

    public override string GetReference(object value)
    {
        PersonReference p = (PersonReference)value;
        _people[p.Id] = p;

        return p.Id.ToString();
    }

    public override bool IsReferenced(object value)
    {
        PersonReference p = (PersonReference)value;

        return _people.ContainsKey(p.Id);
    }

    public override void AddReference(string reference, object value)
    {
        Guid id = new Guid(reference);
        _people[id] = (PersonReference)value;
    }
}

public static void DeserializeWithGuidResolver()
{
    string json = 
@"[
  {
    ""$id"": ""0b64ffdf-d155-44ad-9689-58d9adb137f3"",
    ""Name"": ""John Smith"",
    ""Spouse"": {
      ""$id"": ""ae3c399c-058d-431d-91b0-a36c266441b9"",
      ""Name"": ""Jane Smith"",
      ""Spouse"": {
        ""$ref"": ""0b64ffdf-d155-44ad-9689-58d9adb137f3""
      }
    }
  },
  {
    ""$ref"": ""ae3c399c-058d-431d-91b0-a36c266441b9""
  }
]";
    var options = new JsonSerializerOptions
    {
        ReferenceHandler = new ReferenceHandler(() => new GuidReferenceResolver())
    };
    List<PersonReference> people = JsonSerializer.Deserialize<List<PersonReference>>(json, options);
}

public class PersonReference
{
    internal Guid Id { get; set; }
    public string Name { get; set; }
    public PersonReference Spouse { get; set; }
}

Use case of someone that may want to use a persistent bag of references, as stated in https://github.com/dotnet/runtime/issues/30820#issuecomment-602923448:

private static readonly _resolver = new MyCustomReferenceResolver();

// First call: json = 
// @"[
//   {
//     ""$id"": ""1"",
//     ""Name"": ""John Smith"",
//     ""Spouse"": {
//       ""$id"": ""2"",
//       ""Name"": ""Jane Smith"",
//       ""Spouse"": {
//         ""$ref"": ""1""
//       }
//     }
//   },
//   {
//     ""$ref"": ""ae3c399c-058d-431d-91b0-a36c266441b9""
//   }
// ]";

// Second call: json = 
// @"[
//   {
//     ""$ref"": ""1""
//   },
//   {
//     ""$ref"": ""2""
//   }
// ]";
public static List<PersonReference> DeserializePeople(string json)
{
    var options = new JsonSerializerOptions
    {
        ReferenceHandler = new MyCustomReferenceResolver(() => _resolver)
    };

    List<PersonReference> people = JsonSerializer.Deserialize<List<PersonReference>>(json, options);
    return people;
}

NOTE: This behavior may be a pit of failure since it roots object references and produces outbound memory growth.
One option to make sure that the user is aware about this intent is to add a new analyzer rule that warns when the returning object in the Func<ReferenceResolver> does not call the ReferenceResolver constructor.

Notes

• We are using an abstract class instead of an interface, one reason for that is that there is no strong evidence that shows the need to implement this along with other interfaces, however, there are users of Newtonsoft.Json that does implement this along with some other interface.
  
  
  
  • https://github.com/jaskiew/jNet.RPCDemo/blob/918ab48e56618d609d824135285654418f3611ea/submodules/jNet.RPC/jNet.RPC/Client/ClientReferenceResolver.cs
  
  
  • https://github.com/xyrus02/anno-companion/blob/297e855f5b3af19dfd4988d3aef74d153f014f9c/src/lib/XyrusWorx.Gaming.AnnoCompanion.Common/Serialization/JsonReferenceResolver.cs
  
  
• We may need to add an analyzer for the ReferenceResolver ctor that warns you when you are not returning a new instance.
• Naming change (ReferenceHandling -> ReferenceHandler) will be a "breaking change" since this already ships in .NET 5 preview.

[BreyerW]

@Jozkee thank you for taking my feedback into account

Since changes from json.net are minimal i have no further feedback on it and looks good to me

[BreyerW]

@Jozkee i just thought about abstract vs interface and one thing occured to me: if im not mistaken System.Json.Text during serialization store little on heap (i mean little in fields)? If thats the case for reference resolver then maybe it could be abused to avoid allocation through generic constrain and constrained call

Naturally this apply only if it isnt stored in field and is possible to use constraint otherwise please disregard this digression

[terrajobst]

Video

• The problem with taking a func is that people might accidentally close over a local variable that now becomes a static. Since the resolver is supposed to hold a dictionary this might result in two errors: (1) concurrency issues when multiple threads serialize and (2) a memory leak because the dictionary gets never eligible for GC

C# namespace System.Text.Json.Serialization { public partial class JsonSerializerOptions { public ReferenceHandler ReferenceHandler { get; set; } } public abstract class ReferenceHandler { public static ReferenceHandler Default { get; } public static ReferenceHandler Preserve { get; } public abstract ReferenceResolver CreateResolver(); } public sealed partial class ReferenceHandler<T> : ReferenceHandler where T: ReferenceResolver, new() { public override ReferenceResolver CreateResolver() => new T(); } public abstract class ReferenceResolver { public abstract void AddReference(string referenceId, object value); public abstract string GetReference(object value, out bool alreadyExists); public abstract object ResolveReference(string referenceId); } }

[butler1233]

Regarding ReferenceHandling.Ignore....

I don't see why this wouldn't be useful. Sure, on deserialisation it could throw up some odd results, but I think for serialisation it's actually quitehelpful. The example use case in the very first comment is a perfect example of why it would be useful. I think it would be ideal to have something like this, because attributes like [JsonIgnore] don't really help, as that would just stop that property from serialising ever.

Currently people are suggesting actively dumping System.text.Json, as can be seen in this question on Stack, where 3 of the 4 answers suggest using Newtonsoft.Json, which has the ignore functionality built in. So people are actively gimping the performance of their application to get a fairly basic feature working. Newtonsoft is slow, but at least it works.

EDIT: For reference, I've been using preview 3, I'm yet to try preview 4 but if preserve achieves essentially the same thing without throwing the cycling exception then that would probably be good.

[diegogarber]

@Jozkee can we PLEASE get the Ignore???
A simple scenario:
Let's say that you have your application that's using newtonsoft.json (as most people do) and you swap to system.text.json. What happens? well, you get loop errors. You swap it to preserve and now your front end does NOT WORK!!!!!

If you supported the Ignore, we can have the same output and keep the front end without having to modify it and THEN maybe change it to reference when we fix the front end.

Thank you!

[cjblomqvist]

I have to agree with many other concerning Ignore - and I'd like to try to add some arguments. Unfortunately video is not optimal for me, so I might have missed good arguments in the videos (but I've read through this issue/comments at least).

My arguments are based on that Preserve is the only good alternative. My reasoning for that: [JsonIgnore] is not feasible whenever you have a two-way relationship and you sometimes need to fetch either type of object (and you want the related objects with it, but not the self-references). Except for that there doesn't seem to be a good workaround?

• Like @diegogarber pointed out, it breaks backwards compatibility and can induce significant additional work (e.g. frontend wise)
• It causes a significantly larger payload. The original issues first two examples (Ignore vs Preserve) is pretty clear (~twice the amount of lines). For Web API scenarios (which might be the most common scenario for serializing JSON), that's pretty horrendous (even if the added stuff is quite short).
• The JSON structure is not obvious from a frontend perspective - which the Ignore structure is (imo). For scenarios with Ignore where you really need the references, you could easily identify that specific call (again, assuming a Web API) and do your ViewModel-layer instead. The Preserve structure on the other hand needs parsing - specifically turning lists (arrays) into collections (objects) will be quite confusing.

To sum up: To break backwards compatibility, cause a significant larger payload, as well as breaking the structure by introducing a middle layer (as in Subordinates not being a list/array of values), should be reason enough to consider having Ignore.

I believe the StackOverflow issue mentioned reflects the community's stand on this - the only reasonable way forward (unless for specific scenarios where you really need the C#/serialization performance) is to continue with Newtonsoft. To me, that should seen as a sign that this is not a reasonable way forward. It's been a long wait between 3.0 -> 5.0 (we're still waiting after all), and to still not having this resolved will definitely continue to stir up frustration.

[Jozkee]

The problem with Ignore is that you end-up losing data, this will probably cause an issue even more severe than having a large payload or having to do extra parsing in the front-end.

e.g:

static void Main()
{
    var angela = new Employee { Name = "Angela" };
    var bob = new Employee { Name = "Bob", Manager = angela };

    angela.Subordinates = new List<Employee> { bob };

    var settings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore };
    string json = JsonConvert.SerializeObject(bob, settings);
    Console.WriteLine(json);
    // {"Name":"Bob","Manager":{"Name":"Angela","Manager":null,"Subordinates":[]},"Subordinates":null}

    Employee bobCopy = JsonConvert.DeserializeObject<Employee>(json);
    Employee angelaCopy = bobCopy.Manager;

    Console.WriteLine("Number of subordinates for angela: " + angela.Subordinates.Count);
    // Number of subordinates for angela: 1
    Console.WriteLine("Number of subordinates for angelaCopy: " + angelaCopy.Subordinates.Count);
    // Number of subordinates for angelaCopy: 0
    Console.WriteLine(angelaCopy.Subordinates.Count == angela.Subordinates.Count);
    // False
}

@cjblomqvist How is that similar scenarios are not a concern for you or everyone else asking for Ignore?

[cjblomqvist]

@Jozkee thanks for your reply!

I believe a scenario which is probably quite common is that you have let's say 100 types, with various relationships between them. Some contian many, some not so many. Basically all are related to at least one other type one way or another, and you have a REST-like Web API to get them from a Web/Mobile application. To map out all relationships in each scenario and make sure you handle all self-referencing scenarios is quite cumbersome. So what happens is, the user is asking for a main type A, and then you ensure you also get all the related types that you believe the "user" might need, such as B, C, D, E, F, G (E, F, G might be sub-relationships). You do not filter this properly (i.e. you do not project out properties not needed) because of not wanting to "waste" time on it (as well as laziness). To map out this tree:

A
A -> B
A -> C
A -> D
A -> B -> E
A -> B -> E -> F
A -> D -> G

So far, all good. The problem is when you also have the following relationships:

C -> A
E -> A
D -> A
G -> A

It's cumbersome to properly keep track of them and filter them out (e.g. through JsonIgnore - which also has the problem of filtering out the relationships in all scenarios, not only for this particular call for A). It would also be cumbersome to handle it using preserve due to the A) wasted space/size to transmit over the network (might be less of a concern in above scenario since obviously we're not so picky with what we're including and not), and B) the data structure does not map in a fully obvious way to JSON, but with a quite new-JSON-library-in-.NET specific way.

Then, to finally answer your question: The "user" (frontend consumer) simply does not care about the additional relationships [C, E, D, G] -> A - it only wants to know the original tree. The loss of data is irrelevant - because the relationship properties are not relevant.

I'm not say that Ignore is perfect by any means, but it is darn convenient to simply add it to a project and then not having to think about it anymore. Something like Preserve could be very useful, but the current implementation is too cumbersome for most cases (with all the downsides listed above) - it doesn't fulfil that simple-and-convenient-albeit-not-100%-correct scenario. I do understand that there are reasons behind Preserve being the way it is, so I'm not saying the Preserve is wrong. What I'm saying is that Preserve is good for one thing (100% correctness while still handling self-referencing), while Ignore is good for another (simple/convenient, but not 100% correct in some cases).

I do believe a lot of people in the community feel the same.

Something in between Ignore and Preserve might be very useful, especially that doesn't break the assumed data structure (to avoid the frontend parsing/unexpected data structure for lists/arrays) while still not loosing data, but I understand that there are difficulties/complexities with this, so Ignore might be good enough to solve the relevant actual real world scenarios.

[hairlesshobo]

Regarding ReferenceHandling.Ignore....

I don't see why this wouldn't be useful. Sure, on deserialisation it could throw up some odd results, but I think for serialisation it's actually quitehelpful. The example use case in the very first comment is a perfect example of why it would be useful. I think it would be ideal to have something like this, because attributes like [JsonIgnore] don't really help, as that would just stop that property from serialising ever.

Currently people are suggesting actively dumping System.text.Json, as can be seen in this question on Stack, where 3 of the 4 answers suggest using Newtonsoft.Json, which has the ignore functionality built in. So people are actively gimping the performance of their application to get a fairly basic feature working. Newtonsoft is _slow_, but at least it works.

EDIT: For reference, I've been using preview 3, I'm yet to try preview 4 but if preserve achieves essentially the same thing without throwing the cycling exception then that would probably be good.

I just want to throw in my two cents on this one. I decided to go down the route of porting my application from Newtonsoft.Json for the sake of performance. I mean, after all, that is one of the huge selling points of System.Text.Json.. I invested hours converting everything over only to find out at the end when I was thoroughly testing everything that I couldn't serialize objects that were pulled by EF core. Talk about a surprise to me!! I even tried the ReferenceHandler.Preserve functionality of 5.0 rc1 and serialization still bombs out due to the circular references.

Anyways, the solution I had to come up with was to use System.Text.Json for the input formatter for MVC because it can handle data coming in just fine, but stick with the slower Newtonsoft.Json as the output formatter, just so I can serialize data from EF core. NOT the ideal solution, but hopefully I will at least get performance benefits on POST/PUT methods.

In short, come on, pushing ReferenceHandler.Ignore to 6.0? I feel this is a pretty basic need and something that, from what I have read, many others just assumed would be available, for the sake of feature parity. Especially since EF core and System.Text.Json are both Microsoft projects, it would seem reasonable that the two could easily be used together.

Please consider adding this to a minor update to 5.0 instead of making us all wait another year to be able to use this new JSON serializer!! Please...

[cjblomqvist]

Since this issue has been closed, it's probably wiser to comment in https://github.com/dotnet/runtime/issues/40099 to get attention 😊

[hairlesshobo]

Whoops, juggling so many things at once right now, I didn't even notice that this one was closed! My bad. Thanks for pointing it out to me.

[AlbertoPa]

We considered having ReferenceHandlig.Ignore but it was cut out of the final design due the lack of scenarios where you would really need Ignore over Preserve.

Here: https://docs.microsoft.com/en-us/ef/core/querying/related-data/serialization Literally, straight out of the official MS documentation.

[BrunoBlanes]

@AlbertoPa I think for that example you can set the ReferenceHandler property to Preserve.

[AlbertoPa]

@BrunoBlanes I have tried it but it does not seem to work if there are many-to-many relationships. Currently I have two workarounds:

1) Annotate the two navigation properties in the join entity with [JsonIgnore]

or

2) Use Newtonsoft.Json with the

options.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;

option on the server. In a Blazor app, the System.Text.Json deserializer seems to be OK with this trick so far.