Runtime: StackOverflowException calling JsonSerializer.DeserializeAsync<T>() using .NET Core 3.0 preview 7

[martincostello]

I've updated an application targeting ASP.NET Core 3.0 preview 6 that used Netwonsoft.Json to deserialize JSON from HTTP calls to preview 7.

After upgrading the application successfully to preview 7 with all tests passing, I proceeded to update it to use System.Text.Json instead of Newtonsoft.Json. However, with these changes made the tests fail because dotnet test terminates with a StackOverflowException.

An other application that re-uses very similar models showed the same behaviour when upgraded, but another with a very simple JSON object model did not.

After some trial and error I narrowed this down to _somewhere_ in the System.Text.Json APIs. With that hint, on a hunch I updated the repro I created for dotnet/runtime#29338 and ran it. The original repro for that issue now produces a StackOverflowException.

PS C:\Coding\JsonSerializer-NullReferenceException-Repro> dotnet test
Test run for C:\Coding\JsonSerializer-NullReferenceException-Repro\bin\Debug\netcoreapp3.0\JsonSerializer-NullReferenceException-Repro.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
The active test run was aborted. Reason: Test host process crashed : Stack overflow.


Test Run Aborted.

.NET info below:

PS C:\Coding\JsonSerializer-NullReferenceException-Repro> dotnet --version
3.0.100-preview7-012821

PS C:\Coding\JsonSerializer-NullReferenceException-Repro> dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview7-012821
 Commit:    6348f1068a

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview7-012821\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

My guess would be something to do with the way types are re-used within the object hierarchy is causing an issue that's bypassing the stack depth setting (I'm right in assuming it's not unbounded by default?) and crashing the process.

To repro:

1. Clone https://github.com/martincostello/JsonSerializer-NullReferenceException-Repro
2. Run dotnet test

/cc @BrennanConroy @steveharter

[ericstj]

Here's the callstack that's causing overflow:

    System.Private.CoreLib.dll!System.RuntimeTypeHandle.GetInstantiationPublic() Line 490   C#
    System.Private.CoreLib.dll!System.RuntimeType.GetGenericArguments() Line 3247   C#
    System.Private.CoreLib.dll!System.RuntimeType.MakeGenericType(System.Type[] instantiation) Line 3266    C#
    System.Text.Json.dll!System.Text.Json.ReflectionEmitMemberAccessor.CreatePropertyGetter(System.Reflection.PropertyInfo propertyInfo, System.Type classType) Line 112    C#
    System.Text.Json.dll!System.Text.Json.ReflectionEmitMemberAccessor.CreatePropertyGetter<JsonSerializer_NullReferenceException_Repro.ModelL, int>(System.Reflection.PropertyInfo propertyInfo) Line 81   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfoCommon<JsonSerializer_NullReferenceException_Repro.ModelL, int, int, int>.Initialize(System.Type parentClassType, System.Type declaredPropertyType, System.Type runtimePropertyType, System.Reflection.PropertyInfo propertyInfo, System.Type elementType, System.Text.Json.Serialization.JsonConverter converter, System.Text.Json.JsonSerializerOptions options) Line 40    C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.CreateProperty(System.Type declaredPropertyType, System.Type runtimePropertyType, System.Reflection.PropertyInfo propertyInfo, System.Type parentClassType, System.Text.Json.JsonSerializerOptions options) Line 130    C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 27   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
    System.Text.Json.dll!System.Text.Json.JsonPropertyInfo.ElementClassInfo.get() Line 202  C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.AddProperty(System.Type propertyType, System.Reflection.PropertyInfo propertyInfo, System.Type classType, System.Text.Json.JsonSerializerOptions options) Line 41   C#
    System.Text.Json.dll!System.Text.Json.JsonClassInfo.JsonClassInfo(System.Type type, System.Text.Json.JsonSerializerOptions options) Line 97 C#
    System.Text.Json.dll!System.Text.Json.JsonSerializerOptions.GetOrAddClass(System.Type classType) Line 304   C#
       <Pattern repeats>

Debugging to see what state is causing this cycle.

[steveharter]

Note that ModelH and ModelL are circular, but that is (should be) supported. See also https://github.com/dotnet/corefx/issues/37313.

[ericstj]

Yeah, I was suspecting that. To debug the best way I found was to set a breakpoint at one of the methods in above stack and set the hit-count to something high like a couple 100. Then you can observe the cycle. I didn't get to the bottom of where the bug is but hopefully that can help the next person pick it up.

[steveharter]

In summary, the type cycle between ModelH and ModelL (Root.H --> ModelH --> ModelH.C --> IList<ModelL> --> ModelL.M --> IList<ModelH>) plus that the cycle was caused by a collection (instead of just an object reference) caused the issue.

[steveharter]

@martincostello thanks for reporting this problem and the nice repro.

[martincostello]

I guess this fix missed the deadline for preview 8 and will be in preview 9?

[ericstj]

Correct, or if you want you can pick up the latest package out of master if you want to try it out right away.

[martincostello]

The easiest way for me to validate this in the original application I found it in is via the SDK, so I'll wait until this gets ported to the 3.0 branch and bubbles up to the SDK repo, then I can validate from a 3.0 SDK branch build 👍

[karelz]

Fixed for 3.0 in PR dotnet/corefx#40071 and in master (5.0) in PR dotnet/corefx#39936