Newtonsoft.json: .NET Core 2.0 (preview) regression compared to .NET Core 1.1

[JustArchi]

.NET Core 2.0 is not released yet, so there is no reason to panic, but I spotted a pretty nasty regression which probably should be looked into, assuming it's not .NET core 2.0 bug.

Source/destination types

ConcurrentDictionary<byte, string>

Source/destination JSON

{"TestDictionary":{"0":"0"}}

Expected behavior

Proper deserialization of json to ConcurrentDictionary<byte, string>.

Actual behavior

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Concurrent.ConcurrentDictionary`2.InitializeFromCollection(IEnumerable`1 collection)
   at System.Collections.Concurrent.ConcurrentDictionary`2.OnDeserialized(StreamingContext context)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at Newtonsoft.Json.Serialization.JsonContract.InvokeOnDeserialized(Object o, StreamingContext context)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at test.Program.Main()

Steps to reproduce

using System;
using System.Collections.Concurrent;
using Newtonsoft.Json;

namespace test {
    internal static class Program {
        private static void Main() {
            TestObject o1 = new TestObject();
            o1.TestDictionary[0] = "0";
            string json = JsonConvert.SerializeObject(o1);

            TestObject o2 = JsonConvert.DeserializeObject<TestObject>(json);
        }

        internal sealed class TestObject {
            [JsonProperty]
            internal readonly ConcurrentDictionary<byte, string> TestDictionary = new ConcurrentDictionary<byte, string>();
        }
    }
}

With .NET Core 1.1 csproj file such as:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="10.0.2" />
  </ItemGroup>

</Project>

Everything works fine.

If we switch <TargetFramework> from netcoreapp1.1 to netcoreapp2.0 and try to build with latest .NET Core preview bits, previously flawless result quickly changes into unhandled exception.

I'm not entirely sure whether this is change of some internal API that Newtonsoft.Json is using (that should be adapted), or whether this is actual .NET Core 2.0 bug that should be forwarded, but judging from amount of passing tests for .NET Core, I'm pretty sure this is simply Newtonsoft.Json incompatibility that should be corrected here, hence I'm reporting an issue.

I'd be very grateful if you could look into this, as .NET Core 2.0 is supposed to be released pretty soon, and I'm sure that many people will attempt to use the library with it, as there is no net standard incompatibility with upcoming version.

P.S. this commit could have something to do with it.

Thank you in advance!

.NET Command Line Tools (2.0.0-preview2-005840)

Product Information:
 Version:            2.0.0-preview2-005840
 Commit SHA-1 hash:  8f2fcef544

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         ubuntu.16.10-x64
 Base Path:   /opt/dotnet/sdk/2.0.0-preview2-005840/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0-preview1-002061-00
  Build    : 2b70ec9c3b014af0c2a5f45de0e5b73a1ae51c09

[JamesNK]

Thanks. I can put a fix in Json.NET but I'll also log it with MS and see whether they want a fix ConcurrentDictionary properly.

[JustArchi]

That was my intention for the entire time - just to let you know so you can check it out, as it's definitely better to do that right now, rather than when .NET Core 2.0 goes live. Thank you a lot for everything you're doing! 👍

[JamesNK]

MS is being a bag of dicks about fixing it for some reason so I've fixed it inside Json.NET https://github.com/JamesNK/Newtonsoft.Json/commit/8eecadd0ef5dc5ddb153a6a6f9d36f46f11ec68a

[JustArchi]

Since .NET Core 2.0 preview 1 got released, any chance of seeing updated Newtonsoft.Json nuget package available soon? I'd love to give it a second look and ensure there are no more bugs before it goes stable, thank you 👍.

[JamesNK]

Unlikely in the next month.

[jdarundell]

This is still a problem in the production release of ASP CORE 2.0.0
My experience with ConcurrentDictionary

[JamesNK]

This passes in .NET Core 2

https://github.com/JamesNK/Newtonsoft.Json/commit/bbf735583ffced28ce89594397fbd57db76dfb75

[JustArchi]

I can confirm that this issue is fixed in latest version of the library on .NET Core 2.0 (I tested also preview2 before).