Runtime: JsonSerializer should support field as well as properties

[YohDeadfall]

_Updated by @layomia: I'm pasting the API approved in https://github.com/dotnet/runtime/issues/34558. Please see that issue for review notes and further discussion._

C# namespace System.Text.Json { public partial class JsonSerializerOptions { public bool IgnoreReadOnlyFields { get; set; } public bool IncludeFields { get; set; } } } namespace System.Text.Json.Serialization { [AttributeUsage(AttributeTargets.Property | Attributes.Field, AllowMultiple = false)] public sealed class JsonIncludeAttribute : JsonAttribute { public JsonIncludeAttribute(); } }

Original proposal (click to view)

There is no way to serialize and deserialize fields using JsonSerializer.

While public fields are generally not recommended, they are used in .NET itself (see value tuples) and by users.

This feature was scoped out of v1 (3.x) due to lack of time, as we prioritized supporting simple POCOs with public properties.

We elected to have an opt-in model for field support because it would be a breaking change to support them by default, and also because public fields are generally not recommended. Other serializers, including Newtonsoft.Json, Utf8Json, and Jil, support this feature by default.

[karelz]

Sounds good to me

[YohDeadfall]

@ahsonkhan Should this be implemented via emitting IL or via reflection accessors?

[ahsonkhan]

Should this be implemented via emitting IL or via reflection accessors?

It would depend on which approach yields better performance. I believe IL emit is the way to go (with a fallback to a reflection-based code path if necessary). @steveharter can provide additional guidance, or correct me if I am mistaken.

[ahsonkhan]

Context from a user's issue where fields not being supported causing a pain-point:
https://github.com/aspnet/AspNetCore/issues/11210

Describe the bug

When I use below code in razor page to post a http request. The deserialized object is not correct. There is no exception happened.

billInfo = await Http.PostJsonAsync<BillInfo>("api/TMobileBill", filePath);

I then changed to use Http.PostAsync to get the response string, then deserialize it by using Newtonsoft library. It works fine.

        var myContent = Newtonsoft.Json.JsonConvert.SerializeObject(filePath);
        var buffer = System.Text.Encoding.UTF8.GetBytes(myContent);
        var byteContent = new ByteArrayContent(buffer);
        byteContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json");
        var response = await Http.PostAsync("api/TMobileBill", byteContent);
        var responseStr = await response.Content.ReadAsStringAsync();
        billInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<BillInfo>(responseStr);
        StateHasChanged();

The response json string in my problem is:

{"period":"May 09, 2019 - Jun 08, 2019","dueDate":"Jul 01, 2019","totalAmount":326.03,"billInfoPerPerson":[{"name":"A B","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":4.84,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":5.48,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":60.0},{"name":"C D","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.08,"regularCharge":20.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":1.47,"regularCharge":20.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":40.0},{"name":"E F","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.05,"regularCharge":20.0,"otherCharge":28.03,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":2.15,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":78.03},{"name":"G H","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.52,"regularCharge":20.0,"otherCharge":17.5,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":true,"usage":5.27,"regularCharge":30.0,"otherCharge":20.5,"totalCharge":0.0}],"totalAmount":88.0},{"name":"I J","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":3.67,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":3.05,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":60.0}],"warning":"There are $3 account service fee. Make sure it is expected. "}

The corresponding class is:

    public class PersonBillInfo
    {
        public string Name;
        public List<PhoneBillInfo> PhonesInfo;
        public float TotalAmount;
    }

    public class PhoneBillInfo
    {
        public string PhoneNumber;
        public bool PlanHost;
        public float Usage;
        public float RegularCharge;
        public float OtherCharge;
        public float TotalCharge;
    }

    public class BillInfo
    {
        public string Period;
        public string DueDate;
        public float TotalAmount;
        public IEnumerable<PersonBillInfo> BillInfoPerPerson;
        public string Warning;
    }

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core 'SDK 3.0.100-preview6-012264'
2. Implement a controller action to respond the json string as above.
3. From razor page, use below code to call the controller action.

var billInfo = Http.PostJsonAsync<BillInfo>("api/TMobileBill", filePath);

1. Evaluate the billInfo object. All the members of billInfo object are null.

Expected behavior

The members of billInfo object have values.

[YohDeadfall]

@steveharter Without dotnet/corefx#38233 I'm not able to implement this one. Later it would be possible to access fields without code emit, but it's work in progress by @GrabYourPitchforks.

[Timovzl]

Should this be implemented via emitting IL or via reflection accessors?

IL emit also has the advantage of letting us write to readonly fields, something I have not been able to achieve with expressions or, if memory serves, with reflection.

In case anyone trips on the idea, note that, while writing to readonly fields may seem questionable at first glance, it actually makes perfect sense during deserialization. After all, deserialization is an alternate way to construct a fully populated object. It is comparable to using a specific constructor that takes a parameter for each field - a perfect using case for assigning our readonly fields. The fact that deserialization is often not _implemented_ by using such a constructor does not change the fact that we are trying to constructor a fully populated object. In other words, being able to assign readonly fields during deserialization is valuable and makes sense.

[ahsonkhan]

From @MarkStega in https://github.com/dotnet/corefx/issues/41374

It appears that System.Text.Json.Deserialize does not properly handle an array of POCO's inside another POCO.

Build the following console application:

using Newtonsoft.Json;
using System;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello Json!");

            string json = "{\"pStatus\":0,\"pError\":\"\",\"pResult\":[{\"processIdentifier\":"
                + "\"ANES - 2019 - 9955\",\"associatedProcessIdentifier\":\"\",\"dashboard\":\" "
                + "| SPD_Version | 5 | SPD_AnesthesiaType | Choice | SPD_BloodProducts ||\"},"
                + "{\"processIdentifier\":\"ANES - 2019 - 9964\",\"associatedProcessIdentifier\":\"\","
                + "\"dashboard\":\" | SPD_Version | 5 | SPD_AnesthesiaType | Choice | SPD_BloodProducts ||\""
                + "}]}";

            var newton = Newtonsoft.Json.JsonConvert.DeserializeObject<ServiceResult<Process_DD[]>>(json);

            var stjo = new System.Text.Json.JsonSerializerOptions();
            stjo.PropertyNameCaseInsensitive = true;
            var stj = System.Text.Json.JsonSerializer.Deserialize<ServiceResult<Process_DD[]>>(json, stjo);
        }
    }

    public class ServiceResult<ServiceResultType>
    {
        public enum eStatus { OK, Error };
        public eStatus pStatus { get; set; }
        public string pError { get; set; }
        public ServiceResultType pResult { get; set; }

        public ServiceResult() { }
        public ServiceResult(
            ServiceResultType p_Result,
            eStatus p_Status = eStatus.OK,
            string p_Error = "")
        {
            pStatus = p_Status;
            pError = p_Error;
            pResult = p_Result;
        }
    }

    public class Process_DD
    {
        public string ProcessIdentifier;
        public string AssociatedProcessIdentifier;
        public string Dashboard;

        public Process_DD()
        {
        }
        public Process_DD(string processIdentifier, string associatedProcessIdentifier, string dashboard)
        {
            this.ProcessIdentifier = processIdentifier;
            this.AssociatedProcessIdentifier = associatedProcessIdentifier;
            this.Dashboard = dashboard;
        }
    }

}

using the following csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    &lt;OutputType&gt;Exe</OutputType&gt;
    &lt;TargetFramework&gt;netcoreapp3.0&lt;/TargetFramework&gt;
  &lt;/PropertyGroup&gt;

  &lt;ItemGroup&gt;
    &lt;PackageReference Include="Newtonsoft.Json" Version="12.0.2" /&gt;
  &lt;/ItemGroup&gt;

&lt;/Project&gt;

Debug the program and step through both deserializations. Note that the Newtonsoft result has appropriate values for the two array elements of type Process_DD. The System.Text.Json result has null values for the properties.

C:Usersms>dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 3.0.100
Commit: 04339c3a26

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:Program Filesdotnetsdk3.0.100

Host (useful for support):
Version: 3.0.0
Commit: 7d57652f33

.NET Core SDKs installed:
2.1.802 [C:Program Filesdotnetsdk]
2.2.402 [C:Program Filesdotnetsdk]
3.0.100 [C:Program Filesdotnetsdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.13 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.7 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.13 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.7 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.0.0 [C:Program FilesdotnetsharedMicrosoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.13 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 2.2.7 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.NETCore.App 3.0.0 [C:Program FilesdotnetsharedMicrosoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.0.0 [C:Program FilesdotnetsharedMicrosoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[ahsonkhan]

From https://github.com/dotnet/corefx/issues/41420 by @codexguy:

To reproduce:
Take the sample default ASPNET Core 3.0 app (API-based template), change the WeatherForecast class to include a ValueTuple:

public class WeatherForecast
{

...

    public (int x, int y) Location { get; set; }
}

Change the Get method to populate Location with arbitrary values. Run the project: JSON returned has an empty location. Serializing ValueTuple's is possible with Newtonsoft (being unable to do so wrecks existing code I have that uses ValueTuples successfully with Core 2.1, moving to Core 3.0).

[layomia]

From @kaamil1984 in https://github.com/dotnet/runtime/issues/552:

Can we have tuple support in System.Text.Json.Serializer?

General

Is there any chance, that we can have tuple support in System.Text.Json.Serializer?

I would like to serialize and deserialize object like this:

(int Result, string SuccessMessage, string ErrorMessage) MyTuple;

[marcuslindblom]

Any progress here or what is the prefered workaround for this issue?

[YohDeadfall]

Previously it was blocked by the team due to optimizations. Recently pinged @ahsonkhanon this, but have received no answer.

Anyway I'm going to move the PR to dotnet/runtime in a few days.

[marcuslindblom]

@YohDeadfall does this address named tuple serialization as well? I use Newtonsoft Json for now but it serialize this (string Landscape, string Portrait) Images; as Item1 and Item2.

[YohDeadfall]

Don't think that named tuples worth support because names available only in very limited number of cases and in most of them it won't work well. Names provided by TupleElementNamesAttribute and, as you know, attributes cannot be applied to variables. Therefore, names can be serialized when a value tuple is a part of another object, but this probably means a bad design.

[marcuslindblom]

@YohDeadfall Ok, I see. Do I understand you correct if I could use TupleElementAttribute if I have a model like this?

```c#
public class Home {
public string Heading { get;set; }
public ArtDirection ArtDirection { get;set; }
}

public class ArtDirection {
public string Alt { get; set; }
public (string Landscape, string Portrait, string Fallback) Images { get; set; }
}
```

[YohDeadfall]

Yes, it is only one possible way to get names of tuple members, but in other cases it isn't possible (when you pass a tuple directly to the serializer). Therefore, supporting TupleElementNamesAttribute would cause different behavior in different cases and misunderstanding by users. The final decision on this should be made by Microsoft team.

It's better to have a class/struct for that kind of scenario.

/cc @ahsonkhan @steveharter @layomia

[ahsonkhan]

From @newpost in https://github.com/dotnet/runtime/issues/1519

[ApiController]
    [Route("[controller]")]
    public class DefaultController : ControllerBase
    {
        [HttpGet]
        public ValueTuple<int, int, int> Index()
        {
            return new ValueTuple<int,int,int>(1,2,3);
        }
    }

actual output: {}.
expected output:{"item1":1,"item2":2,"item3":3}

[ahsonkhan]

I mention a workaround for ValueTuples (up to 3) based on custom converters if folks need it.
https://github.com/dotnet/runtime/issues/1519#issuecomment-572751931

[ahsonkhan]

From @Greypuma in https://github.com/dotnet/runtime/issues/32294

I was trying to serialize an object into json. I get an empty string. below is the code i used.
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Text.Json;
using System.Text.Json.Serialization;
using Newtonsoft.Json;

namespace test
{
public class AdbEmulator
{
//needs to be public otherwise wont work
public string name;
internal string attached;

}
public class AndroidJson
{
    public AndroidJson()
    {
        AddData();
    }
    public void AddData()
    {
        AdbEmulator adb1 = new AdbEmulator() { name = "emulator-5554", attached = "ll" };
        emulators.Add(adb1);
        string jsonAdb = System.Text.Json.JsonSerializer.Serialize<AdbEmulator>(adb1);
        string newtonjson = Newtonsoft.Json.JsonConvert.SerializeObject(adb1);
    }
}

Program.cs
static void Main(string[] args)
{ AndroidJson aj = new AndroidJson(); }

Output
jsonAdb = {}
newtonjson ={"name":"emulator-5554"}

I'm not sure why newtonsoft works and the microsoft one does not.
i'v'e tried the code in an dotnet core 3.1 project and windows application ( including the nuget package for that project).

[JVimes]

@YohDeadfall, it looks like you have a pull request for this in the old repo. Should that be moved here, like maryamariyan suggested?

[YohDeadfall]

Already did this by #2192, but since field addition is a breaking change there should be an API review. Will write all things @layomia requested tomorrow or on Saturday.

[YohDeadfall]

@layomia @karelz updated the first message to list problems which block the PR.

[rafaelbenavent]

Any update on this?

[layomia]

From @leung85 in https://github.com/dotnet/runtime/issues/34001:

core version 3.1.102

 public class TestDict
    {
        public Dictionary<string, string> Test = new Dictionary<string, string>(){
            { "09:00", "In"},
            { "18:00", "Out"}
        };
    }

 var option = new JsonSerializerOptions();
            option.WriteIndented = true;
            string configJson = System.Text.Json.JsonSerializer.Serialize(new TestDict(), option);

the configJson return {},
But it can be serialized if just use Dictionary directly.

[layomia]

@YohDeadfall, are you still interested in implementing this feature? API & behavior spec approved in https://github.com/dotnet/runtime/issues/34558:

C# namespace System.Text.Json { public partial class JsonSerializerOptions { public bool IgnoreReadOnlyFields { get; set; } public bool IncludeFields { get; set; } } } namespace System.Text.Json.Serialization { [AttributeUsage(AttributeTargets.Property | Attributes.Field, AllowMultiple = false)] public sealed class JsonIncludeAttribute : JsonAttribute { public JsonIncludeAttribute(); } }

[YohDeadfall]

Will take a look and implement. Probably will resurrect the closed PR with the requested API above.

[mnns]

Spent a lot of time understanding why I don't see the fields in my output. Anyway, using 3.1, still don't see this option. Any news? thanks

[YohDeadfall]

It should be .NET 5 feature.

[layomia]

@YohDeadfall, can you take another look soon? Field support is one of the top priorities for System.Text.Json in 5.0, and we'd like to get the feature in a preview for folks to try.

[YohDeadfall]

Planned to reopen the PR on this weekend. Working on this won't be problematic, but the snake case naming policy will since there is a stuck discussion.

[layomia]

Planned to reopen the PR on this weekend.

Sounds good 👍

but the snake case naming policy will since there is a stuck discussion.

We should address snake case separately from this issue - https://github.com/dotnet/runtime/issues/782#issuecomment-629435686.

[YohDeadfall]

While it isn't too late, there are a few Ignore properties in JsonSerializerOptions. Should IncludeFields be renamed to IgnoreFields with the default value true for consistency?

[layomia]

Should IncludeFields be renamed to IgnoreFields with the default value true for consistency?

This was considered, but we chose IncludeFields so that the default value would be false for consistency with other options (where CLR default is used).

[jessicah]

Could we have the JsonInclude also apply at the struct level?

[YohDeadfall]

What did you mean? It's not working for structs or something else?