Aspnetcore: HttpClient.PostJsonAsync deserialize json failed

[yan0lovesha]

Describe the bug

When I use below code in razor page to post a http request. The deserialized object is not correct. There is no exception happened.

billInfo = await Http.PostJsonAsync<BillInfo>("api/TMobileBill", filePath);

I then changed to use Http.PostAsync to get the response string, then deserialize it by using Newtonsoft library. It works fine.

        var myContent = Newtonsoft.Json.JsonConvert.SerializeObject(filePath);
        var buffer = System.Text.Encoding.UTF8.GetBytes(myContent);
        var byteContent = new ByteArrayContent(buffer);
        byteContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json");
        var response = await Http.PostAsync("api/TMobileBill", byteContent);
        var responseStr = await response.Content.ReadAsStringAsync();
        billInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<BillInfo>(responseStr);
        StateHasChanged();

The response json string in my problem is:

{"period":"May 09, 2019 - Jun 08, 2019","dueDate":"Jul 01, 2019","totalAmount":326.03,"billInfoPerPerson":[{"name":"A B","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":4.84,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":5.48,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":60.0},{"name":"C D","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.08,"regularCharge":20.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":1.47,"regularCharge":20.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":40.0},{"name":"E F","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.05,"regularCharge":20.0,"otherCharge":28.03,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":2.15,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":78.03},{"name":"G H","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":1.52,"regularCharge":20.0,"otherCharge":17.5,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":true,"usage":5.27,"regularCharge":30.0,"otherCharge":20.5,"totalCharge":0.0}],"totalAmount":88.0},{"name":"I J","phonesInfo":[{"phoneNumber":"[phone]","planHost":false,"usage":3.67,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0},{"phoneNumber":"[phone]","planHost":false,"usage":3.05,"regularCharge":30.0,"otherCharge":0.0,"totalCharge":0.0}],"totalAmount":60.0}],"warning":"There are $3 account service fee. Make sure it is expected. "}

The corresponding class is:

    public class PersonBillInfo
    {
        public string Name;
        public List<PhoneBillInfo> PhonesInfo;
        public float TotalAmount;
    }

    public class PhoneBillInfo
    {
        public string PhoneNumber;
        public bool PlanHost;
        public float Usage;
        public float RegularCharge;
        public float OtherCharge;
        public float TotalCharge;
    }

    public class BillInfo
    {
        public string Period;
        public string DueDate;
        public float TotalAmount;
        public IEnumerable<PersonBillInfo> BillInfoPerPerson;
        public string Warning;
    }

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core 'SDK 3.0.100-preview6-012264'
2. Implement a controller action to respond the json string as above.
3. From razor page, use below code to call the controller action.

var billInfo = Http.PostJsonAsync<BillInfo>("api/TMobileBill", filePath);

1. Evaluate the billInfo object. All the members of billInfo object are null.

Expected behavior

The members of billInfo object have values.

[ctrl-alt-d]

Issue for Net Core Preview 6

( runs fine in preview 5 )

Same problem here with HttpClientJsonExtensions at Microsoft.AspNetCore.Components on dotnet core preview6:

• post is done ✔
• response is received ✔
• json is not deserialized ✘

Tis code is running fine on preview5 and fails silently on preview6:

r = await HttpClient.PostJsonAsync<DtoCapsule<SessionDto>>(url, parms);

@yan0lovesha 's workaround make the code running:

var myContent = Newtonsoft.Json.JsonConvert.SerializeObject(parms);
var buffer = System.Text.Encoding.UTF8.GetBytes(myContent);
var byteContent = new ByteArrayContent(buffer);
byteContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json");
var response = await HttpClient.PostAsync(url, byteContent);
var responseStr = await response.Content.ReadAsStringAsync();
r = Newtonsoft.Json.JsonConvert.DeserializeObject<DtoCapsule<SessionDto>>(responseStr);

Additional info: the classes to deserialize in previous example ( "my code" ):

    public class SessionDto
    {
        public int Id { get; set; }
        public string JWT { get; set; }
    }


    public class DtoCapsule<T>
    {
        public T data;
        public IEnumerable<BrokenRule> brokenRules = new BrokenRules();
    }

    public class BrokenRule
    {
        public string Member { set; get; }
        public Severity Severity { set; get; }
        public string Message { set; get; }
    }

[anurse]

Seems like it might be a difference with how System.Text.Json deserializes. cc @ahsonkhan

@yan0lovesha thanks for the sample JSON and classes, we can use that to see if we can repro the issue.

[danroth27]

@pranavkm @rynowak System.Text.Json issue

[yan0lovesha]

@danroth27 Yes you're right. It was too late last night. I went a little bit deeper this morning. Below code statements on LinqPad6 can prove it:

var jsonString = "{\"period\":\"May 09, 2019 - Jun 08, 2019\",\"dueDate\":\"Jul 01, 2019\",\"totalAmount\":326.03}";

var jsonObjFromDotnetCore = System.Text.Json.Serialization.JsonSerializer.Parse<BillInfo>(jsonString);
var jsonObjFromNewtonsoft = Newtonsoft.Json.JsonConvert.DeserializeObject<BillInfo>(jsonString);

jsonObjFromDotnetCore.Dump();
jsonObjFromNewtonsoft.Dump();

}

public class BillInfo
{
    public string Period;
    public string DueDate;
    public float TotalAmount;

Seems that we need to wait for next preview release on this.

[danroth27]

@yan0lovesha I believe the issue here is that System.Text.Json doesn't support doing serialization with public fields. Can you try using public properties instead?

[pranavkm]

Here's what I tried with the settings PostAsJsonAsync currently uses and it works fine:

```C#
static void Main(string[] args)
{
var jsonString = "{\"period\":\"May 09, 2019 - Jun 08, 2019\",\"dueDate\":\"Jul 01, 2019\",\"totalAmount\":326.03}";

var jsonObjFromDotnetCore = JsonSerializer.Parse<BillInfo>(jsonString, new JsonSerializerOptions {PropertyNamingPolicy = JsonNamingPolicy.CamelCase});
System.Console.WriteLine(jsonObjFromDotnetCore.Period);

}

public class BillInfo
{
public string Period { get; set; }
public string DueDate { get; set; }
public float TotalAmount { get; set; }
}
```

I'd recommend filing an issue in https://github.com/dotnet/corefx/issues to consider adding support for public fields in a future release.

[ahsonkhan]

Yes, the reason for the missing info is that fields are not supported in the new System.Text.Json library (only public properties are supported).

We have an issue open for this already: https://github.com/dotnet/corefx/issues/36505

I have added the initial repro/post to that issue.

[yan0lovesha]

As this blog mentioned. I tried to use services.AddMvc().AddJsonOptions(...) in Startup class. The default WeatherForecast api works fine. It can respond a correct json string. But for my own api controller, it just output an empty json object "{}". With the same api controller code, AddNewtonsoftJson works fine. This maybe another issue? Or it needs some special configuration?

[ctrl-alt-d]

@yan0lovesha I believe the issue here is that System.Text.Json doesn't support doing serialization with public fields. Can you try using public properties instead?

Works for me changing public fields to public properties.

Doesn't run:

    public class DtoCapsule<T>
    {
        public T data ;
        public IEnumerable<BrokenRule> brokenRules = new BrokenRules();
    }

Runs fine:

    public class DtoCapsule<T>
    {
        public T data { set; get; }
        public IEnumerable<BrokenRule> brokenRules { set; get; }  = new BrokenRules();
    }

[petroemil]

I ran into this very same issue with my Client side Blazor web app, after upgrading to Preview 6. Serialization just doesn't work. But for me it doesn't work even if I make my properties public. I only get an empty object back.

[pranavkm]

@yan0lovesha \ @petroemil could you please share a simple application that reproduces the error? It's difficult to investigate without knowing what it is that's being serialized.

[petroemil]

@pranavkm

My C# class looks like this:

public class BlogPostMetadata
{
    public bool IsDraft { get; set; }
    public DateTimeOffset PublishDate { get; set; }
    public string Title { get; set; }
    public string HeroImageFile { get; set; }
    public string SummaryMarkdownFile { get; set; }
    public string? MarkdownFile { get; set; }
    public string SocialSharingFile { get; set; }
}

A sample JSON content for it looks like this

{
  "Title": "Hello World",
  "PublishDate": "2019-05-17 16:19:34 -08:00",
  "HeroImageFile": "heroimage.jpg",
  "SummaryMarkdownFile": "summary.md",
  "MarkdownFile": "post.md",
  "SocialSharingFile": "socialshare.html"
}

My referenced ASP.NET libraries are:

<PackageReference Include="Microsoft.AspNetCore.Blazor" Version="3.0.0-preview6.19307.2" />
<PackageReference Include="Microsoft.AspNetCore.Blazor.Build" Version="3.0.0-preview6.19307.2" PrivateAssets="all" />
<PackageReference Include="Microsoft.AspNetCore.Blazor.DevServer" Version="3.0.0-preview6.19307.2" PrivateAssets="all" />

I'm getting this JSON from a file with the GetJsonAsync<T> extension method on the HttpClient but I also tried to manually use JsonSerializer.Parse<T> but of course with no luck.

I'm not getting any exception, just an empty object.

[pranavkm]

@petroemil is the JSON produced from an MVC action? They're usually camel case which would explain why JsonSerializer.Parse does not work unless you also specify the camel case option.

[petroemil]

@pranavkm the JSON is coming from a file (wwwroot content) as shown in the example above. The JSON property names match the C# property names.

[ahsonkhan]

Since the DateTimeOffset within the JSON is not ISO 8601 compliant (it contains spaces), the JsonSerializer throws an exception (see below, at least on preview 7). If the contents of PublishDate was changed to ""2019-05-17T16:19:34-08:00"", it works as expected (and the object is deserialized fully).

```C#
public class BlogPostMetadata
{
public bool IsDraft { get; set; }
public DateTimeOffset PublishDate { get; set; }
public string Title { get; set; }
public string HeroImageFile { get; set; }
public string SummaryMarkdownFile { get; set; }
public string MarkdownFile { get; set; }
public string SocialSharingFile { get; set; }
}

[Fact]
public static void TestingStuff()
{
string temp = @"{
""Title"": ""Hello World"",
""PublishDate"": ""2019-05-17 16:19:34 -08:00"", // spaces don't work
""HeroImageFile"": ""heroimage.jpg"",
""SummaryMarkdownFile"": ""summary.md"",
""MarkdownFile"": ""post.md"",
""SocialSharingFile"": ""socialshare.html""
}";

BlogPostMetadata foo = JsonSerializer.Parse<BlogPostMetadata>(temp);
Assert.Equal("Hello World", foo.Title);

}

```text
System.Text.Json.JsonException : The JSON value could not be converted to System.DateTimeOffset. Path: $.PublishDate | LineNumber: 2 | BytePositionInLine: 45.

cc @steveharter, @layomia

[petroemil]

@ahsonkhan wow, thank you very much.

So even in Client side Blazor the serializer only got changed to System.Text.Json in Preview 6 and before that it was still Newtonsoft? Because in Preview 5 this still worked fine.
Also I haven't seen the exception, only getting an empty object back for seemingly no reason.

[newline-coder]

https://github.com/dotnet/corefx/blob/master/src/System.Text.Json/docs/SerializerProgrammingModel.md#date-support

[pranavkm]

@petroemil trying this out with preview7 I can see the error appear in the output:

C# info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished in 4.3179ms 200 application/json warn: Microsoft.AspNetCore.Components.Browser.Rendering.RemoteRenderer[100] Unhandled exception rendering component: The JSON value could not be converted to System.DateTimeOffset. Path: $.PublishDate | LineNumber: 2 | BytePositionInLine: 45. System.Text.Json.JsonException: The JSON value could not be converted to System.DateTimeOffset. Path: $.PublishDate | LineNumber: 2 | BytePositionInLine: 45. at System.Text.Json.ThrowHelper.ThowJsonException(String message, Utf8JsonReader& reader, String path) at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType, Utf8JsonReader& reader, String path) at System.Text.Json.JsonPropertyInfoNotNullable`3.Read(JsonTokenType tokenType, ReadStack& state, Utf8JsonReader& reader) at System.Text.Json.JsonSerializer.HandleValue(JsonTokenType tokenType, JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& state) at System.Text.Json.JsonSerializer.ReadCore(JsonSerializerOptions options, Utf8JsonReader& reader, ReadStack& readStack) at System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader) at System.Text.Json.JsonSerializer.ParseCore(String json, Type returnType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Parse[TValue](String json, JsonSerializerOptions options) at Microsoft.AspNetCore.Components.HttpClientJsonExtensions.GetJsonAsync[T](HttpClient httpClient, String requestUri)

@yan0lovesha I'm closing this issue since it looks like using properties addressed the original issue you filed. Feel free to file a new work item if you're running in to further issues in this area.

[petroemil]

As a last note, if we are already here.

I'm still having issues getting the properly deserialized object.

Even if my class has { get; set; } properties, when I try to use .GetJsonAsync<T>() extension method on the HttpClient, I only get an empty object.
If I explicitly use .GetStringAsync() + JsonSerializer.Parse<T> together, only then it works.

I also noticed that { get; private set; } properties are not supported, neither are read-only ( { get; } ) properties with constructor initialization. So basically I can't deserialize an immutable object.