Runtime: JsonSerializer.Deserialize is intolerably slow in Blazor WebAssembly, but very fast in .NET Core integration test

[szalapski]

In my Blazor app, I have a component that has a method like this. (I've replaced a call to GetFromJsonAsync with code from inside it, to narrow down the slow part.)

```C#
private async Task GetData()
{
IsLoading = true;
string url = $".../api/v1/Foo"; // will return a 1.5 MB JSON array
var client = clientFactory.CreateClient("MyNamedClient");

  Console.WriteLine($"starting");

  List<Foo> results;

  Task<HttpResponseMessage> taskResponse = client.GetAsync(url, HttpCompletionOption.ResponseContentRead, default);

  var sw = Stopwatch.StartNew();
  using (HttpResponseMessage response = await taskResponse)
  {

    response.EnsureSuccessStatusCode();
    var content = response.Content!;

    if (content == null)
    {
      throw new ArgumentNullException(nameof(content));
    }

    string contentString = await content.ReadAsStringAsync();

    sw.Stop();
    Console.WriteLine($"Read string: {sw.Elapsed}");
    sw.Restart();

    results = System.Text.Json.JsonSerializer.Deserialize<List<Foo>>(contentString)!;
    //results = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Foo>>(contentString); // comparable

  }

  sw.Stop();
  Console.WriteLine($"Deserialize: {sw.Elapsed}");

  StateHasChanged();
  IsLoading = false;

My download of 2-6 MB takes 1-6 seconds, but the rest of the operation (during which the UI is blocked) takes 10-30 seconds.  **Is this just slow** deserialization in `ReadFromJsonAsync` (which calls `System.Text.Json.JsonSerializer.Deserialize` internally), or is there **something else going on here**?  **How can I improve the efficiency** of getting this large set of data (though it isn't all that big, I think!) 

I have commented out anything bound to `Results` to simplify, and instead I just have an indicator bound to `IsLoading`. This tells me there's no slowness in updating the DOM or rendering.

When I attempt the same set of code in an automated integration test, it only takes 3 seconds or so (the download time). Is WebAssembly really that slow at deserializing?  If so, is the only solution to retrieve very small data sets everywhere on my site?  This doesn't seem right to me.  **Can this slowness be fixed?**


Here's the resulting browser console log from running the above code:

VM1131:1 Fetch finished loading: GET "https://localhost:5001/api/v1/Foo".
Read string: 00:00:05.5464300
Deserialize: 00:00:15.4109950
L: GC_MAJOR_SWEEP: major size: 3232K in use: 28547K
L: GC_MAJOR: (LOS overflow) time 18.49ms, stw 18.50ms los size: 2048K in use: 187K
L: GC_MINOR: (LOS overflow) time 0.33ms, stw 0.37ms promoted 0K major size: 3232K in use: 2014K los size: 2048K in use: 187K
```

Using Newtonsoft.Json (as in the commented-out line) instead of System.Text.Json gives very similar results.

For what it's worth, here's the Chrome performance graph. The green is the download and the orange is "perform microtasks", which I assume means WebAssembly work.

[szalapski]

See also comments at https://stackoverflow.com/questions/63254162

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mkArtakMSFT]

Thanks for contacting us.
There has been a big push to optimize this further. You can learn more about the improvements https://github.com/dotnet/runtime/discussions/40318

[mkArtakMSFT]

@steveharter FYI

[szalapski]

Thanks. The performance is so poor that I am still skeptical that this is just a slow area--I still suspect that something is wrong with the way I am doing it. Would a deserialization of a few megabytes take 10-30 s?

[szalapski]

needs tag area-System.Text.Json

[lewing]

Which version of Blazor are you using? You'll want to avoid creating a string from the content and use a Stream instead. If you are using the net5.0 you should look at the System.Net.Http.Json extensions.

[szalapski]

I'm using Blazor 3.2.0 with System.Text.Json 5.0.0-preview.7.

Yes, I used the extensions, but when I saw they were slow, I refactored to the code above so I could narrow the issue down to serialization. Here's the code before my performance refactoring.

C# private async Task GetData() { IsLoading = true; string url = $".../api/v1/Foo"; Results = await clientFactory.CreateClient("MyNamedClient").GetFromJsonAsync<List<Foo>>(url); IsLoading = false; }

[steveharter]

You'll want to avoid creating a string from the content and use a Stream instead.

Yes this will allow the deserializer to start before all of the data is read from the Stream and prevent the string alloc.

System.Text.Json should be ~2x faster for deserialization than Newtonsoft so it would be good to see your object model to see if you hit an area that is slow on STJ.

In either case, since both Newtonsoft and STJ are slow there is likely something else going on.

The Large object graph benchmark section in https://github.com/dotnet/runtime/discussions/40318 has deserialization perf of 372ms for a string of length 322K. This also includes a "polymorphic" mode due to using System.Object that causes deserialization to be much slower (almost 2x) than without it. Anyway, extrapolating 332K to your 1MB is a 3x factor, so I assume it would take about 372ms * 3 = ~1.1 seconds to deserialize (on my fast desktop in isolation).

Some thoughts:

• Can you share your hardware specs? Memory\CPU
• Is the test running in isolation on dedicated hardware or is it hosted?
• Can you share your object model (your Foo type from the link)?
• Is there also rendering going on (or other CPU tasks) that would affect perf significantly?
• Change to Async\Stream mode as mentioned earlier.

[HenkHolterman]

I posted an MCVE as answer on StackOverflow, based on the WeatherForecast page.
My results are much better: < 100ms download and < 4 seconds for deserialize.

See https://stackoverflow.com/q/63254162/

[szalapski]

@steveharter ,

Yes, Intel Core i5 8350-U with 16 GB RAM. The test is running on my laptop. Foo is actually as follows, with minimal name changes only to protect the proprietary. Nothing significant on the CPU, this is my only focus when I am doing this.

I did start with a stream per the code just above --that was how I found this issue. I refactored to the code in the issue post just to narrow it down to slowness in deserialization.

```C#
public class Foo
{
public int? FundId { get; set; }
public int? FiscalYear { get; set; }
public string? FundTypeCode { get; set; }
public string? CategoryDescription { get; set; }
public string? CustomerLevelCode { get; set; }
public string? CustomerCode { get; set; }
public string? CustomerDescription { get; set; }
public string? ChangedFieldName { get; set; }
public decimal OriginalAmount { get; set; }
public decimal NewAmount { get; set; }
public string? Comment { get; set; }
public string? CreateUser { get; set; }
public DateTime CreateDate { get; set; }
public bool AdjustedBySystem { get; set; }
public Guid? ChangeBatchNumber { get; set; }
public string? FundDescription { get; set; }
public string? ParentCustomerCode { get; set; }
public string? ParentCustomerLevel { get; set; }
public string? ParentCustomerDescription { get; set; }
public decimal ChangedAmount { get { return NewAmount - OriginalAmount; } }
public bool? CreatedFromNFile { get; set; }
public string? Reason => (FundTypeCode == "N" ? (CreatedFromNfdaFile == true ? "From N File" : "N Manual") : "")}

}

```

[steveharter]

That model is simple and should be fast (no System.Object, non-generic collections or custom converters that could slow it down).

I suggest running the perf test that @HenkHolterman suggested in stackoverflow to compare against the baseline.

[szalapski]

@steveharter , I tried it just as suggested. It indeed takes 7-12 seconds to return 17000 items (about 1.6 MB) of WeatherForecast. (Download time on localhost is about 20 ms.) using the default code, await Http.GetFromJsonAsync<WeatherForecast[]>("WeatherForecast"); So this seems consistent with the timings on my slightly more complex case in the original question. (FYI, this is on Blazor 3.2.0; I also updated System.Text.Json via NuGet to v 5.0.0-preview.7, but it didn't help much.)

(also, I tried to increase the payload to 5 MB and that took 23-27 seconds.)

[lewing]

Blazor in net5 should be considerably faster. Are you running this test from inside VS or from a published build?

[msftbot[bot]]

Tagging subscribers to this area: @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

[szalapski]

Running it from Visual Studio, "run without debugging" in Release configuration.

@lewing Do you mean just System.Text.Json should be faster? If so, I already have the latest preview.

Or are you suggesting I move the app to Blazor 5.0.0 latest preview?

[lewing]

@szalapski could you please try your timings with the a published app outside of VS? It looks like there is an issue where the runtime is always initialized in debug mode when run from inside VS.

Additionally if you update the app from 3.2 to 5.0 there are several interpreter optimizations and library improvements. Some are in preview7 some are in later builds.

[szalapski]

Just tried outside of VS -- using dotnet run at the command line in Windows. No problems, very similar timings.

Also tried running the .exe after running dotnet publish --configuration Release, just to be sure. (I think this should be virtually the same as running dotnet run, right?) The timings were again similar.

[marek-safar]

@lewing what are the next steps here?

[steveharter]

what are the next steps here?

I assume no attempt to run on Blazor 5.0 yet? If so I think that should be next.

Both Newtonsoft and STJ are slow. This indicates a likely environmental or systemic issue, and not likely a (de)serialization issue.

The StackOverflow test runs <4 seconds for @HenkHolterman and 7-12 seconds for @szalapski. Different hardware and\or different Blazor versions could account for that 2x-3x slowness; would need a standard CPU benchmark and same Blazor version to actually compare apples-to-apples.

Also @szalapski on download perf you originally said:

My download of 2-6 MB takes 1-6 seconds

but with your latest test from StackFlow you said:

It indeed takes 7-12 seconds to return 17000 items (about 1.6 MB) of WeatherForecast. (Download time on localhost is about 20 ms.)

So download time went from 1-6 seconds for 2-6MB to 20ms for 1.6MB -- any thought on why that's the case?

[szalapski]

The 1-6 seconds was over the internet, whereas the 20ms was running against a local web service. I just did that comparison to ensure that the download speed is not relevant--regardless of whether the download is 20 ms or 20,000 ms, the deserialization is quite slow.

I will try it on Blazor 5 preview 8 soon.

"The StackOverflow test runs <4 seconds for @HenkHolterman and 7-12 seconds for @szalapski. Different hardware and\or different Blazor versions could account for that 2x-3x slowness; would need a standard CPU benchmark and same Blazor version to actually compare apples-to-apples."

Why shouldn't it be on the order of tens of milliseconds? Are the optimizations we see in .NET Core just not possible in WebAssembly?

"Both Newtonsoft and STJ are slow. This indicates a likely environmental or systemic issue, and not likely a (de)serialization issue."

Wait, I thought all agreed that the slowness is in the deserialization code, not in a problem with my system or environment. You are saying that I have a problem that is not inherent to deserializing in WebAssembly? How can I diagnose that?

[tareqimbasher]

@szalapski I can confirm without a doubt that the slowness is with the deserialization and not a system or environment issue. We are developing a blazor wasm application and deserializing a ~1.8 MB JSON payload takes about 5-6 seconds (time to complete network request is not part of that time). We have a 20 member team and everyone from developers to business experiences this slowness of deserialization so I can state from experience that it is not a system or environment issue.

Interested to see how your times would change when you try it on Blazor 5 preview 8!

[steveharter]

The 1-6 seconds was over the internet, whereas the 20ms was running against a local web service. I just did that comparison to ensure that the download speed is not relevant--regardless of whether the download is 20 ms or 20,000 ms, the deserialization is quite slow.
I will try it on Blazor 5 preview 8 soon.

@szalapski OK thanks for clarifying on the download speed. Hopefully you will see a large improvement on Blazor 5.

@tareqimbasher are you running on Blazor 5?

As discussed in https://github.com/dotnet/runtime/discussions/40318, due to current Blazor architecture, there is an expected perf hit that has a wide range depending on the exact scenarios, but ~35x slower than Core is a rough number that is in line with expectations.

However, there are a couple areas known to be slow that could be made faster in the serializer. These include large strings (say > 1K) and using System.Object or a non-generic collection such as IList (where elements are System.Object) as a property type.

[szalapski]

I see total time including serialization to get thousands of weather forecast lines cut in half when using .NET 5.0.0-rc1 in release configuration. It took around 13 seconds to get 53,000 weather forecasts in v 3.1 but 7 seconds in 5.0.0-rc1. Good improvement.

My controller in this example is returning an IEnumerable<WeatherForecast> which is just a WeatherForecast[] (ordinary array) underneath. The client deserializes that using HttpClient.GetFromJsonAsync<WeatherForecast[]>(string). Is there any certain types or techniques that could speed this up? I'm already avoiding non-generic lists and objects that are of type System.Object. Any other tips? I don't imagine there's any difference between using array and List, etc. but thought I'd ask--anything else we could do to "help" the deserializer along for collections with thousands of items?

I hope still that it can start approaching the performance of .NET Core.

[akoeplinger]

Doesn't solve the issue but from https://twitter.com/JamesNK/status/1310875638585204738 it looks like gRPC is a lot faster to deserialize:

I wrote a Blazor WebAssembly app that shows the performance benefits of gRPC-Web compared to JSON.

Both are fast with small payloads. With large data sets:
• gRPC network usage is 70% smaller
• gRPC deserialization is 10 times faster

Check it out here: https://grpcblazorperf.azurewebsites.net

[szalapski]

I will consider gRPC but it's not my preferred way to fix this. Thanks, though.

[szalapski]

We're finding ways to manage things, but it does seem like there ought to be a way to get 50,000 small objects deserialized in a second or two. I recommend setting a reasonable goal for the next release. :)

[saulyt]

I've been having similar issues. I've found Utf8Json to be much faster than both Newtonsoft and System.Text.Json. I am using a PipeReader and Utf8Json is still faster even though I have to copy the bytes to an array to deserialize while with STJ it can be read directly.

I am having issues with other things being slow as well, and I suspect this issue not strictly related to deserialization. Perhaps it is an issue with the way memory is accessed. For example, when I try to create an Excel file using EPPlus, ClosedXML, or similar APIs (I tried a bunch), it takes well over a minute for a 2MB file. Running the same exact code on Blazor server produces the file in about a second.