Runtime: Introduce System.Json as a Json.Net-compatible foundational library

[vermorel]

JSON has entered the global software ecosystem with a force that is as strong, arguably stronger, than XML was in its time. In the .NET ecosystem, for years, it has been striking that the framework was not supporting JSON in a built-in way. As a result, Newtonsoft.Json is now the No1 package on Nugget.

I suggest that the foundational libraries should include a System.Json library with an API compatible with Newtonsoft.Json.

Indeed, as it stands Json.NET represents a tax that every .NET developer need to pay: you have to discover first _why_ there is no built-in way to deal with JSON, then you have to discover that this one specific third party package is worthy to be trusted. I am not dismissing the quality of this excellent package, but this massive hit primarily demonstrates that JSON should be part of the foundational libraries of .NET.

Then, taking the longer view, I am worried - and I believe others should be - about the long term implications of having such a _narrow_ yet _mission-critical_ library developed and maintained separately. .NET Core can be a viable economic product to maintain over time even without the backing of Microsoft. However, Json.NET _alone_ is not economically viable, not even through commercial support, not only 10 years time scale anyway.

The project might or might no go the way ServiceStack went (another great project by the way). Also, the commit profile of this project, with the _first contributor having literally over 100x more commits than the second_, a scenario ala NDoc is not to be ruled out either. Presently, Newtonsoft is taking the path of building complementary products on top of Json.NET, which again is very good, yet should not be relied upon for the future of .NET.

Opening .NET to the community is great. However, the fundational libraries should cover the _very basic_ use cases. XML is covered, JSON should be as well. In particular, "outsourcing" narrow yet critical components is paving the way for upcoming messes ala left-pad.

Ideally, System.Json would be API-compatible with Newtonsoft.Json. There are multiple ways to achieve that. Starting a discussion on this precise matter between corefx representants and James Newton-King would be a start.

[svick]

JSON should be part of the foundational libraries of .NET.

Are you saying that the proposed System.Json should be part of NetStandard.Library? Wouldn't it be easier to just make Newtonsoft.Json part of it instead?

[vermorel]

Wouldn't it be easier to just make Newtonsoft.Json part of it instead?

Maybe. The package name feels inconsistent though. .NET developers - present and future - should not have to investigate packages one by one to be sure they can be trusted. Consistent naming is one of strong point of .NET compared to many alternative frameworks.

[Thaina]

@svick No, System.Json is actually way better basic and ease to use than NewtonSoft.Json

NewtonSoft.Json is good when you want to work with dynamic and arbitrary type serialization. But sometimes it just overkill and System.Json is plainly simple

Also I'm actually disappointed about the fact that NewtonSoft.Json.Linq does not even have JObject.Load(Stream stream). So, well, NewtonSoft.Json should drop it and use System.Json internally instead

[BoltR]

It's worth taking into account that while Newtonsoft's Json.NET is a very good library; It's fast, but it's not the fastest. As such, it is probably a matter of time before one of the 'make it go fast' guys in the Kestrel space identifies it as the bottleneck and wants to make it go faster. Kestrel is doing well in the plaintext benchmarks, but falls down a bit in comparison when it starts dealing with JSON.

With the amount of features that the library supports I don't think strapping Channels onto it and using Span<> everywhere is going to be a straight forward change. So I wouldn't be surprised if it's deemed less work to create yet another library (System.IO.Pipelines.Json ?). Though I very much doubt it would be as feature rich as Json.NET.

[agatlin]

@vermorel, You make excellent points, and I agree with them completely. I have been using Newtonsoft.Json and have been almost since the time it was first released. With every new update of the .NET framework, I always expect that Microsoft will have included some equivalent of Newtonsoft.Json. To my repeated surprise, they never do. As you have mentioned Newtonsoft.Json 1) fills a major gap in .Net performance, and 2) is, by far, the most popular NuGet package.

The Internet sits at the center of the entire computing industry. JSON has become the defacto method of transferring data payloads across that Internet. Therefore, a high-performance JSON library should be a key component of any Web development framework.

It doesn't matter to me so much whether Microsoft includes Newtonsoft.Json generally intact (but with updated namespaces) or if they roll their own equivalent. What matters is that they fix the deficiency that exists now.

[svick]

Therefore, a high-performance JSON library should be a key component of any Web development framework.

It is. Newtonsoft.Json serves as that library in ASP.NET Core.

What matters is that they fix the deficiency that exists now.

I'm unclear what exactly the deficiency is, according to you.

Namespace itself doesn't mean anything, I can publish a package like System.Nonsense (or Microsoft.Nonsense), but it doesn't mean it contains high-quality code approved by Microsoft.

Also consider all the issues that occur when there are two competing ways to do the same thing (confusion, bad interoperability, breaking existing code, etc.). What are the advantages that outweigh these significant disadvantages?

[Thaina]

it contains high-quality code approved by Microsoft.

I wish something as common as JSON should have a library with high-quality code approved by Microsoft. Like System.XML

Introducing System.Json (back) into corefx would not breaking anything of anyone already using Newtonsoft.Json because it has difference namespace so I don't know why two competing ways to do the same thing can leads to _bad interoperability_ and _breaking existing code_. Even confusion is not likely to be

Also as many was mentioned. Newtonsoft.Json is great library. But is not high-performance JSON library compared to the original System.Json

As I said. NewtonSoft.Json is good when we want to work with dynamic or arbitrary type serialization. But sometimes it just overkill and System.Json is plainly simple

System.Json is actually way better basic and ease to use than NewtonSoft.Json when we want to just manipulate JsonObject as a dictionary

[Thaina]

Also what System.Json actually contains can only be compared to just NewtonSoft.Json.Linq

But the thing is. NewtonSoft.Json should be extended to System.Json and System.Json itself should be the reliable library anywhere because JSON is so common nowaday. It too common to the point that it should not need nuget. If we don't need nuget for System.Text.Encoding or System.Net or System.Xml then System.Json should also be there

[svick]

@Thaina

I wish something as common as JSON should have a library with high-quality code approved by Microsoft. Like System.XML

Newtonsoft.Json is like that too. If the code wasn't high-quality and approved by MS, they wouldn't use it in their products, like ASP.NET Core.

Newtonsoft.Json is great library. But is not high-performance JSON library compared to the original System.Json.

You seem to want something different than other people here. They keep saying what they want is basically renamed Newtonsoft.Json.

Also, according to this benchmark, Newtonsoft.Json is significantly faster than System.Json.

I don't know why two competing ways to do the same thing can lead to bad interoperability […]

If one piece of code uses Newtonsoft.Json and another uses System.Json and I need to use both of them, I will have issues. That's why having one library that almost everyone uses is better.

I don't know why two competing ways to do the same thing can lead to […] breaking existing code

Some Microsoft libraries and frameworks already use Newtonsoft.Json. Are you saying System.Json should be added, but all those libraries should keep using Newtonsoft.Json? Yes, that wouldn't break existing code, but it would make System.Json much less appealing.

It too common to the point that it should not need nuget.

It needs NuGet exactly as much as ASP.NET Core does (e.g. on .Net Core 2.0, Newtonsoft.Json is included in the package store, along with ASP.NET Core and other 3rd party libraries like libuv, Serilog or Remotion.Linq).

[Thaina]

Some Microsoft libraries and frameworks already use Newtonsoft.Json. Are you saying System.Json should be added, but all those libraries should keep using Newtonsoft.Json? Yes, that wouldn't break existing code, but it would make System.Json much less appealing.

I don't see that would be breaking anything exist. Well, because I just want the original System.Json not renaming the NewtonSoft.Json. System.Json object class even have different name than NewtonSoft.Json

And I'm also against renaming NewtonSoft.Json. Even if it included it in corefx I would like it still having the creator name as namespace

It needs NuGet exactly as much as ASP.NET Core

ASP.NET core does not the only place we use C#. I use C# in unity (well, as well as dotnet core) and there can't use nuget. Many places does not adopt nuget system. So, again, I think something as common as System.Json should always exist as standard

And even NewtonSoft.Json would be faster. Common functionality should still exist. And the thing that make NewtonSoft.Json faster is about serialize and deserialize a class. That should be able to extended from System.Json or maybe not related to System.Json at all. Because the original System.Json has different functionality compare to NewtonSoft.Json

As I said, original System.Json could only be compared NewtonSoft.Json.Linq

Newtonsoft.Json is significantly faster than System.Json

I remembered that in that BenchMark they only profiling the serialization to class. Which I know NewtonSoft.Json is really great for that work. But what I mean is, the most simple way we use Json is as JsonObject and JsonArray without serialize or deserialize, just parsing and manipulate. System.Json is better and simpler

NewtonSoft.Json has the same kind of functionality as NewtonSoft.Json.Linq and that cannot use in unity. I can bring System.Json to use in unity because it lighter in almost all aspect

[Li-ReDBox]

As others have mentioned, JSON is one of the popular data formats, used more often than XML. Most of languages have native support of JSON. It only makes sense to include the support of it in Core.

[oliverjanik]

Json in std lib is a great idea. Adopting Json.net however, not so great. It has too much baggage from it's early decisions.

[Squirrelies]

Please add this into the core libraries.

[ygoe]

JSON.NET is way too big to be included in small helper tools. My app is 20k and JSON support adds 600+k to that. See the issue? I understand that .NET adds a lot more, but it's part of Windows so it's just there, like System.Xml. Right now I can have XML config files without any additional space costs. JSON config files are overly expensive to support.

I'm going to have to try out those hard-to-discover simpler JSON libraries for C#.

[oliverjanik]

Newtonsoft.Json is way too broken to be added, period. We need a new light and swift implementation.

[danmosemsft]

System.Json was made available for .NET Core does that satisfy the original request here?

[Li-ReDBox]

I have been using System.Json and find it is OK. The problem is, as @ygoe mentioned, if you want to use JSON config in you application, then JSON.NET is one dependency which is not good for certain applications. Of course, you can use XML but ...

[Thaina]

@danmosemsft I am very appreciate that System.Json was available in nuget but still it should just included in corefx to be standard like System.Xml

[danmosemsft]

OK, I'm going to close this as resolved then.

System.Json hasn't received much investment in years - I think it was a Silverlight library originally. We would take changes to improve it, as long as it remained compatible.

[danmosemsft]

I am very appreciate that System.Json was available in nuget but still it should just included in corefx to be standard like System.Xml

@Thaina the boundary of the core package is really chosen as a convenient package of the most commonly used types. It balances size vs usage. If a library is not included, it does not necessarily mean that it isn't fully supported.

[Thaina]

@danmosemsft Please reopen and close issue when we really have it in corefx. Just in nuget is not really enough. Or else we should separate System.Xml from corefx to nuget

[Thaina]

@danmosemsft It's dilemma that if you don't support System.Json as standard. We would have to select between System.Json and Json.NET and it will not become _commonly used_ when we need to deal with JSON

JSON is really common. It is standard file format right now. It just because System.Json was not included from the start we have an unnecessary fragment for dealing with json in the system for long times. That's problem all along. It should be promoted as soon as possible so everyone can adopt it eventually

I am fine with any library that could completely deal with Json string but it must be one that be a standard in the corefx

[danmosemsft]

Happy to reopen to continue discussion. Perhaps there is a misunderstanding. System.Json _is_ in CoreFX. It is simply not in the default references. The default references are essentially the most used types and their dependencies. It does not indicate what is more or less supported. Just that System.Json isn't getting used much right now. Does that help? Many things we ship aren't in the default referenced set.

For the separate issue of whether System.Json is good enough -- I see performance mentioned above --
we are certainly interested in feedback and contributions. It is true that we don't currently have plans to work on it, but we welcome feedback on that also. It might be clearer to open a separate more focused issue for that. It would help to have evidence there is lots of interest, and/or community members willing to drive/contribute.

[Thaina]

@danmosemsft Sorry for me using the wrong word

I don't really care much about active working on System.Json itself. What I concern is it should included in the base class library like System.Xml. And I think this is what this issue was really about _foundational library_.

It about that if it is standard class it would be used by default. Then every library and framework could use it. And then every library would be compatible with each other

Currently now it seriously frustrating because we don't have default Json library in BCL. Every third party library who deal with Json that try to be compatible with many platform would need to make their own Json type system and parser. Each of them are not compatible with each other. And the developer need to bridge them when using more than one third party. Not to mention we need to include Json.NET to use on our own too. And Json.NET also not compatible with any library

If just Json being standard they can making library and API for it

[oliverjanik]

Is it not true that Aspnetcore includes Newtonsoft.Json by default? So technically that is the blessed Json serializer for .net core?

[ygoe]

Just to let you know, I've found fastJSON which seems to support what I need, does it fast, and is small enough to include the source code directly in the application – no separate DLL. I haven't got much experience with it yet but it looks promising. Maybe it's helpful for others, too. The source code could use a little cleaning, VS shows me style warnings all over the place, but I could make two changes I needed easily.

[svick]

@Thaina

It about that if it is standard class it would be used by default. Then every library and framework could use it. And then every library would be compatible with each other

It's too late for that. JSON.NET is so popular that it's never going to go away. Applications that already use it are not going to be rewritten for no good reason. Libraries that already use it are not going introduce breaking changes without sufficient justification.

The only way a standard JSON library would become the one everyone uses is if it was significantly more convenient and had significantly better performance than the best already existing libraries. I doubt that's even possible and it's almost certainly not true for the current System.Json.

[oliverjanik]

@svick Existing apps do not need to be rewritten. All that needs to happen is we get a better and officially blessed json library that comes with the project templates. I don't think the current System.Json is that library.

Also, I think Json.NET is popular by necessity not by choice...

[Thaina]

@svick It never be late just sooner is better. It will be late just if there would be no new library in the world anymore

When the new library keep relying more and more on BCL json. Old library would just plug API that accept and return BCL json to cope with it. Old library don't need to replace core json they had written. Just have api to accept it is enough for compatible

Convenient is also the sum from many factor. Being in BCL is first and important point for convenient and that's is a chain reaction. When we could just using BCL for convenient and publish raw code or library without dependency it then convenient for transport code and sample. It will be a feedback loop that then everyone will do that and make everyone familiar with it. It will built the significantly convenientness eventually proportionally to usage of JSON

The only time when include any json library (I don't mind it be Json.NET) into BCL would not being convenient is only when everyone stop using JSON, or everyone stop using C#, or everyone stop creating new library

JSON.NET is so popular but it not dependable enough. It proven by the fact that every platform need to create their own json library and every major third party need to build their own json parser internally. We just want a very efficient parser in the BCL

Also, I think Json.NET is popular by necessity not by choice...

Agree

[stephentoub]

This post outlines our plans around JSON:
https://github.com/dotnet/announcements/issues/90