Runtime: System.Net.Http.Json: Consider relaxing content type verification in ReadFromJsonAsync

[huoyaoyuan]

I'm doing interop with aria2. It's using application/json-rpc for its returning content type, which is refused by ReadFromJsonAsync.

Proposed solutions:

1. Add a boolean parameter to totally bypass content type checking.
2. Pass a string as the expected content type.
3. Widen the accepted content types to include it. I'm not very familiar to the specification of content type, and not sure whether StartsWith is abusing it.

Current workaround:

Copy the implementation and delete code doing verification.
This is opposed to the purpose of System.Net.Http.Json. They are mostly simple helpers. If there's some restriction makes it unusable, the whole type becomes no-sense.

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[scalablecory]

@Jozkee @layomia

[samsosa]

Yeah. There are so many weird servers that send weird content media types, in which case System.Net.Http.Json would be worthless.

The first option would solve all problems:

1. Add a boolean parameter to totally bypass content type checking.

[scalablecory]

This seems worth doing to me. At least to make it configurable.

[halter73]

Why would this need to be configurable? Couldn't you just check the Content-Type header yourself before calling ReadFromJsonAsync if you want content type checking? If the body's not actually Json, parsing would fail anyway.

[geoffkizer]

Do we enforce content type for other methods like ReadAsStringAsync?

Seems like we just shouldn't check Content-Type here at all.

[scalablecory]

Why would this need to be configurable? Couldn't you just check the Content-Type header yourself before calling ReadFromJsonAsync if you want content type checking? If the body's not actually Json, parsing would fail anyway.

Yep, I should have been more clear -- I think we should get rid of the check. And if there is a reason to care (@rynowak? @Jozkee?) then it should be configurable.

[Jozkee]

The main reason to have it was to have parity with the old APIs from System.Net.Http.Formatting
https://dotnetfiddle.net/ijQAdJ

Linking the original discussion for such check https://github.com/dotnet/runtime/pull/33459#discussion_r390700364.

If other ReadAs* methods does not have such validation I would agree on adding a parameter to bypass the validation or even remove it so people that is not very experienced in using the HTTP APIs don't get frustrated by this error (https://github.com/dotnet/aspnetcore/issues/21288).

cc @pranavkm @terrajobst

Keep in mind that these APIs are already available as a NuGet package that targets netcoreapp3.1 and netstandard2.0; if we decide to change behavior here would that be considered a breaking change?

[pranavkm]

System.Net.Http.Formatting performed content-negotation, hence it made sense for it to complain about an unknown media type.

With S.Net.Http.Json since you're explicitly asking to read JSON, there isn't much value in trying to validate this. What would help, is to include the media type as part of the error if deserializing fails and the content-type is unknown. This may help users self-diagnose a little better:

C# try { return DeserializeAsync(); } catch (JsonException ex) when (IsNotJsonContentType()) { throw new DivideByZeroException("It does not look like you're reading json. Here's your content-type {content-type}. ", ex); }

Keep in mind that these APIs are already available as a NuGet package that targets netcoreapp3.1 and netstandard2.0; if we decide to change behavior here would that be considered a breaking change?

The code would have previously thrown when it encountered an unsupported media type. It wouldn't be a breaking change if it was more accomodating.

[rynowak]

The original intent for the check (in this iteration of the library) is to make sure you get a reasonable error message when parsing fails. The really common case is when you're interacting with a website that returns an HTML page on a failure.

[terrajobst]

@pranavkm @rynowak

So what do you think the ideal change is? We could check the content type and throw a better exception for known content types that are common and unlikely to produce JSON, such as text/html. This would catch common issues resulting in parser errors while avoiding an overly aggressive content type enforcement.

[scalablecory]

I think we should not check content type, but if parser throws an exception, throw with a message like "Content of type '{0}' was not valid json...".

[pranavkm]

👍 with @scalablecory's suggestion. We want to give users good diagnostics when parsing fails rather than police the web.

[Jozkee]

I think we should not check content type, but if parser throws an exception, throw with a message like "Content of type '{0}' was not valid json...".

In that case this is my understanding of the change:

1. Remove MediaType-related validations.
2. Wrap JsonSerializer.DeserializeAsync on a try-catch.
3. Append the MediaType before rethrowing.