Mvc: Add support for facets and structured syntax suffix to MediaType

[khellang]

It would be nice if MediaType had support for parsing faceted subtypes and structured syntax suffixes:

Additionally, I think the default output formatters should support +json and +xml suffixes out of the box.

See RFC6838 for more info.

[danroth27]

@khellang Thanks for this suggestion! It would be great if you could share with us any specific use cases you have for facets and structured syntax suffixes. We're trying to get a better understanding of how these features would get used.

[rynowak]

/cc @tuespetre

[tuespetre]

Suffix support would be the 💣; I would be interested to know what use someone could come up with for the 'facets', though, since I can't think of any.

One thing I would like to see also is some more support around media type parameters -- specifically things like version parameters and being able to specify how they are considered during negotiation (e.g. 'can we write this media type for any given value of this parameter?')

Would it be unfair to heap that onto this same issue? :mountain:

[rynowak]

@tuespetre - can you expand on what you mean by 'suffix support'? In your ideal world what would the framework provide and how would you program it?

Is this as simple as us allowing any application/*+json in the Json formatters?

One thing I would like to see also is some more support around media type parameters -- specifically things like version parameters and being able to specify how they are considered during negotiation (e.g. 'can we write this media type for any given value of this parameter?')

I think this will always be a 'build it yourself' kind of thing. We can't write this in a generic way that I can see, and we'd much rather provide building blocks here than say, writing a rule engine.

You can do this today by overriding CanWrite/CanRead, the question is how hard is that code to write?

[khellang]

I would be interested to know what use someone could come up with for the 'facets', though, since I can't think of any.

@tuespetre You just mentioned one; versioning :wink:

It's pretty common to use faceted sub-types for versioning, i.e. application/vnd.github.v3+json or application/vnd.github.v3.raw+json. You can read more about GitHub's media types and how they use facets here.

@danroth The reason they both got lumped into the same issue (when they probably should've been two) is that I was implementing a small library for API versioning based on different strategies.

You can find the library here if anyone wants to take a look.

Some of these strategies are based off the Accept header, either using facets or a parameter (typically version). Today, you only have a split between type and sub-type. This means that Accept: application/vnd.github.v3+json would return vnd.github.v3+json as a string for the SubType property. As you can see, there's still some info to be pulled out of that sub-type. That's where the suffix and facets come in. In this case, I'm not even interested in the suffix, so I'd :heart: if sub-type had an indexed list of facets and a suffix property as well. That would make it really convenient to pull out this info.

[khellang]

Fat fingers typing on the phone :stuck_out_tongue:

Anyway. The most pressing issue here, that I can see, is that the built-in conneg doesn't support suffixes, so vnd.github.v3+json would not be supported by the default JSON output formatter.

@tuespetre Re: version parameter and negotiation. Have a look at that lib I linked. Something like that?

[danroth27]

@Tratcher

[Tratcher]

So the first thing you want is MediaTypeHeaderValue to parse things down further for you? That should be possible. Move that part of the bug to HttpAbstractions.

[khellang]

@Tratcher Yes, that's basically what I want. But I don't necessarily want this for MTHV; MVC has its own MediaType type, used for conneg. It does more or less the same as MTHV. The question is; where to put this? I can still move part of the issue though.

[tuespetre]

@rynowak:

@tuespetre - can you expand on what you mean by 'suffix support'? In your ideal world what would the framework provide and how would you program it?

Is this as simple as us allowing any application/*+json in the Json formatters?

I think it is that simple, but it's also already trivial to add an item to the SupportedMediaTypes list. So I guess I don't know exactly how much value it would be adding.

I think this will always be a 'build it yourself' kind of thing. We can't write this in a generic way that I can see, and we'd much rather provide building blocks here than say, writing a rule engine.

You can do this today by overriding CanWrite/CanRead, the question is how hard is that code to write?

I have to agree with you here, no rule engines please 😸

What I found with overriding CanWriteResult/CanWriteType/CanRead is that you have to write an IConfigureOptions<MvcOptions>, where you carefully construct your inherited formatters (which require a handful or more of odd-and-end dependencies to be passed into a base constructor) and supply them to MvcOptions. Oh, and don't forget to add services.AddSingleton<IConfigureOptions<MvcOptions>,MyConfigureMvcOptions>() in ConfigureServices. 😉

It's just an obscure process. Personally, I think I could be happy if the built-in formatters offered another 'soft extension point', kind of like SupportedMediaTypes, except something like a list of MediaTypeMatchers. They could even just be delegates -- for instance, the InputFormatter ultimately uses IsSubsetOfAnySupportedContentType, which consults SupportedMediaTypes. The OutputFormatter has something similar in CanWriteResult except it is currently inlined. Those default behaviors could be included as default MediaTypeMatchers entries, and developers could add their own when they configure the MvcOptions.

@khellang:

Oh, yes, versioning with facets. I don't like it but at the same time GitHub is one of the APIs that I hold in most high regards.

But I don't necessarily want this for MTHV; MVC has its own MediaType type, used for conneg.

👍

[khellang]

I think it is that simple, but it's also already trivial to add an item to the SupportedMediaTypes list. So I guess I don't know exactly how much value it would be adding.

Yes, it's trivial to add an item to the SupportedMediaTypes, but that doesn't mean structured syntax suffixes are supported. You can have any application/*+json media type that should be supported by the formatter, so there's no way to add them all to the list :smile:

[tuespetre]

There's the "exactly how much value it would be adding" bit we needed! 😄

[rynowak]

Yes, it's trivial to add an item to the SupportedMediaTypes, but that doesn't mean structured syntax suffixes are supported. You can have any application/*+json media type that should be supported by the formatter, so there's no way to add them all to the list

Yep. That's what I meant, there's no ability to do wildcarding today with the current code. I guess what I'm asking is would you use wildcarding or would you add specific media types? both?

[khellang]

I guess what I'm asking is would you use wildcarding or would you add specific media types? both?

I'd expect the formatter to work for all SupportedMediaTypes AND supported structured syntax suffixes, i.e. +json.

[tuespetre]

I'd expect the formatter to work for all SupportedMediaTypes AND supported structured syntax suffixes, i.e. +json.

public interface IMediaTypeMatcher
{
    bool Match(MediaType mediaType);
}

public class JsonSuffixMediaTypeMatcher : IMediaTypeMatcher
{
    public bool Match(MediaType mediaType)
    {
        return mediaType.Suffix == "json";
    }
}

services.Configure<MvcOptions>(mvc =>
{
    var inputFormatter = mvc.InputFormatters.OfType<JsonInputFormatter>().First();

    inputFormatter.MediaTypeMatchers.Add(new JsonSuffixMediaTypeMatcher());
});

[khellang]

Definitely more open/closed, but might be a bit overkill? Can you think of a 3rd way of matching the media type? :stuck_out_tongue_closed_eyes:

[tuespetre]

Can you think of a 3rd way of matching the media type?

I was thinking 'parameters', but after reviewing the notes I took the last time I went through all of these APIs and reviewing how we've used custom media types, I don't think that really applies -- or at least, it wouldn't be a good idea, since suffixes _should_ be used instead of parameters to negotiate the format anyways.

The reason I was thinking of parameters was that I had to register all of the combinations of them in the SupportedMediaTypes for the OutputFormatter, and that was because the OutputFormatter only wants to write formats that are equal to or more specific than the content type for a given object result -- in other words, you can't just add application/vnd.us.ourapp+json to SupportedMediaTypes and have it work with [Produces("application/vnd.us.ourapp+json; param=value")]. With suffix support, that problem would be completely taken care of and all other dealings with parameters could be handled with [Consumes] and [Produces].

So, for your first question:

might be a bit overkill?

Nnnyes, quite. Maybe the suffixes should just be baked into their respective formatters? 🎱

[lvermeulen]

I'm not sure whether any conclusions have been reached on this issue, but as I mentioned in #5143 the added value is correct rfc support and ootb support for all iana registered media types. I believe support can be as simple as subType.Equals("xyz") || subType.EndsWith("+"+"xyz"), no?

EDIT: thumbed myself up :-)

[SteveSandersonMS]

We've had some discussions and are ready to proceed with implementing this. Hopefully we will cover most of the functionality requested here.

The idea is:

1. Out of the box, the JSON/XML formatters would support parsing requests of the form Content-Type: application/*+json (with any parameters), and would support formatting responses when your ObjectResult's ContentTypes or Response.ContentType value were of the form application/*+json (with any parameters).
2. We'll extend the MediaType API as @khellang suggested, adding properties SubtypeSuffix and SubtypeWithoutSuffix.
3. To enable support for matching custom subtype suffixes, the InputFormatter base class will allow registration of wildcard patterns such as */*+mysuffix. Similarly, the OutputFormatter base class will get an extra property, SupportedMediaTypePatterns, where you can register wildcard patterns such as */*+mysuffix. Like before, you can optionally specify parameters on those patterns if you want (e.g., application/*+haiku;v=3) - the absence of a parameter is understood to mean "allow any value or none" like it does today.

Note: When (3) is implemented, we can implement feature (1) trivially just by adding extra default config entries for the built-in formatters. You can even remove feature (1) by changing the config back in your app if you want.

Note: The incoming Accept header would continue to be used only to filter the set of candidate formatters, and does not automatically get copied to the response Content-Type by default. For example, if the request specifies Accept: application/vnd.blah+json;v=3, then we still don't set the Content-Type to anything custom like that unless it's also specified by a [Produces] attribute or on the ObjectResult or Result objects. This is for security (clients shouldn't be able to get arbitrary strings into response content-types).

Now, as for your specific requests:

It would be nice if MediaType had support for parsing faceted subtypes and structured syntax suffixes

We'll parse the subtypes; you can string-split it further to get facets if you want.

Additionally, I think the default output formatters should support +json and +xml suffixes out of the box.
...
The most pressing issue here, that I can see, is that the built-in conneg doesn't support suffixes, so vnd.github.v3+json would not be supported by the default JSON output formatter.
...
You can have any application/*+json media type that should be supported by the formatter, so there's no way to add them all to the list

Covered by (1)

Personally, I think I could be happy if the built-in formatters offered another 'soft extension point', kind of like SupportedMediaTypes, except something like a list of MediaTypeMatchers

We already have a concept of wildcards. Extending this to support matching suffixes hopefully covers everything we need, so we don't have to introduce a whole new concept.

The reason I was thinking of parameters was that I had to register all of the combinations of them in the SupportedMediaTypes for the OutputFormatter, and that was because the OutputFormatter only wants to write formats that are equal to or more specific than the content type for a given object result

Covered by (3) - you'll be able to add one pattern that works with all [Produces] attributes that specifiy something more specific (e.g., extra params).

[khellang]

Awesome 👍 Thanks for the writeup @SteveSandersonMS 😄

the absence of a parameter is understood to mean "allow any value or none" like it does today.

This is not the case for output formatters today, though? See #5815.

[SteveSandersonMS]

the absence of a parameter is understood to mean "allow any value or none"
...
This is not the case for output formatters today, though?

I meant that's how the IsSubsetOf logic is defined, but you are right that there's still a difference between A.IsSubsetOf(B) and B.IsSubsetOf(A). Today, we check whether outputFormatterSupportedType.IsSubsetOf(proposedResponseType), which means that - like you described in #5815 - the outputformatter won't be used if proposedReponseType has extra params. Whereas it would be used if outputFormatterSupportedType has extra params.

I can see arguments for needing the IsSubsetOf rule for output formatters going in either direction, depending on the scenario. That's why I'm proposing both SupportedMediaTypes and SupportedMediaTypePatterns, so you get to control the direction. Update: Rather than two collections, we might just trigger the matching direction based on the presence/absence of a * in your string.

However, this aspect of the design is still not sealed. If we didn't care about breaking changes, I'd personally prefer to reverse the subsetting direction and say that's all we support (as in, an output formatter that supports application/mything would get used for all media types like application/mything;v=1, ...;v=2, etc., and then that output formatter's own logic would be responsible for varying its output as needed according to the version). But that would be a breaking change.

[tuespetre]

I'd personally prefer to reverse the subsetting direction ... But that would be a breaking change.

Yeah... 😢

Update: Rather than two collections, we might just trigger the matching direction based on the presence/absence of a * in your string.

Now THAT is clever (the good clever!) Does that mean something like application/vnd.woohoo+json;some-param=* would match application/vnd.woohoo+json;some-param=some-value and application/vnd.woohoo+json for output formatters by default?

[SteveSandersonMS]

Does that mean something like application/vnd.woohoo+json;some-param=* would match application/vnd.woohoo+json;some-param=some-value and application/vnd.woohoo+json for output formatters by default?

We haven't concluded on the exact syntax, but yes, you would be able to achieve your desired behaviour (matching arbitrary param values on output formatters) with whatever syntax or API shape we finalise on.

[SteveSandersonMS]

Implemented in #6032