Httpie: Non-conformant Accept header

[vfaronov]

With the recent fix for #470, HTTPie sends:

Accept: application/json, */*

This header is not quite correct. According to RFC 7231 § 5.3.2, it says: “I accept JSON or anything else”. Because there are no explicit quality values (q=) specified for either media range, they are both assumed to be 1, so application/json and */* are equally acceptable.

*/* should be marked with a lower quality value, for example:

Accept: application/json, */*;q=0.5

[jakubroztocil]

@vfaronov thanks for the report. I considered including the q weight qualifier but then decided against it because it's optional and specific media (application/json) have automatically higher priority than ranges (*/*):

Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence. For example,

Accept: text/*, text/plain, text/plain;format=flowed, */*

have the following precedence:

1. text/plain;format=flowed
2. text/plain
3. text/*
4. */*

— https://tools.ietf.org/html/rfc7231#section-5.3.2

[vfaronov]

@jkbrzt The text you quoted defines “precedence” for purposes of the text that follows:

The media type quality factor associated with a given type is determined by finding the media range with the highest precedence that matches the type.

Thus, given:

Accept: application/json, */*

• if we consider the application/json media type, the _media range with the highest precedence that matches the type_ is application/json, and its quality value is 1;
• if we consider the text/xml media type, the _media range with the highest precedence that matches the type_ is */*, and its quality value is also 1.

In other words, this “precedence” only tells you how to match a given type to one of the quality values specified in Accept, but does not influence the quality values themselves.

That is my reading of the spec.

[bpinto]

I would also like to ask --json to send Accept: application/json only.

From the documentation:

--json, -j
(default) Data items from the command line are serialized as a JSON object.
The Content-Type and Accept headers are set to application/json
(if not specified).

With the change on #470, it's no longer possible to force Accept: application/json using config file unless we use sessions.

[reschke]

Including */* with a default q value is indeed meaningless. Note that there is no server requirement to honor the Accept request header field, so a server doesn't need to return 406 if it doesn't have an application/json representation.

[kjmph]

I've been down this path as well, @vfaronov is representing a common interpretation. That is, the specificity affects the media type, so a more specific subtype sets precedence over a similarly typed media range. However, once the specificity resolves media types, they are ordered according to quality scores. I believe that this part of the RFC is ambiguous to readers.

In this interpretation, when a server has an internal preference of serving text/html over application/json, httpie, in JSON mode, is telling the server it has no preference of which media type to respond with (because both media type quality scores are 1.0). This causes non-obvious media type choices when using httpie. Perhaps it helps to see what some browsers set in their Accept headers:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values

In that link, Internet Explorer and Edge are the only two browsers that don't specify a quality score for */* media ranges.

You can see this in the Apache documents:

https://httpd.apache.org/docs/trunk/content-negotiation.html#better

Apache understands the intention of clients to have */* be a lower quality score, and thus set that internally to emulate the desired behavior. However, there are servers in the wild that do not follow this interpretation. Thus, it is desirable for httpie to be explicit, and set a quality score that is lower than 1.0 for the */* wildcard when operating in JSON mode.

[inkstak]

Another example.

Sending a request with such Accept header to a Rails app, which is serving browsers & API, will always return an HTML response :

https://github.com/rails/rails/issues/8987

As @kjmph said, it's because some browsers don't set quality score correctly.

We can assume this behavior should be quite common to many other web servers

[inkstak]

Hi @jakubroztocil, did you take a look on last comments ? Is it possible to reconsider your position on this problem ?

[kjmph]

Thank you!