Fetch: Proposal: Support GET bodies

[bcherny]

More and more, applications are built as thick clients sending complex queries to a thin server (see this). Many of those queries are requests for data, and are free from side effects. A few projects that do this (ie. support arbitrarily large browser queries to retrieve data) include:

• ElasticSearch (query API uses GET with body)
• Facebook's GraphQL (query syntax is a nested JSON structure)
• Netflix's Falcor (query syntax is a JSON structure)
• Meteor's DDP (queries support EJSON params)

RFC 7231 allows GET bodies (spec), but Chromium's XMLHttpRequest implementation does not (source). This prevents us from GETting data that requires an arbitrarily large query to fetch.

There are work arounds, each with their own drawbacks:

• Use a GET, and serialize the query into the query string (subject to max-length restrictions on URIs, not human-readable)
• Use a GET, and serialize the query into a header (semantically incorrect, often subject to practical server-side limits)
• Use a POST (semantically incorrect, imposes restrictions on browser-level caching)
• Use a custom HTTP method (doesn't play well with application/network middleware)
• Use websockets with a custom subprotocol (difficult to implement in a RESTful way)

It seems to me that the best, most backward-compatible solution is to permit bodies for GET requests with the fetch API.

What does everyone think?

(This is continuing the discussions I had with @albertywu, @pathikrit, and @igrigorik)

[annevk]

It seems this restriction was added here for GET, and later HEAD:

• http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.src.html.diff?r1=1.165;r2=1.166;f=h
• http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.src.html.diff?r1=1.182;r2=1.183;f=h

Based on this thread:

• https://lists.w3.org/Archives/Public/public-webapi/2007Dec/thread.html#msg8

Concerns are networking libraries used by user agents, unsuspecting servers, RFC conformance (now clarified), ... Convincing those skeptical in that thread that this is now a good idea would be a good start.

[wanderview]

These examples are somewhat compelling, though. These are very established APIs.

In particular, requiring POST for use with these APIs also prevents the Request from being stored in Cache API. Maybe we should loosen the POST constrain on Cache if fetch won't change to all body on GET.

What do you think @jakearchibald?

[jakearchibald]

I'm not against allowing bodied requests into the cache, although wouldn't the body be wasted space? I guess you'd need to it replay the request at a later date to update the cache.

[wanderview]

I think it might have to take part in the matching algorithm. If you want to cache elastic search results you would probably want each query to be a separate cache entry. It seems to me the URL and headers will be identical and only the body will differ.

@bcherny How is this sort of thing handled with http caching semantics today? Do you just cache your last query or is there something different in the headers the http cache can key on?

[bcherny]

@wanderview I agree, the pkey for a bodied GET should be its URL-headers-body combination.

In the applications I've worked on, we issued bodied POSTs and used an app-layer cache keyed on a hash of the URL-headers-body combination.

[wanderview]

Right, but I guess my point is that you don't get the right caching semantics for this API even if you could use GET because the http cache is not going to look at the body. So I'm not sure its worth trying to support GET with a body or POST in cache. It does seem like this needs special app layer caching semantics.

[bcherny]

That makes sense - to properly handle GET w/ body caching, you'll need to either:

1. Add a checksum header, and cache the response, or
2. Set no-cache in the response, and cache in the app layer instead

For reference, the relevant spec is here: https://tools.ietf.org/html/rfc7234#section-4

[annevk]

To be clear, I agree that the examples are compelling and I'm saying to make progress here someone needs to convince the people in that referenced thread. This is not a high priority for me, so if you want to drive this, please do contact them.

[mnot]

To give some context - RFC7230 allows _all_ request methods to have bodies, because we want generic parsers; they shouldn't have to have knowledge of a specific method's semantics to parse a message.

The relevant text in 7231:

A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.

... and indeed this is the case; most HTTP APIs (client and server side) don't provide access to GET bodies, and many firewalls, etc. will block requests with them. It's also not uncommon for GET bodies to outright cause bugs, because most software doesn't expect them.

So, HTTP doesn't "allow" GETs to have bodies in the sense that it's recommending it as a good practice; it's just an artefact of how HTTP parsing is specified.

We're having a discussion of this issue in HTTP WG now; semantic Web folks want to be able to perform complex queries as well, and the current proposal is to use a new method. See:
http://tools.ietf.org/html/draft-snell-search-method
That isn't going forward yet, because there isn't a compelling argument against just using POST.

Besides that, note that there is no defined maximum length on URIs in HTTP; 7230 says:

Various ad hoc limitations on request-line length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support, at a minimum, request-line lengths of 8000 octets.

http://httpwg.github.io/specs/rfc7230.html#request.line

I.e., this could equally be addressed in fetch by requiring support for longer URLs.

Also, AFAICT, the only one of the links above actually using a body in a HTTP GET is the first one, which says:

Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well.

[annevk]

The upshot of a body over a URL is that you can use a wider range of a byte and therefore put more data in less space. Perhaps H2 compression takes some of this away though.

But yeah, given the complexity of introducing this now just using POST plus some custom logic for caching might make things a lot easier for everyone involved.

[bcherny]

@mnot SEARCH looks very promising! that would be a great alternative to bodied GETs!
@annevk there are downsides to using a POST, see my initial comment.

[annevk]

@bcherny the only downside seems to be about semantics. I'll happily violate some semantics in order to avoid introducing extra complexity to this system. If you're happy with SEARCH, that works for me too.

[annevk]

http://stackoverflow.com/a/35685945/23607 has another use case. Submitting some data with GETs for authentication in middleware.

[bcherny]

@annevk i don't think that issue is about auth, it's about querying a store. see the 1st use case in my original post.

[bcherny]

the only downside seems to be about semantics. I'll happily violate some semantics in order to avoid introducing extra complexity to this system. If you're happy with SEARCH, that works for me too.

@annevk semantics are ostensibly important for a specification, but more important is the ability to cache queries without resorting to app-level caches. as more of the web moves on to sending large queries from the browser, reliable, standardized, built-in caching is a necessity.

[mnot]

Semantics are also important for interop. If you put arguments on a GET body and send it through a proxy or gateway cache (e.g., a CDN, Varnish, etc.), you'll likely get the wrong response.

[stevenvachon]

Wanting this with near-desperation.

Perhaps use the name query or search so that GET and POST variables can both be sent at the same time.

[annevk]

I'm going to close this. If implementer interest materializes we can reconsider. I thought about this a bit and it seems that from a conservative security perspective we could only allow this same-origin, perhaps even then requiring client-side opt-in since currently we drop the body silently. Cross-origin gets harder as even with CORS unsuspecting servers have never been exposed and tested against this so we'd have to extend CORS. That just seems like too much complexity.

[pouncilt]

I too agree that GET should support Request bodies. This used to be supported back in 2012 to 2014 via AngularJS which I think used XMLHttpRequest; but now it isn't any more. You guys do know that there is a limit on how lone the URL can be and thus we can't use query parameters? Be cause of this limitation, we need to be able to pass data for a HTTP GET via the request body. Per the HTTP Status codes, the HTTP GET does not say you can't pass data via the Request body; so I don't know why this implementation is so limiting. Lastly there are a lot of other frameworks that interpret the specification this way too. Please reconsider and allow for HTTP GET to send Request body. It kind of stupid to have to do an HTTP POST when all we really want to do is HTTP GET. I guess it also comes down to semantics; as we don't want to do a POST unless we are actually save/changing state of some server resource. We want to be a HTTP GET when we want to retrieve resource/data from the server. We need to send Request body here because the query parameter may be too long for the URL.

At very least the, we should be able to opt into sending Request body with HTTP GET!

Thank you.

[SirWumpus]

Our micro-services application recently came up against this issue when the request URL blew the server's 8K limit. Having confirmed RFC 7231 does not disallow it, just that the semantics are undefined, then between private in-house client/server applications a GET-with-body should be permissible. When we went to implement a fix, we find that using a 3rd party nodejs module minipass-fetch explicitly denies it.

This issue is going to continue to return again and again while the RFC and the fetch API specification disagree. It clearly needs to be fixed.

[mnot]

FYI - https://github.com/httpwg/http-core/issues/202