Serviceworker: Is fetch() before or after Content-Encoding

[annevk]

E.g. if I fetch something gzipped, what bytes should end up in the stream? XMLHttpRequest handles this automatically, should fetch()?

Existing browser architecture handles this pretty low-level. Are we prepared to change that? @bzbarsky @sicking @domenic @michael-nordman

[domenic]

My position is that fetch() should be a low-level API, exposing as close to the raw HTTP primitives as we can. In this way it would be similar to Node.js's http.get.

In practice this means that the raw body stream it returns would be just the raw bytes coming down the pipe (as array buffers). You wouldn't ungzip them or anything of the sort.

The asXXX() methods, on the other hand, are designed for convenience. asJSON() would be: raw array buffers -> decompressed array buffers -> UTF-8 decoded text -> parsed JS object.

This lower-level API provides more power for the user in a variety of ways:

• If they want to store the data compressed, or pipe the stream to another server (e.g. via a PUT call), they can just store/stream the raw bytes, without having to re-compress them after fetch decompressed them.
• If they want to opaquely store a response, or pass it off to someone else, they can just do so, without having to pay the unnecessary overhead of the browser decompressing in the middle.
• They can now get an accurate idea of the network progress, by looking at the count of total raw bytes that have come down. Whereas if the ungzipping happened before the user saw the bytes, they would have no way of gauging progress accurately, e.g. in the case where the beginning of the stream was highly compressible whereas the end wasn't.

It also allows us to _explain_ the ungzipping as something along the lines of: do the headers contain Content-Encoding: gzip? If so, .pipeThrough(new GunzipTransform()).

[domenic]

/cc @tyoshino

[wanderview]

I believe cache would prefer to store compressed bytes. That's an implementation detail, but would be easier if fetch did not automatically decompress.

I assume that since cache returns fetch Response objects, the answer here will also apply to data returned from cache. Correct?

[annevk]

@wanderview yeah.

[jakearchibald]

Keeping the stream low-level makes sense. Could have a .decode() method that returns a transform stream that does the stuff XHR would usually do.

[bzbarsky]

@domenic I seriously doubt anyone wants the actual raw bytes coming down the pipe, assuming by pipe you mean network socket. For example, as an HTTP client you don't actually want to think about whether chunked transfer-encoding was used for the transfer, just like you don't want to see the encrypted bytes when TLS is used, or whatever ends up going on the wire when SPDY/http2 is used (interleaving of responses!).

It does make sense to me to expose the data after undoing below-HTTP layers like TLS encoding and after undoing HTTP transfer-encodings but before undoing content-encodings. Note that this would include undoing gzip transfer-encoding if someone uses it (not that browsers support that in practice anyway).

It's funny, actually. Technically Content-Encoding modifications happen above the HTTP layer, so caches do in fact store the data before undoing Content-Encoding, whereas Transfer-Encoding happens on the HTTP layer and so caches will typically undo it before storing. But at the same time, the typical use of gzipped Content-Encoding on the web should have been use of Transfer-Encoding instead. The fact that browsers and hence servers screwed that up is sort of letting us act as if the compression has semantic meaning instead of being a transmission artefact.

In terms of browser implementation, not decompressing gzip Content-Encoding would be pretty trivial. We already have to do that in some cases.

[domenic]

@domenic I seriously doubt anyone wants the actual raw bytes coming down the pipe, assuming by pipe you mean network socket.

Good point, thanks for clarifying that. I definitely agree.

From a naive user perspective, fetch should be "HTTP level" and not "TCP level." And again from a naive user perspective, I think gzip is thought of as something you layer on top of HTTP, instead of part of it.

Of course as you note it's all more complicated than that, partially as a historical accident, and partly because categories like "HTTP level" and "TCP level" are not very good compared to what's actually going on. But I think it's still helpful to see it through that lens.

[annevk]

So what should asArrayBuffer and asBlob do?

And what exactly do we want to consider part of "network and cache fetch" and what should be handled above it?

[tyoshino]

I support Boris's comment basically. Basic mode should unwrap any HTTP or lower layer things as XHR does now.

We could have some special mode to bypass decompressor / decoder for well-known codecs (not sure but e.g. there could be some security consideration about exposing compressed data to apps) and give un-processed data to apps.

For cache optimization, if we have some identifier system with it we can represent each response body but hiding it's contents unless required to convert into ArrayBuffer, etc., implementation details may do optimized transfer/store. Kind of "lazy evaluation". Streams API could play this role.

(Though we might be bothered by error in delayed decoding ...)

[domenic]

asArrayBuffer and asBlob should be consistent with asJSON and asString, I think, and should thus include decompression.

[bzbarsky]

I'm not sure that @tyoshino and I are talking about the same exact thing...

One more related comment. If API consumers are going to be doing decompression themselves, how do we want to handle the common Apache misconfiguration where a foo.gz file is sent as Content-Type: application/x-gzip and also Content-Encoding: gzip? This would generally fail, since after decoding the claimed content-encoding you no longer have gzip content, so handling per the Content-Type doesn't work. Browsers in practice have workarounds for this; e.g. Gecko will actually drop the Content-Encoding header from the response entirely in this situation. See http://hg.mozilla.org/mozilla-central/file/366b5c0c02d3/netwerk/protocol/http/nsHttpChannel.cpp#l4042

If API consumers are going to be doing decompression and such themselves, should the UA still perform this sort of under-the-hood fixup? Or should we expose the probably-bogus response and let consumers deal as they see fit? What do other browsers do with such responses?

[domenic]

If API consumers are going to be doing decompression and such themselves, should the UA still perform this sort of under-the-hood fixup? Or should we expose the probably-bogus response and let consumers deal as they see fit? What do other browsers do with such responses?

I'm not as aware of the practicalities and real-world constraints as you guys, but from my perspective the asWhatever methods can do as many workarounds and "do what I mean" fixes as are necessary, whereas the actual stream itself should be "raw," and consumers who wish to deal with such cases will handle them themselves.

Ideally you'd explain the asWhatever in terms of rawStream.pipeThrough(specifiedTransformStream). In this way such things can be exposed and specified, instead of hidden in nsHttpChannel.ccp. And you could even expose such a thing to JS, if you think users would use it.

WDYT?

[bzbarsky]

whereas the actual stream itself should be "raw,"

Note that Gecko doesn't change the _data_ in this case as a workaround. It changes the _headers_ that the HTTP library consumer sees. I guess that's already sort of visible via XHR, even, but it becomes more pressing in a fetch API world. There's tons of other header-munging that happens, by the way, that is not really covered by any specs...

[domenic]

Note that Gecko doesn't change the data in this case as a workaround. It changes the headers that the HTTP library consumer sees.

Right, I guess what I really meant was: asArrayBuffer could ungzip for non-bogus gzips, but leave the bogus-gzips alone. I.e., "do what I mean."

[bzbarsky]

Sure. The question about specifying the header munging still remains; the exact behavior of the workaround needs to be specced to the extent that it's consumer-detectible.

[tyoshino]

@bzbarsky The second and third paragraph in https://github.com/slightlyoff/ServiceWorker/issues/339#issuecomment-46837837 are just my rough ideas to benefit from passing (as much as possible)-raw data without allowing direct access to raw bytes at HTTP protocol layer.

Even after addressing weird things like your example about Apache, there should be still room of delaying uncompress/decode. E.g. asRawDataAsMuchAsPossible() may return {type: 'gzip', data: actualData}. actualData doesn't necessarily the original body bytes as-is. But we could make it by less processing.

Stream piping from FetchBody to Cache may hide everything like this. That's what I meant by the third paragraph. All what the JS app sees is "HTTP body data is transferred to the cache". Actual bytes are not exposed.

[annevk]

I would be okay with putting http://hg.mozilla.org/mozilla-central/file/366b5c0c02d3/netwerk/protocol/http/nsHttpChannel.cpp#l4042 in http://fetch.spec.whatwg.org/#http-network-or-cache-fetch

That would also be the place where I'd say that Content-Encoding does not happen there. (Maybe via a flag, as most Fetch consumers do want it to happen before they get a response.)

I guess what you're saying above is that Transfer-Encoding and TLS does happen there? I shall explicitly call that out. Hopefully at some point we have a better HTTP specification we can layer upon here. Although given one was just released I'm not sure how likely that is. @mnot?

[annevk]

@bzbarsky I would really like to define all the network layer stuff as well, including where we violate or not follow HTTP. That seems somewhat important now that we are exposing more and more of that.

[annevk]

One extra note, although we do not expose TLS (and maybe other things), progress events in XMLHttpRequest will get the number of raw bytes transfered, correct? That needs to be called out and explained in the same place.

[bzbarsky]

I think progress events should get number of raw bytes of HTTP entity body transferred.

That is, they should get number of bytes of the thing you get after undoing TLS and transfer-encoding, but before undoing content-encoding. This is the thing whose overall size is represented by the Content-Length header.

[annevk]

That sounds good. That way progress events and fetch() streams would be aligned.

[tyoshino]

Let's say that response's body is set to the payload body (term of RFC 7230) in the spec clearly in this sentence. This sentence is too simple currently.

Let response be the result of making HTTP request using HTTPRequest following the requirements from HTTP as appropriate

[annevk]

No, it should be the HTTP message body: https://tools.ietf.org/html/rfc7230#section-3.3 We want transfer encodings to be handled, just not content encodings.

[bzbarsky]

I believe in terms of RFC 7230 terminology @tyoshino is correct. The "payload body" of RFC 7230 corresponds to the "entity body" of RFC 2616. The "message body" of RFC 7230 is the "message body" of RFC 2616. Compare the phrasing:

The message body (if any) of an HTTP message is used to carry the payload body
of that request or response.  (RFC 7230)

and

The message-body (if any) of an HTTP message is used to carry the entity-body
associated with the request or response.   (RFC 2616)

I'm not sure what the HTTP spec term is in the 723x family of RFCs or the thing you get after undoing content encodings, but it doesn't really matter for our purposes here.

[annevk]

No, that is incorrect. Quoting the most recent RFC: "The Transfer-Encoding header field lists the transfer coding names ... that have been (or will be) applied to the payload body in order to form the message body."

[bzbarsky]

Exactly. So you start with the payload body, apply the transfer encodings to produce the message body, and send the message body. The recipient receives the message body, undoes the transfer encodings, and ends up with the payload body.

[annevk]

Mea culpa.

[annevk]

The one open question here is where do we want to handle content codings? For fetch() it is clear that we only want to handle content codings at the API layer. But is that true for <img> and such too? Should we require each specification to have a "deal with content codings" step? Should this be a flag in Fetch?

[annevk]

A "preserve content codings flags" seems like the simplest solution for most specification authors. I will go with that for now.

[bzbarsky]

What do HTTP libraries do?

What Gecko's does is that it will undo Content-Encoding by default unless the consumer explicitly asks for it to be left alone. The consumer can do this up through the end of the "headers available" notification (which means it can examine the actual headers sent to make this decision).

[annevk]

Not sure how fetch() can easily provide such decisions at that point. For now fetch() simply doesn't deal with content codings. Everything else will have it handled by default. FetchBodyStream's as* methods also handle them.

I think this bug is fixed now. I defined the Apache workaround in the specification.

If there's more HTTP quirks they are ideally added to http://wiki.whatwg.org/wiki/HTTP and I will work on fixing them over time. Either by monkey patching HTTP, or by getting the base specification sorted. It's important for Servo and other new browsers that this happens as well as for web developers to have to deal with less divergence among browsers.

I'll leave this open for comments for a while.

[mnot]

/me wakes up

A flag makes sense. I'm a _tiny_ bit nervous about making it default to stripping content-codings, since that might have some surprising results, if the consumer counts on headers that change when the body does (e.g., ETag, Content-MD5, various signature stuff).

Also, you say 'codec', but a payload can have multiple content-codings...

WRT the monkey patch for Apache - I don't see a bug against Apache mod_deflate; has someone raised one?

[mnot]

Checking Apache/2.2.26, I don't see it setting both the content-type and content-encoding;

GET /~mnot/foo.gz HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: Thu, 26 Jun 2014 06:25:32 GMT
Server: Apache/2.2.26 (Unix) DAV/2 mod_ssl/2.2.26 OpenSSL/0.9.8y
Last-Modified: Thu, 26 Jun 2014 06:21:20 GMT
ETag: "24510cf-1f-4fcb7337eb800"
Accept-Ranges: bytes
Content-Length: 31
Vary: Accept-Encoding
Content-Type: application/x-gzip

Are you sure this is still a widespread problem?

[annevk]

Would /codings/ be a better variable name? Stripping them by default seems the most sensible given that there's only one caller of Fetch that really cares (fetch(), not even XMLHttpRequest cares).

[annevk]

As for the Apache issue, I guess we'd have to do more research as to why Gecko has that hack: https://bugzilla.mozilla.org/show_bug.cgi?id=1030660

[bzbarsky]

https://bugzilla.mozilla.org/show_bug.cgi?id=35956 is the Gecko bug where this got added; it has pointers to various Apache bug reports which may or may not be relevant, as well as specific examples of the behavior we were working around...

[mnot]

codings makes sense.

Like I said, just a little nervous. As long as the default behaviour is well-documented, should be OK.

[annevk]

I think I addressed all feedback and feedback pending research is now tracked all over. Please file new issues/bugs if there's more.

[RReverser]

Having looked through the Fetch API and then through this thread as it seemed the most relevant, I'm still a bit confused: is there a way to read raw body of the HTTP response before decompression or was it decided that this low-level API should be an implementation detail available only to the Cache API and not to JS consumers?

[jakearchibald]

is there a way to read raw body of the HTTP response before decompression

Not right now, no.

[RReverser]

@jakearchibald Ok, thank you, at least now I have definitive answer.

[wanderview]

or was it decided that this low-level API should be an implementation detail available only to the Cache API

FWIW, I don't think Cache API has access to this either. Stuff like Cache.add() effectively does a fetch() which will strip any content-encoding. At least from a spec perspective.

[sheerun]

is there a way to read raw body of the HTTP response before decompression

Not right now, no.

@jakearchibald Can this be addressed? I also need to access original response