Fetch: Request body streams should use chunked encoding

[ddragana]

Chunked encoding in the client-server direction is not widely used on the web. Stand-alone apps use it, but browser(s?), at least Firefox, do not support it.

the chunked encoding from server to client is broken on some server (they are probably old once, I do not have any data about this).

I am wondering what kind of bugs we will get if we start using chunked encoding in the client-server direction.

should we limit request body streams only to h2? We definitely should restrict it to https.

[annevk]

cc @yoichio @yutakahirano

[yoichio]

To protect the legacy servers, the fetch spec states UA doesn’t allow no-cors requests to have a streaming body, and UA forces CORS preflight for any cross-origin request with a streaming body. [1]
Thus client and server are owned by single author.

[1] (https://fetch.spec.whatwg.org/#dom-request)

[annevk]

I think same-origin-policy-wise we indeed got it covered, but I think we should strongly consider not requiring clients to implement chunked encoding for uploads and require H2 as a minimum for upload streams.

If we do decide to implement chunked encoding for uploads requiring HTTPS would help avoid problematic middleware.

cc @whatwg/http

[MattMenke2]

Sending chunked uploads over the web without some sort of server advertisement of support also makes me pretty nervous about middle-box/server compat issues, though requiring a server advertisement first addresses the server issue.

I don't think we want to make chunked uploads an optional client feature, just from a cross-browser compatibility standpoint. If one browser supports it and others do not, seems like that would bite a lot of web developers.

Requiring H2 instead presents its own set of problems with SSL-decrypting middleboxes, but I, at least, am less concerned about breaking just those.

[ricea]

I think it would be a pity to introduce an H2 dependency as H2 won't work in all environments, which could push the need to carry a perpetual fallback code path onto applications.

However, the same middleboxes that are breaking H2 may also break H1 chunked uploads, so there may be no good choices.

I'd like to at least experiment with permitting it over H1. We can compare success rates for H1 and H2 and see if there is a significant difference.

[yutakahirano]

Requiring https sounds reasonable. I'm not sure about requiring H2...

We could instead add another header to the CORS preflight response (note that we already require preflight for all cross-origin requests).

[yutakahirano]

Anyway, I'd like to hear opinions from server side developers.

[ioquatix]

I am the engineer behind Ruby's async-http HTTP/1.x and HTTP/2 client & server and falcon which is a zero-configuration HTTP/1 & HTTP/2 server for Ruby web applications.

When I first started working on async-http it was not clear to me that bi-directional streaming was possible via HTTP/1. But there are several ways it can work:

a/ For HTTP/1.0 (and 1.1) you can leverage shutdown(WR) and use a request body without a content-length (and obviously connection: close). This is in violation of the spec but only slightly and in practice it does work in certain situations. We don't expose this by default since it violates the spec, but at least I wanted to comment on it since I played around with it and it allows bi-directional streaming without chunked encoding.

b/ For HTTP/1.1, using chunked encoding for the request body works perfectly and is completely standards compliant.

c/ For HTTP/2, obviously this is a non-issue.

Falcon supports (b) and (c) by default, out of the box.

I wanted to try this out and provide a way for others to try it out too, so I quickly threw together a remote echo server.

Firstly, you can get the client and server here: https://github.com/socketry/utopia-falcon-heroku/

The application is hosted on Heroku right now: https://utopia-falcon-heroku.herokuapp.com/

Response Streaming

http://utopia-falcon-heroku.herokuapp.com/beer/index?count=100

It is actually going via an HTTP/1.1 middle box and working perfectly.

Request Streaming

Locally (working)

Using the code above:

$ bundle install
$ falcon serve

To start the client:

$ rake echo URL=https://localhost:9292/echo/index

Type some lines and they will be echoed back in real time.

Heroku (not working?)

Run this on the client:

$ rake echo URL=https://utopia-falcon-heroku.herokuapp.com/echo/index

You should be able to type lines and get an interactive response, but it seems like the request might be getting buffered. I can add more logging on the server to see what is going on, and will report back.

Conclusion

I wish that we would focus on the standards more than the existing implementations. The point of the standard is to say how it should work. And the point of the engineer is to implement it. The standard can say "This should work on HTTP/1.1" and it's within the existing design parameters. The fact it doesn't work is a bug. By encoding this into the standard, you are essentially encoding buggy designs into the standard. If you do this repeatedly, I can imagine that we end up with a huge mess based on crappy implementations, and standards that are getting repeatedly more messy.

[ioquatix]

I did some more testing.

It seems that while Heroku supports streaming uploads, and streaming downloads, it doesn't support bi-directional streaming correctly.

The server log indicates it's receiving the individual lines I send it, but the response appears to be buffered until the request body is completed. I'm going to create an issue on the appropriate Heroku repo.

That being said, this should work. And if the standard can mandate that it should be supported by H1, then it requires all middle boxes to be upgraded to support it.

Given that the vast majority of existing infrastructure still runs on H1 - I think this is going to be necessary. This isn't just about client -> server, but client -> heroku -> server etc.

[bradisbell]

Are there more details about the reasoning behind the proposed restrictions?

Streamed HTTP PUT requests are made all the time. Many of these likely violate spec by not using chunked encoding but also not specifying content length (in the case of indefinite lengths, like live streaming media). Some do successfully use chunked encoding, but I don't have a sense for how widespread this is. In any case, it seems strange to limit what the browser can do out of a sort of protection for a server. I don't understand the reason for a server to declare that it supports chunked encoding from the client.

Is there a particular potential side effect that is the source for concern?

[MattMenke2]

From a practical standpoint, if a significant enough number of users are on browsers that don't implement a "standard", it's not really a standard, and developers will generally either not use it, or if they do use it, ensure they have some sort of fallback logic, unless they can compel their users to use a particular browser.

Switching a half-duplex API with hundreds of bits dangling off of it with a full-duplex API, things can fall apart badly, so we're talking a large investment of time in working through all that. It also adds a lot of new opportunities for implementation divergence, as it touches so many HTTP-layer things.

There are things like auth (you get a challenge in the middle of a streaming upload), to socket pool limits, to what to do if we get a 4xx response while still uploading a stream, to keeping streaming uploads with completed response bodies alive, to what these look like to the WebRequest extension API, request events will no longer be in the order that consumers expect them to be in, etc. Alternatively, a different set of internal APIs could be introduced for just this case, but then they'd need to plug in very differently (below the layer that hooks up auth, cookies, etc), which would present its own set of issues around duplicating behavior in two very different codepaths.

Of course, restricting it to H2, you still have all those problems. So I'm not actually sure restricting things to H2 makes things much simpler. The code unique to H1 isn't exactly all that complicated, so even having a second copy of it just for this use case shouldn't be too bad - it's just dealing with everything else between the lower HTTP layer and the web page that would be a major investment.

Anyhow, I don't want to give the impression I'm speaking for anyone who has the power to determine whether this gets done in Chrome or not (I'm not even on the network stack team any more), but it would be a pretty major investment in terms of engineering time. I think it would be good to see if there's buy-in from someone who says, and has the power to say, "yes, I'd be willing to devote 3 to 18 months of engineering time to be sure this gets done", if folks on the Chrome team think this is an API worth supporting.

Edit: Removed confusing, redundant bonus clauses.

[annevk]

To be clear, we already more or less agreed that full-duplex is out-of-scope for this initial round of upload streams as no browser has the architecture for that and nobody is willing to invest in it at this point. It's an open question to what extent Fetch should call that out.

Not using chunked encoding for uploads and also not specifying a Content-Length is not something anyone has considered to date for H/1 I think. I'm not sure we can get agreement on that however as it's non-compliant and a bit of a hack.

Chunked encoding is being considered, but is also new code for all clients involved. For a protocol on its way out it's not clear it's worth investing in. (As a reminder, security-wise we are fine as there's a CORS preflight for it in all cases.) Also, if most deployments end up using H/2 we might end up with these subtle bugs in certain scenarios that never get fixed.

[ioquatix]

To be clear, we already more or less agreed that full-duplex is out-of-scope for this initial round of upload streams as no browser has the architecture for that and nobody is willing to invest in it at this point. It's an open question to what extent Fetch should call that out.

I can respect that.

However, you should be mindful about building the interfaces to support the fetch functionality. If you don't consider the needs of bi-directional streaming, it might be tricky to retrofit it in the future. There are nice symmetries to be had. The way I'd define the spec, is to assume bi-directional streaming is viable, build the interface around that, and then limit it to uni-directional streaming. Nothing is lost, but when browsers are ready to go with bi-directional streaming, the interfaces are ready to go and are nice to work with.

Bearing in mind that even if browsers don't support it, there is nothing to stop Node.js implementing the same fetch style APIs, and that's much more likely to support bi-directional streaming and have valid use cases for it.

[ioquatix]

For a protocol on its way out it's not clear it's worth investing in.

I tried to address this specifically above - maybe it's not needed but a lot of application servers use HTTP/1 proxies. So, maybe browsers don't need it (i.e. they make HTTP/2 connections to load balancer) but the load balancer -> application server which is still often HTTP/1 definitely needs to support it.

[MattMenke2]

Apologies, I completely lost context on what the issue was here in my last comment.

Anyhow, chunked encoding wouldn't be new code for Chromium (unless we have to sniff for HTTP/1.1 server support before using it, at least - which I'm not sure is necessary, given the preflight?) - it would just be exposing an obscure but already existing code path to the web.

[wenbozhu]

For HTTP/1.1, not allowing chunked-encoding and not specifying C-L will force the U-A to half-close the connection which may be interpreted by servers as a client-initiated cancellation, and we even have a http-wg ticket to standardize this behavior.

I highly doubt any proxies will break with chunk-encoded request bodies. Server implementations might but this is a design-time decision, i.e. knowing which http-server or API you app is talking to.

===

Agreed with @ricea ... It will be very hard to use the streamed fetch upload only with H2 if streamed upload will not always work (as decided by intermediaries) and the client application has to decide in the runtime which server-side end-point (URL) to target e.g. one with upload streaming and one without.

===

It's ok to make https a requirement although it might complicate local testing. With HTTP, chrome will always speak http/1.1 (?).

[mnot]

I don't think requiring H2 or TLS will do much good here; many (most?) CDNs talk H1 on the back end, so if there are interop problems, they still can happen (although I think by far the most common behaviour will be generating a 411 Length Required; many/most servers do this if Content-Length isn't present, as they want to know how many bytes they're about to handle).

Don't use half-close to delimit a request; you'll have a bad time.

[yutakahirano]

I am (again) proposing the following:

• Restrict the feature for secure contexts.
  We have no checks for same-origin requests, which means we want some reasons to trust the same-originness. Without TLS an attacker can use this feature to attack servers not ready for it.
• Have a new HTTP header attached to CORS preflight.
  With the header we'll be sure that both the client and the origin server are ready for this feature. We don't need to rely on the use of H2 to exchange the signal.

These don't cover the proxy problem @mnot mentioned, but with TLS the server side developer should have contacts to proxies - because at least they share the certificate. Hence asking server side developers to ensure that the streaming upload feature can be used for their service sounds reasonable to me.

What do you think?

[yutakahirano]

I talked with @annevk at IRC.

• We agreed that the secure context restriction is a must-have.
• I misunderstood the motivation for the H2 restriction. I thought the main motivation was for security and that was why I proposed the new header on CORS preflight.
• @annevk pointed out that for some user agents don't want to implement chunked encoding from implementation cost point of view.
• @annevk and I agreed that it's not clear whether we __need__ the new headers from security point of view.

@ddragana, can you tell us Firefox's implementation status of chunked encoding? If it's not there is it very hard to implement it?

I also would like to know, If some user agents don't want to support H1.1 with streaming upload do we actually need H2 restriction? Alternatively we can expose the support status and/or error status to web developers so that they can fall back.

[annevk]

I chatted with @ddragana in parallel and adding chunked encoding is not a lot of work for Firefox.

I'd be opposed to leaving the decision whether to support chunked encoding up to implementers.

Now that you make the H/1.0 vs H/1.1 distinction, what if the server replied to the CORS preflight using H/1.0? Would we still try talking to it using H/1.1 or refuse to do upload streams for that particular URL?

[yutakahirano]

I think blocking H/1.0 preflight response will bring us some safety and we won't lose much coverage. @wenbozhu @ioquatix Do you agree with the idea?

[sleevi]

@yutakahirano I appreciate the desire to mitigate intermediaries, but I don't think preflights really addresses that holistically. While unencrypted HTTP may have intermediaries present anywhere along the network path, even encrypted (TLS) connections can still have intermediaries. They generally fall into one of two places - intermediaries near the server (which @mnot mentioned) and intermediaries near the client (e.g. local security policy, antivirus, etc).

A preflight doesn't really address intermediaries near the client, and it seems unlikely in practice that it'd successfully consistently address intermediaries near the server, since in both cases, they would generally transmit the header unmodified.

It might be tempting to say "intermediaries near the server need to have the server's private key, so that's the server's problem", and that "intermediaries near the client won't be able to use publicly trusted certificates, so we can require them" - but now that exposes a side-channel to servers that we presently and intentionally don't expose to the Web.

I'm sympathetic that "the ecosystem may be (... is probably) ossified" is not a reason not to do it, but I think this gets back to @MattMenke2 remark in https://github.com/whatwg/fetch/issues/966#issuecomment-555837984

This same conceptual challenge is the same underlying problem the ecosystem faced with TLS 1.3. This took years to successfully do, and generally required a variety of 'hacks' and workarounds in order to eventually get something successfully through. This worked reasonably well for something evolving and that had flexibility to have those workarounds, but I don't think we'd have the same leverage here.

That's why this still represents a major investment to "do it right", even if we could potentially "do it quick", and have it cause a host of long-tail issues. My only worry is that we then punt those long-tail issues to web servers, and we've tried to avoid that (see also the WebSockets handshake).

[annevk]

What is the side channel? That the user is behind an intermediary that does not support upload streams? Isn't that already revealed if an intermediary does any kind of meddling with the traffic (e.g., remove certain headers)?

[sleevi]

If the feature is restricted to only sites with publicly-trusted
certificates, it reveals the user is behind a proxy that may be terminating
TLS.

This could further be used to implement pseudo-DRM schemes to prevent users
from inspecting their own traffic (e.g. by requiring it use such uploads).

[annevk]

Oh, but that restriction was thus far not proposed, was it? Secure contexts + H/1.1 + a forced CORS preflight is the status quo proposal as I understand it.

[sleevi]

Right. I was showing how the existing proposal would be insufficient, and
then trying to head off the most common proposals I’ve heard to mitigate
those issues (typically based on how we’ve solved other problems, but how
those don’t work here).

Sorry for the confusion, was trying to get ahead of a cycle of mitigations
;)

[ioquatix]

@yutakahirano if I understand you correctly, I think your idea is at the very least reasonable and probably required.

[yutakahirano]

@sleevi How severe such attack (using near-client proxies) would be? For usual breakage I think it's reasonable to ask server side developers for help after we make efforts to mitigate the pain (as we are doing right now), as this is not a breaking change.

[yutakahirano]

ping?

[sleevi]

@yutakahirano Thanks for the ping, I missed your question. I'm not sure I understand what you mean by "attack" here. To attempt to summarize the discussions/iterations:

Problem Statement

1. Chunked uploads are not widely deployed at Internet scale (i.e. both client and servers), so there are compatibility concerns.
2. These compatibility concerns involve questions of compatibility with the origin server, and with any intermediates on the path: client-authorized intermediates and server-authorized intermediates.
   
   
   
   • We know and expect some intermediate compatibility issues
   
   

Solutions to date

Origin/Server-side Intermediate

1. Do nothing (e.g. because it's under the full control of the server operator)
2. Do nothing, but declare any issues are not our fault and must be fixed by the server/intermediate
3. Require HTTPS (to prevent accidental intermediates)
4. Require some explicit advertisement/opt-in by the server, even when same-origin
5. Restrict to HTTP/2

Client/Client-side Intermediate

1. Restrict to HTTP/2, as the complexity for client intermediates goes down if they support HTTP/2, and if they don't support HTTP/2, it's a good sign they may have issues
   
   
   
   • Alternatively, if we don't restrict to HTTP/2, try to restrict to some other property of the connection that indicates an intermediate is likely present, yet without revealing more information about the user to the server, which is hard.
   
   
2. Do a preflight with a special header and require the preflight response MUST come over HTTP 1/1+

Challenge

At the core, the question is "We expect some things to break, because these joints are not well oiled" - the broader context of UAs doing chunked uploading is rare today, and thus we expect servers and services are not prepared for it, even if it would be a nice feature to add. Similarly, we want to make sure that when adding a feature, servers don't have to try and deploy workarounds for client issues.

Suggestion

The total set of requirements to minimize risk of incompatibilities that seem reasonable:

1. Require HTTPS. This prevents any unauthorized intermediates (i.e. not explicitly authorized by the server or the user) from breaking.
2. Require HTTP/2. This is to minimize, but perhaps not totally mitigate, the risk of client-side intermediates breaking.
   
   
   
   • This also reduces the complexities involved with HTTP/1.1 and client signaling, even when no intermediates are involved.
   
   
3. Continue to prohibit cross-origin streaming/chunking by default, and add an explicit header to allow a destination server to opt-in.
4. Require implementors to minimally support the above (i.e they MUST attempt to support chunked encoding if the above invariants hold)

If folks are uncomfortable with those requirements, and want to propose alternatives, it might be useful to figure out how each proposal addresses the client side and the server side issues. It may be that something better for the server makes it worse for the client. But when we talk "middleboxes", we're really thinking about two sets: one set which are in control of the origin (e.g. CDNs they've contracted with), and another set that they are not in control of (e.g. because the user configured them)

[yutakahirano]

Sorry for the really belated response.

@wenbozhu and I talked about this.We (@wenbozhu and @yutakahirano ) are happy to restrict the feature to secure contexts, and force CORS preflight for crossorigin requests. However, we want to allow the feature for HTTP/1.1 connections.

The core issue of restricting this feature to HTTP/2 only is that it will force client-side frameworks to implement complicated fallback logic in the runtime, i.e. when HTTP/2 is blocked by forward proxies. The fallback may not even be possible e.g. if the server interfaces don’t support non-streaming “upload”, esp. for API based frontends. That is, those APIs will have to implement two sets of methods, one for accepting streamed upload and one for accepting non-streaming upload (e.g. with multiple POSTs).

In addition, this will make the fetch/stream API more complicated, to signal when streams are disallowed due to transport versions, unless there is a standard API to query the current HTTP version of an origin … but then the 1st request to an origin will not have this data as the connection is not yet established. Also, proxies may change too, e.g. during network handoff, for a given origin.

Overall, we just don’t see how this can be deployed for any end-user facing Google services when we don’t control the client-side environments.

Currently we don’t know the impact of client-side intermediates breaking - We don’t know how many servers would break, and how badly they would break. I would like to have a joint experiment with @wenbozhu to see the impact of the breakage by having an origin trial.

[sleevi]

Don’t all of those concerns exist when supporting HTTP/1.1 as well, just
with less determinism? That is, when intermediates choke on this, the same
considerations for client side developers would apply. Or did I
misunderstand?

[ricea]

Currently we don’t know the impact of client-side intermediates breaking - We don’t know how many servers would break, and how badly they would break. I would like to have a joint experiment with @wenbozhu to see the impact of the breakage by having an origin trial.

+1 to experimentation.

[yutakahirano]

@sleevi Yes that's right, and that's why we want to run an experiment to see the feasibility.

[sleevi]

I see.

I think the complexity of the H/2 path is overstated, and unreasonably
biased towards presumed server-side complexity, while overlooking the
client-side complexity (that is, complexity for the JS developer primarily,
although to a much lesser extent the UA developer as well).

Having an API that only works over H/2, which does not require the
complexities previously described and deterministically fails fast, seems a
much better developer experience than a non-deterministic API that
heuristically encounters issues. That seems like a much more useful
starting point, and provides a forward path to allowing developers to get
determinism by default, and opt-in to non-determinism if they’d like to try
to handle it. That would seem to allow shipping something to users sooner,
and that works more reliably, then trying to optimize for a single server
implementor.

[yutakahirano]

@sleevi in that case we could run two experiments simultaneously, by introducing a temporary flag, say requireH2ForStreamingUpload, into RequestInit, for example. @sleevi are you fine with the idea? Can you help @wenbozhu and me set up the experiment (for the h2-only case)? @wenbozhu, what do you think about the idea?

[wenbozhu]

Having an API that only works over H/2, which does not require the
complexities previously described and deterministically fails fast, seems a
much better developer experience than a non-deterministic API that
heuristically encounters issues.

The same argument goes with the H/2 restriction, as the app doesn't control what forward proxies the client will interact with and not all forward proxies support H/2.

OTOH, H/1 proxies should support chunked T-E (*), and for servers that have issues with chunked T-E, the app (or client libraries for the end-points) would know not to use request streaming.

[determin1st]

First, implicitly bind H2 to HTTPS.
Next, state only HTTPS

Enforcing your corp's policy?

[yutakahirano]

ping?

[bradisbell]

@sleevi To reply to your comments inline...

I think the complexity of the H/2 path is overstated, and unreasonably
biased towards presumed server-side complexity

There are servers today (such as Icecast) that will probably never support HTTP/2, which work just fine by receiving a stream sent over an HTTP request. In my opinion, the complexity of going to HTTP/2 is massive for projects like these. There are 20 years worth of deployments for Icecast alone.

Other servers are often backed by applications that may not work in an HTTP/2 compatible way. they've been receiving streams of data since their inception.

while overlooking the client-side complexity (that is, complexity for the JS developer primarily,
although to a much lesser extent the UA developer as well).

It's possible I don't know all of the complexities you're talking about, but it feels trivial to me for the JS developer:

fetch('https://example.com/some-stream', {
  method: 'PUT',
  body: someReadableStream
});

We can do the equivalent with CLI cURL. We can integrate this type of thing with most any other HTTP client library in applications. FFmpeg supports it. Seemingly the only place we cannot do this is the web.

In other words, streaming a PUT body isn't new... and therefore any intermediate problems you're expecting to hit likely already exist and won't be made worse by simply allowing us JS developers to simplify and bypass a ton of web socket proxy cruft we've had to add, just to make a streaming HTTP request.

[sleevi]

@bradisbell wrote:
There are servers today (such as Icecast) that will probably never support HTTP/2, which work just fine by receiving a stream sent over an HTTP request. In my opinion, the complexity of going to HTTP/2 is massive for projects like these. There are 20 years worth of deployments for Icecast alone.

It does not seem "massive" to potentially place a forward proxy in place (e.g. a service like Caddy) for such services. That may seem dismissive, but it's about respecting the priority of constituencies and placing users first. Using our Windows users over a 28 day window, approximately 6.7% of requests in Chrome go through a locally-configured TLS endpoint. However, by user, that's approximately 17.3% of Windows users going through such a service within that same time period, indicating a substantial portion of our users.

The most common cause of this, by far, is the use of antivirus that performs TLS inspection of traffic, acting as a transparent proxy to inspect requests and responses, scanning and modifying them. We have ample experience in the challenges these services pose to making otherwise sensible ecosystem changes.

However, unlike TLS 1.3, which involved cryptographic negotiation, the failure scenario is not so easily tested. Historically, when considering features to roll out, such as the development of QUIC and TLS 1.3, the Chrome networking team constructed careful test scenarios to measure the success rate of various proposals. That is, using a well-known server endpoint which was configured correctly, carefully measuring the success rate of various client approaches. If @yutakahirano and @wenbozhu were proposing that approach first, to gather real world data without necessarily making it part of the Web Platform, I'd be much less concerned.

However, even when that approach was used, it's not perfect, because not all users opt-in to supporting Chrome experiments and metrics, and as a result, their environments are not considered in the design. The aforementioned link regarding TLS 1.3 shows what can happen there, while ultimately, QUIC had to decide to refuse to negotiate in such scenarios, because of middleboxes improperly implementing the protocol.

I can understand your appeal to other libraries, and the spec, as it seems like it should "just work". However, the Chrome team has ample experience that the real world is anything but that. The most depressing example of this I know was a popular DSL router (the most popular in the US at the time) shutting down DNS if too many requests were received. The combination of DNS prefetch plus a change in our network resolver code, both fully spec conformant, would have knocked millions of users offline, had it not been caught by a Chrome VP who happened to use that particular router at home, and was able to have multiple Chrome engineers over to investigate and diagnose why Chrome Canary was knocking them offline.

It's possible I don't know all of the complexities you're talking about, but it feels trivial to me for the JS developer:

fetch('https://example.com/some-stream', {
  method: 'PUT',
  body: someReadableStream
});

Indeed, the API is the simple part. The concern is the intersection that chunked encoding is going to have for that 17.3% of Windows users (ignoring other platforms for sake of discussion). The 'best case' scenario is this feature simply does not work, and such fetch requests fail. The Web developer now has to try to figure out why it's failing, and the user experience of such failure is, more often than not, poor, because the developer has failed to account for the possibility, since it's "trivial" to do. The worst case, however, is that this can interfere with or disrupt such transparent proxies.

Is it a bug in the proxy if they do? Absolutely. Is it something someone could do with Curl, wget, ffmpeg, or any other native program running on the device? Yup, unfortunately. However, the prospect of exposing this to the Web, with the risk of disruption and incompatibility, is high, particularly given this is an area we know has extremely high likelihood of being implemented incorrectly.

The proposal to restrict to HTTP/2 attempts to mitigate this risk, on the client side, as generally, those proxies will not negotiate HTTP/2 or, if they do, are less likely to have issues in this particular area of code because it's a well-oiled mechanism of HTTP/2.

I can understand this seems overly conservative, but the Chrome Networking team has seen all manner of depressing things, and that's why it makes us nervous. Typically, before exposing something to the Web, let alone as an API, we've done wide-scale measurements as best we can to assess compatibility risks. For features that we expect may affect users who have not opted in to User Metrics (e.g. Enterprise users), we're even more cautious in the deployment, because we know it represents a significant investment to get working correctly.

The suggestions here are based on strategies to reduce the risk, while still hopefully making forward progress on addressing authors' real needs, even if not completely. I'd much rather there be 10,000 sad Icecast authors than 172 million sad Chrome users. Even if we assume only 0.1% of users would encounter some breakage, that's still an order of magnitude more sadness. And I hate sad users 💔

[sleevi]

@yutakahirano wrote:
@sleevi in that case we could run two experiments simultaneously, by introducing a temporary flag, say requireH2ForStreamingUpload, into RequestInit, for example. @sleevi are you fine with the idea? Can you help @wenbozhu and me set up the experiment (for the h2-only case)? @wenbozhu, what do you think about the idea?

How do you feel about inverting the flag? Say, allowH1ForStreamingUpload?

The thinking is this:

1. If this turns out to be not an issue in practice (despite the precedent), then it becomes easy to allow H/1 generally, simply treating this flag as a no-op in implementations that always allow H/1
2. It sets the default state to be conservative, and requires explicit opt-in to the riskier approach. This strikes a balance with picking a 'safe' default (given past precedent)

The only thing I'm not sure of, with any restriction or permission, is that it would indirectly leak the nextHopProtocol from Resource Timing, as either way, the flag would be a way to probe for H/1 vs H/2 support. I know we talked about some form of advertisement, and I realize there is a CORS preflight for any cross-origin request, but it may be worth also contemplating the added header or otherwise a signal that such information may be disclosed.

[bradisbell]

It does not seem "massive" to potentially place a forward proxy in place (e.g. a service like Caddy) for such services.

I have yet to see a proxy such as this that handles all of the compatibility edge cases required for streaming media over HTTP. A lot of user-agent-specific modifications are needed.

In any case, thank you for explaining in detail the compatibility issues you foresee. The context is very hepful. I just want to respond to this:

And I hate sad users 💔

We all do. Chrome isn't in isolation. The problematic hacks we have to put in place to work around simple things like being unable to make an HTTP request with a streaming request body lead to problems for users. I appreciate the efforts going into getting it right and building the most solid foundation possible. It's important however that the whole picture be considered. For an alternative example, when a browser developer says they won't implement X API for potential Y security/privacy concern... if the user is then just going to download some native binary and run it with root/admin permissions locally, then the security problem hasn't been resolved... it's just been made worse, and someone else's problem.

In this case, I suppose we'll continue on with duplicating servers and bandwidth so that we can receive streams from browsers. I believe strongly in keeping web applications first-class citizens in the work that I do, but there is a real cost to it, in terms of complexity and resources, which trickles down to the user as less reliability and more money.

[determin1st]

And I hate sad users

I'm a Fetch API user and I'm sad.

..this seems overly conservative

Nope, another word - egoistic. It's you, who bring more restrictions and complexity into conservative HTTP. (preserve simplicity to be conservative)

[yutakahirano]

@yutakahirano wrote:
@sleevi in that case we could run two experiments simultaneously, by introducing a temporary flag, say requireH2ForStreamingUpload, into RequestInit, for example. @sleevi are you fine with the idea? Can you help @wenbozhu and me set up the experiment (for the h2-only case)? @wenbozhu, what do you think about the idea?

How do you feel about inverting the flag? Say, allowH1ForStreamingUpload?

I'm fine with this. @wenbozhu, thoughts?

The only thing I'm not sure of, with any restriction or permission, is that it would indirectly leak the nextHopProtocol from Resource Timing, as either way, the flag would be a way to probe for H/1 vs H/2 support. I know we talked about some form of advertisement, and I realize there is a CORS preflight for any cross-origin request, but it may be worth also contemplating the added header or otherwise a signal that such information may be disclosed.

So you are thinking about Timing-Allow-Origin, right? We can require the header. On the other hand my vague understanding is that ResourceTiming people are looking to a direction that CORS implies TAO.

[yutakahirano]

For timing-allow-origin I filed #1007.

[wenbozhu]

(sorry about the delay, with all the WFH stuff going-on now ...)

I am happy to implement any experiments Yutaka mentioned.

===

I'd also like to point out that with H2, the value of request-streaming is actually less than with H1, v.s. parallel on-demand POSTs, which

• are multiplexed, with headers compressed;
• may be sent over multiple TCP connections;
• are able to ack the delivery of client-sent data immediately while with streaming, any amount of data could be lost when the HTTP transaction is aborted.

There are certainly use cases with very high message rates that will benefit from (short-lived) request-streaming due to reduced overhead for processing individual requests.

===

Re: fallback, agreed that fallbacks are always needed even when both H1 and H2 are supported.

There are three compatibility issues as I understand:

1. proxies buffering HTTP bodies.
2. proxies not supporting H2.
3. servers not supporting H1 chunked-encoding or H2.

There is no fail-fast for 1), independently of H1 or H2. The fallback solution is often complicated.

For 2), fail-fast by the proxy, but unknown at the design time (of the web app).

For 3), fail-fast by the server due to the lack of Content-Length etc, and known at the design time. This is the same situation as servers not supporting H2.

With HTTPS, I believe 2) is a much more common issue than 1). Restricting request-streaming to H2 doesn't avoid 1) either.

While 3) is a design-time issue for web apps, I don't know the scale of the problem for those "file-uploading" end-points that don't author any JS library for their clients. We won't be able to run any experiment for them either.

[sleevi]

The compatibility concern you describe as 1) is not the compatibility concern we’ve been speaking of. If proxies uniformly buffered, I don’t think that’s a compat concern.

Instead, it’s proxies that fail the request or end up in bad states (and there’s plenty of prior art in WAFs to highlight state confusion around chunked responses that would support this). Here, the compat risks range from failure to incomplete data to having the client machine or network enter a “bad” state.

[mnot]

See also https://tools.ietf.org/html/draft-nottingham-bikeshed-length which seems to have support in the HTTP WG (although not adopted yet). It might be helpful for case 3).

[wenbozhu]

https://tools.ietf.org/html/draft-nottingham-bikeshed-length

This is good to know.

I assume that for implementations that don't support chunked T-E, the server will immediately fail a request that misses C-L in the request headers.

[MattMenke2]

Assuming servers behave rationally on failure is not a safe assumption - I wouldn't be surprised if some servers treated it as a 0-length upload body, for instance.

[benbucksch]

I'd like to hear opinions from server side developers.

I would expect this to behave like any other PUT, without chunked encoding.

[yutakahirano]

I'd like to hear opinions from server side developers.

I would expect this to behave like any other PUT, without any special chucked encoding.

But we don't have content-length, so we need to use chunked transfer encoding (for HTTP/1.1).

[benbucksch]

content-length is optional. In some cases, it's even forbidden.

RFC 2616 Section 4.4: "3. If a Content-Length header field (section 14.13) is present, its ... value ... represents both the entity-length and the transfer-length. The Content-Length header field MUST NOT be sent ... if a Transfer-Encoding header field is present"
(emphasis added by me)

[MattMenke2]

Browsers don't currently send Transfer-Encoding request headers at all (and servers likely don't support them). Moreover, without a content-length, the only way for a client to indicate the end of the request is Transfer-Encoding: chunked (which is what we're discussing..) or half-closing the socket. Browsers don't do the latter (and lot of servers probably wouldn't work if browser did that), and making an API require sockets not be reused if using HTTP/1.x seems not great.

[benbucksch]

closing the socket

You're right, that's not a good idea. Thanks for the clarification.

[gterzian]

How about using chunked encoding only if the developer explicitly provided the header as part of HeadersInit?

I can imagine developers wanting to provide a stream as request bodies simply for ergonomic reasons, and one could start supporting streams as request bodies, without having to make a final decision on the underlying networking model.

For example, I think step 4.2 of https://fetch.spec.whatwg.org/#http-network-fetch could be dropped, instead relying on the dev to provide the header if appropriate.

We could then rewrite https://fetch.spec.whatwg.org/#concept-request-transmit-body to look for the header, or whether http2 is enabled(or some other yet to be defined heuristic), and define what "transmit bs" at step 5.2.2 means in more detail.

Worst case, if a stream has been provided as a body, but no "chunked" header set, or for other reasons the UA can't support chunked, then the bytes could simply accumulate in a buffer as they are read from the stream, and when the stream is done, transmitted in one chunk as a "normal" request.

FYI this what I ended-up doing in Servo to make fetch/api/basic/request-upload.any.js pass...

[benbucksch]

the bytes could simply accumulate in a buffer as they are read from the stream, and when the stream is done, transmitted in one chunk as a "normal" request.

@gterzian : I need the streaming, because I'm actually streaming live audio. I need the audio to be processed immediately, before the end of the stream. That's the whole reason why I need this.

[ioquatix]

Remember that half closing a stream is a very viable option for HTTP/2.

[gterzian]

I need the streaming, because I'm actually streaming live audio. I need the audio to be processed immediately, before the end of the stream. That's the whole reason why I need this.

Ok, so in that case I assume you also know exactly what the server expects, and you would be able to either set the "chunked" encoding header, or some other flag that would be provided by the Fetch API, to "tell" the UA what underlying networking model to use?

What I'm saying is that, if I were to try to sum-up the discussion so far, it would be "it's complicated". And I think one reason for this is the attempt to tie using a ReadableStream as a request body with a specific networking model.

I'm instead arguing the API can be supported, with a worst-case implementation involving simply buffering the bytes as they are read from the stream, and further flag/headers can be defined by Fetch and used by users who "need" a certain networking model, to modulate what step 5.2.2 of https://fetch.spec.whatwg.org/#concept-request-transmit-body exactly does.

For example step 5.2.2 could switch on certain headers being present, or certain flags set, and then "transmit" bytes over the network according to certain networking models. Let's say similarly to how extract init body switches based on the type of the object. This approach would be extensible I think, since the switch statement could be extended as new networking models are supported.

[benbucksch]

the attempt to tie using a ReadableStream as a request body with a specific networking model.

As a developer (API caller), when I use a stream, I expect that the data is consumed in an ongoing fashion. That is the reason why I use a stream, in any platform and API. If the data was buffered and processed only at the end of the stream, I would be surprised and consider that a bug.

How to make that possible, is another question.

In HTTP 1.1, my suggestion to not send a Content-Length and simply let the data stream stop has the problem that the TCP connection needs to be closed to signal the end, which is not ideal.

How does HTTP2 play into this? Does HTTP2 binary framing and streams allow to signal the (possibly premature) end of one stream, without closing the connection?

If that's possible, we'd have 3 solutions:

• HTTP2: No chunking, just end stream (but not TCP connection)
• HTTP 1.1 with chunking
• HTTP 1.1 without chunking: Close TCP connection

Personally, I would prefer the latter (hard closing TCP connection) over buffering the entire content to the end.

[domenic]

I would be actively against passing a ReadableStream causing buffering. That completely defeats the point of streams, and I would much rather have fetch() throw if it cannot properly stream the stream.

The point of this feature is not to check a checkbox in the matrix of supported types as arguments to fetch(). The point is to support streaming upload.

[ioquatix]

HTTP/2 supports full bi-directional streaming with shutdown read/write on a per-stream basis.

HTTP/1 uploads need to use chunked encoding.

[gterzian]

I would be actively against passing a ReadableStream causing buffering. That completely defeats the point of streams, and I would much rather have fetch() throw if it cannot properly stream the stream.

I think we could add a switch statement to https://fetch.spec.whatwg.org/#concept-request-transmit-body, one that would switch on the various variables mentioned in this discussion(such as HTTP/1 or /2), and then proceed(or not) to transmit bytes accordingly over the network as they are read from the stream.

I could have imagined the default "we don't know what to do" could be to buffer the bytes, however terminating the fetch could be done at that point as well. I'm not sure if throwing would be possible from there, however it might be done earlier as part of the fetch method(although perhaps code running there should not/cannot be aware of things like whether the network is running HTTP/1 or /2).

HTTP/1 uploads need to use chunked encoding.

The current spec already seems to agree, as it is setting the chunked header, in case of a ReadableStream(null source), only if the connection is not HTTP/2, at step 4.2 of https://fetch.spec.whatwg.org/#http-network-fetch, with the following wording:

If connection is not an HTTP/2 connection, request’s body is non-null, and request’s body’s source is null, then append Transfer-Encoding/chunked to request’s header list.

[annevk]

Note that #538 is the higher-order concern here.

[gterzian]

The below describes the heuristic used by Hyper(Rust HTTP libary) which I find informative:

If the body's size_hint method does not return an exact size, the hyper will check if you've included a specific header to describe the body. If that doesn't exist either, hyper will resort to a streaming body, which over HTTP/1.1, will use chunked transfer-encoding. It _should_ automatically add the header. (source: https://github.com/hyperium/hyper/issues/2232#issuecomment-644322104)

I'd say it even matches roughly the current (un-implemented) wording of the spec, with the exception of the default fallback to streaming.

On thing worth noting is fetch sets the "content-length" header at step 5.6 of #http-network-or-cache-fetch only if total-bytes is set, and that seems to only be set at step 39 of the Request constructor.

So it might make sense to set total-bytes as part of extracting a body in the non-stream case, to ensure the request will have a content-length header in the non-stream case. This is something that is already tracked at https://github.com/whatwg/fetch/issues/604

If total-bytes were to be set as part of extracting the body, I think a request reaching the "transmit body" stage would either:

• have a 0 content-length header, if the body is null
• have a positive content-length header, if the body is non-null and the source is non-null(not a stream)
• have a chunked content-encoding header if the body is a stream, and the connection is not HTTP/2.
• have none of the above headers, if on HTTP/2 and body source is null(stream).

And this would seem to be useful information to then decide how to "transmit bytes" as part of https://fetch.spec.whatwg.org/#concept-request-transmit-body.

This is also how it's prototyped in Servo(where total-bytes is set as part of extracting the body, and Hyper is used as the HTTP libary).

[annevk]

@gterzian again, I would not invest too much effort into this at this point as we still haven't reached agreement on a number of design aspects. Perhaps to clarify that further we should remove what's in the specification now as it's not complete (see the various streams issues in this repository).

[jakearchibald]

Allowing it to work over HTTP/1.1 seems important during development, as most development servers are HTTP/1.1 and not-https. I guess that could be a devtools flag or something, if we can't support HTTP/1.1 in the wild.

[annevk]

@jakearchibald do most development servers support chunked encoding in requests though? E.g., the one from WPT does not.

[jakearchibald]

I'm not sure about 'most', but Node & ExpressJS support it https://github.com/jakearchibald/chunked-encoding-request-test/.

[ioquatix]

Falcon is a Ruby web server which supports HTTPS and HTTP/2 for development, so it certainly is possible.

[jakearchibald]

Node also supports HTTPS and HTTP/2, but I don't see it used often in dev. To clarify, I'm not saying that running HTTPS and HTTP/2 locally is impossible. I'm saying it isn't often used.

[jakearchibald]

From the spec, it looks like we use chunked encoding even if the developer provides a Content-Length. If this to avoid cases where the developer sends less than the content length they provided?

[annevk]

You cannot set Content-Length.

[jakearchibald]

Hah well that'd do it 😀