Fetch: Why does Fetch specify a max redirect count of 20?

[cdumez]

Why does Fetch specify a max redirect count of 20?
See https://fetch.spec.whatwg.org/#http-redirect-fetch (step 5)

It seems Chrome and Firefox have implemented this. I wanted to align Firefox but found out that CFNetwork currently uses a max redirect count of 16 internally which is not configurable. This is preventing us (and any CFNetwork client) to align with the Fetch specification.

Would people consider lowering the max redirect count in the Fetch specification given this (new?) information? The alternative for us would be to ask CFNetwork to make it configurable but this would likely take time.

[cdumez]

cc @youennf

[sleevi]

@cdumez Asking folks internal to see if there's any Ancient Wisdom behind this limit, but from source archaeology for Chrome, our limit of 20 was chosen to match Gecko/Firefox at the time we first open-sourced

See https://chromium.googlesource.com/chromium/src/+blame/5cbb127e6b67a646b98a460faf89740f29b91403/net/url_request/url_request.cc - which traces its provenance of kMaxRedirects to Gecko.

Given that Safari is limiting to 16, that seems compelling enough to suggest there is limited risk, and interoperability is always good. Barring any concerns being shared, I think we'd be open to aligning with Safari.

[cdumez]

FYI, from HTTP 1.1 RFC:

https://tools.ietf.org/html/rfc7231#section-6.4

A client SHOULD detect and intervene in cyclical redirections (i.e.,
"infinite" redirection loops).

  Note: An earlier version of this specification recommended a
  maximum of five redirections ([RFC2068], Section 10.3).  Content
  developers need to be aware that some clients might implement such
  a fixed limitation.

cc @annevk

[mnot]

If you can agree on a common limit between browsers, we can refine this in the next HTTP; tell us on httpwg/http11bis if so.

[wanderview]

This changed in gecko 15 years ago:

https://bugzilla.mozilla.org/show_bug.cgi?id=173147

We went from a max of 10 to 20 because "it seems that nytimes.com has been known to string together more than 10 redirects". Its possible nytimes.com has changed since then.

[wanderview]

The limit of 10 was introduced here based on lynx's behavior:

https://bugzilla.mozilla.org/show_bug.cgi?id=83471

Prior to that it appears we just looped forever.

[sleevi]

One thing that may present a challenge for us lowering - we consider certain types of internally-generated rewrites (extensions, HSTS) as redirects for purposes of redirect counting. I believe, perhaps incorrectly, that Firefox does something similar with HSTS rewrites.

@cdumez Do you know what CFNetwork does in that space? I don't know if CFNetwork handles HSTS or whether that's added by the Safari layering atop, and I don't know whether or not that's counted as a redirect for purposes of limits.

[wanderview]

Yea, we do internal redirects for all kinds of things and they appear to be included in our limit checking:

http://searchfox.org/mozilla-central/source/netwerk/protocol/http/HttpBaseChannel.cpp#3381

[cdumez]

It does not appear CFNetwork includes HSTS in the redirect count.

[sleevi]

@wanderview Does Firefox have metrics on redirect counts, to assess what the impact would be?

We don't have general redirect counts, but Chrome 61 added one for top-level navigations ( PageLoad.Navigation.RedirectChainLength in https://cs.chromium.org/chromium/src/chrome/browser/page_load_metrics/observers/core_page_load_metrics_observer.cc?rcl=413e72f9569c5b502335883143d9b96f6cd1474c&l=264 ), but that'll take a while before we get representative samples, and wouldn't count sub-resources.

I suspect given the HSTS+Extensions implications, we'd probably want to measure to see what our breakage rate would be. Even from those preliminary results (from Beta users), we do see that there's a non-zero (roughly .003% from the current dataset) of users with >16 redirects in the top-level navigation.

[wanderview]

I don't see any telemetry probes for the keyword "redirect" on https://telemetry.mozilla.org.

[sleevi]

So, I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=755117 to measure this, given our differences re: extensions, to assess the compat risk. This will help gather data about our compat risk and see if we anticipate any risk with changing to 16.

It may also be worthwhile for Chrome and Firefox folks to discuss the implications of extensions/HSTS counting in that limit. I think from a Chrome side, it keeps the code much simpler "as implemented" (hence why we did it), but it potentially represents an observable compat difference for sites, since a page load that might work in Safari (with no extensions) might fail in Chrome (with extensions). I'm not sure if we care about predictability to that degree of specificity, hence I didn't directly open a bug, but if folks say "Huh, good point", then we should track that somewhere :)

[annevk]

@sleevi it does seem important to track, especially given HSTS and Upgrade-Insecure-Request. @mikewest at some point asked the standard to change its model to have these kind of "internal redirects", but didn't follow up on the questions I asked: https://github.com/whatwg/fetch/issues/324. What the specification does seems simpler to me, but I might be missing something.

[sleevi]

@annevk Exposing internal redirects significantly complicates the API surface. I realize that's an "implementor concern", but one pragmatically chosen to maintain a well-defined separation of layering between components. We've evaluated tracking this in several other contexts/feature bugs, and somewhat consistently arrived at the conclusion that the complexity and cognitive overhead of the API doesn't match how our consumers (in Chrome) use it.

From a server viewpoint, as @mnot mentioned, servers should be prepared for clients that accept less, and I also totally appreciate an argument that "extensions can change how Fetch is observed". So I don't _think_ it's terribly important that we track, but I also totally appreciate that I may be missing some considerations.

Anyways, the metrics bug should get us some data points in a few releases (looking at the calendar, looks like Chrome 62). Hopefully Firefox will considering adding similar telemetry and we can see if reducing to 16 is viable :)

[annevk]

I don't care much for extensions, but I do care about HSTS and Upgrade-Insecure-Requests which are not extensions.

[wanderview]

We use internal redirects to do things like "reset the network request" after a service worker decides not to intercept, etc. Probably a fair number of internal redirects in firefox.

[domenic]

Naive question for implementers, is it possible for internal redirects to not increase the redirect-count, at least as compared to the limit? (Or, equivalently, for internal redirects to increase the limit, for that fetch at least.)

(My perspective is just that it sure would be nice if everyone was observably the same :) )

[wanderview]

Naive question for implementers, is it possible for internal redirects to not increase the redirect-count, at least as compared to the limit?

Yea, we could probably do that. We definitely have all the information in the same place. We just have to check the flag.

[sleevi]

@domenic No, because for Chrome, we intentionally do not make a distinction between internal and external redirects, because as an API concept, they do not align with the design or goals of the network stack (and it's use across multiple Google products).

I can understand the principle of wanting things observably the same, although I suppose I disagree with it as a general rule (I don't think every API and behaviour should be overly specified, as otherwise that adds unreasonable overhead and stop energy towards evolving and experimenting). Given that the specs explicitly call it out as an implementation specific behaviour, I think it may be overspecified in Fetch to include such a limit, but I don't have the energy to argue for that separation, just that we don't further constrain things without good justification :)

[domenic]

Well, it sounds like we maybe able to get this to be an everyone-but-Chrome standard, at least...

[sleevi]

@domenic That reads as unnecessarily snarky and hostile, but I could be misreading your tone.

In the interest of finding a solution that works for everyone, rather than change for change sake, could you perhaps elaborate why all redirects shouldn't be counted as just that - redirects? Treating HSTS towards the limit does not seem wrong or incorrect, perhaps you could share if you feel otherwise and why?

[domenic]

Maybe I am conflating Firefox's internal redirects and Chrome's, but it sounded like internal redirects were very implementation-specific, and not limited to well-known cases that everyone would interoperate on.

Reading up, it does seem you are consistently saying "HSTS+extensions". So apart from extensions, is it indeed only HSTS redirects? If so then we may indeed be in a slightly better position, although it sounds like Safari and Chrome's network stacks are somewhat opposed and neither are easy to change...

[sleevi]

@domenic I don't think they're opposed. Chrome's limit of 20 is much easier to change than Safari's, provided we get metrics on risk. It's unclear the overhead of changing HSTS to be counted as redirects for purposes of Safari - @cdumez hasn't weighed in on that, and I don't want to assume. So it's possible that if we _don't_ change to a segmented internal/external redirect, and given that two implementations don't (presently) make that distinction, we could potentially align on a 16-redirect limit and call it done. That's why I did not understand the advocacy for an internal/external split - it's something we could potentially do without.

That said, I think it's also useful to understand what impact, if any, that internal/external split has on the Web Platform. To the extent the Fetch spec represents "API docs", I'm totally on board with documenting API contracts, but I don't think we should overly specify those API contracts in a way that prevents us from making changes, integrations, or improvements. That's why I think over-specifying redirects would be undesirable, and it doesn't sound like we have a compelling reason to make that split other than "We should document that" - that is, it's unclear how or why that should cause different behaviour.

@annevk identified one other - which is Upgrade-Insecure-Requests. But it's unclear to me what, if any, benefit is derived from having HSTS or U-I-R not treated as redirects. We could equally specify that the application of these counts as a redirect for purposes of Fetch's limit, and be done with it.

So the concrete question is:

• Would implementations have issue with treating HSTS and U-I-R (e.g. specified bits, as opposed to internal implementation details) as counting as redirects towards the limit? Would that present challenges for implementations commiserate with the challenges of splitting internal/external?

[cdumez]

Upgrade-Insecure-Requests happen at WebKit-level and we could probably easily count them towards a WebKit-level max redirect count if we wanted to. Given that CFNetwork would go to up to 16 HTTP redirects, WebKit can artificially limit further by taking into account U-I-R.

HSTS is another issue though. I believe this happens below WebKit, at CFNetwork level. I'll ask but I am not sure WebKit is aware of HSTS currently.

[annevk]

Well, for Firefox there's also the service worker not handling the request per @wanderview. And I wouldn't count on all that being exhaustive.

Also, I don't understand how these internal redirects in Chrome are not a little bit special. It seems CORS would break if they're not.

[sleevi]

@annevk Our network stack takes a different approach than Firefox, and is more similar to the CFNetwork+WebKit split or WinHTTP+Edge split - two separate layers. HSTS is implemented by our //net layer, and is thus opaque in handling and processing to the Chrome (and, from the POV of Chrome, Fetch) layer. This is where the opposition to internal/external comes from - it is, from an API perspective, exposing an implementation detail of our networking stack, and one whose ontology differs based on different consumers (e.g. whether a given redirect is injected from a higher layer - e.g. U-I-R or extensions - or comes from a lower layer - like HSTS - can result in different categorizations based on different //net consumers' needs). This is further amplified by future steps Chrome is taking to move its networking into a dedicated sandboxed process (and potentially consumed by non-Chrome applications)

In evaluating whether or not to make this split (which affects a variety of features that have redirect-specific behaviours, such as our omnibox search logic), we've consistently decided that treating them holistically is more desirable and more understandable for the various //net consumers. Thus, we've allowed a small CORS layering violation to be introduced into //net, rather than force all //net consumers to be aware of the ontology (or to support extending the ontology, as necessary for some of these consumers). That logic is in https://cs.chromium.org/chromium/src/net/url_request/url_request_redirect_job.cc?rcl=49b1594550778be788e86953fe6d17401e96b5f8&l=100 (while that code is shared by some, but not all, of internally generated redirects), as detailed in https://chromium.googlesource.com/chromium/src/+/4e0be1f338c3bb9bb3c347cdb4c5e3b7944c5c67

[cdumez]

Not considering the the difference in what counts as a redirect in various engines for a second, and given @mnot 's comment from earlier. I'll mention that if the limit of 20 were to be spec'd in an HTTP RFC, it would probably be easier to get CFNetwork updated. The limit of 20 in the Fetch spec is unfortunately not a very strong argument to get such change done in our lower networking stack level.

Otherwise, I think Chrome gathering data on how safe it would be for them to lower their limit to 16 sounds like a good idea. Just note when adding such logging that we'd want to know if users experience over 16 redirect but the load actually succeeds (i.e. total redirect count was not over 20).

[sleevi]

@cdumez Yeah, my plan was to count the final redirect count for successful loads and measure that - which would give an indication for loads in the 17, 18, 19, and 20 range (which would stop working), while naturally wouldn't report for redirect counts >20 (which would count as failed). Did I correctly understand your remarks?

[cdumez]

Yes

[wanderview]

FYI, I am in the process of moving our limit on "internal" redirects to be separate from the "real" redirect count. This would make it easier to align our redirect limit with other browsers in the future if that is decided.

My work is in:

https://bugzilla.mozilla.org/show_bug.cgi?id=1405739

[sleevi]

Looking at a 28DA of Chrome 62 users

| Total Redirect Count | Individual Percentage | User Breakage Percentage |
|------------------|-------------|---------|
| 0 | 97.95675 | - |
| 1 | 01.62609 | 2.04310 |
| 2 | 00.28255 | 0.41622 |
| 3 | 00.10431 | 0.13367 |
| 4 | 00.01719 | 0.02935 |
| 5 | 00.00582 | 0.02159 |
| 6 | 00.00284 | 0.00634 |
| 7 | 00.00086 | 0.00350 |
| 8 | 00.00076 | 0.00264 |
| 9 | 00.00043 | 0.00188 |
| 10 | 00.00044 | 0.00145 |
| 11 | 00.00018 | 0.00101 |
| 12 | 00.00019 | 0.00083 |
| 13 | 00.00014 | 0.00064 |
| 14 | 00.00012 | 0.00050 |
| 15 | 00.00017 | 0.00038 |
| 16 | 00.00008 | 0.00021 |
| 17 | 00.00009 | 0.00013 |
| 18 | 00.00003 | 0.00004 |
| 19 | 00.00002 | 0.00002 |

These counts are Beta, and do include internal redirects. I should have better numbers in ~10 weeks (once 62 is released to stable and has time to widely distribute and bake)

Based on Blink principles of web compatibility, dropping the max below 11 is probably at the "larger compatibility threshold" (it's > 'small-but-non-trivial'). Dropping to 6 would break ~ 0.006% of Chrome loads.

[sleevi]

@wanderview @annevk Now that Chrome 62 has been on Stable for ~3 weeks, I went back and revisited these numbers over a 28DA for our stable users (e.g. including 'early' stable). Looking at just the buckets under discussion, I end up with

| Redirect Limit | Estimated Breakage percentage |
|---------------|------------|
| 16 | 0.00037 |
| 17 | 0.00024 |
| 18 | 0.00009 |
| 19 | 0.00004 |

So the numbers do show that, when factoring in a more broad set of users, there is a greater chance of breakage, but these numbers seem within the thresholds that Chromium has generally been happy with for deprecations.

Notably, this is still not distinguishing internal vs network redirects (which is not easily accomplished with our current architecture).

I'm curious if Chrome were to look at aligning with Safari, what the sense is from Firefox about similarly aligning, so 16 redirects. I'm also curious for folks sentiments regarding the internal v network - I don't actually think that's a thing that necessarily needs alignment, since we're talking about things such as, say, extensions, which already carve holes into the assurances of the platform (e.g. CSP)

[wanderview]

I think we would need to ask the network team about this. @mcmanus @valenting @mayhemer, what do think about aligning with other browsers at a 16 redirect limit instead of 20? (And Dragana, but I don't know her github handle.)

Note, we now track internal redirects separately from external redirects, so we can more easily change the redirect limit without breaking features using internal redirects.

[annevk]

@sleevi I agree that extensions are not important when it comes to internal redirects, but HSTS, service workers, etc. are.

cc @ddragana

[sleevi]

@wanderview Any updates?

[ddragana]

@sleevi do we have any newer stats on this https://github.com/whatwg/fetch/issues/576#issuecomment-342990393

If data looks similar I am ok with changing the limit.

[sleevi]

@ddragana It looks like for Chrome, there's generally more redirects these days. I have not done a deeper analysis as to when or why that transition may have occurred, such as internal redirects, mor HSTS, etc, but continuing to use the same methodology in https://github.com/whatwg/fetch/issues/576#issuecomment-342990393 (28DAU, Stable, all Chrome platforms), it looks like:

| Redirect Limit | Estimated # of Loads Broken, as percentage (0-100) |
| -- | -- |
| 16 | 0.00244% |
| 17 | 0.00231% |
| 18 | 0.00213% |
| 19 | 0.00177% |

You can work those numbers backwards - that is, 0.00177% of loads have 20 redirects. Setting the limit to 16 would break every load that used 17, 18, 19, or 20 redirects - hence the .00244% aggregate increase in URLs that would fail to load.

[annevk]

It also seems like there's still the question of "internal redirects" vs "HTTP redirects" and without having metrics on those the above numbers don't paint a complete picture. E.g., if there are reliable 5 internal redirects of the kind that would be counted outside CFNetwork when you get close to 20 lowering the overall limit would in fact be practical.

We'd also need to clearly identify the "internal redirects" that would have to be counted. (From that perspective I'm somewhat more aligned with Domenic above in that I think it would be better if we only counted network redirects, but in the end that's not what matters.)

[ddragana]

If we get number of only network redirects, would it be possible to align all browsers?
@sleevi in one of your comment above you wrote that having a limit for only network redirects is not possible.

[sleevi]

@ddragana I'm not sure I understand your question/comment?

I'm probably not a good person for this, as I've transitioned off of Chrome Networking, but as explained in https://github.com/whatwg/fetch/issues/576#issuecomment-322244707 , the layering does not really lend itself to tracking internal-redirects from network-redirects, throughout the stack. So we don't have metrics that can measure network-only requests, nor can we presently implement a network-only redirect limit. And for something like HSTS, it's entirely reasonable to question "what is a network redirect", and then that also interacts with understanding extensions that try to be HSTS-like and how they should be treated. Certainly, trying to do something "network-only" will be much more work, and I wouldn't know who would own that work now.

[ddragana]

That was my question has something change in regard Chrome being able to implement this. We can add telemetry to collect number of network and no-network redistrict in Firefox. We can do that if we want some data, my question is can we then agree and implement the change in all browsers.
Anyway we will need someone from Chrome Networking.

[sleevi]

@kinu @tyoshino Given that some of the complexities are due to things like SXG, service workers, and the loader stack, would y’all be suited to triage and comment on https://github.com/whatwg/fetch/issues/576#issuecomment-588143486 ?

I’m not sure the criteria are clear for a network redirect, and the intended interactions between Web Platform features, but y’all might be more familiar about the sharper platform edges and ways to distinguish?

@ddragana The complexity and effort is going to be gated on what’s defined and the criteria (so that new features have a chance of behaving predictably). Perhaps you could document the logic Firefox uses to determine the distinction?

[annevk]

I would posit that the criteria for a network-level redirect are one resulting from an HTTP response with a redirect status.

[sleevi]

So does that mean there would be no bound on SW-level redirect loop issues? And possibly doubling the time it took to redirect loop for HSTS redirects? I thought the intent was to separate out implementation-detail type redirects (such as signed exchanges or extensions) from web platform, but that seems to be far more restrictive? Do you have a cap for those issues?

[annevk]

Ah, I should elide HTTP from what I wrote, but HSTS is not commonly understood as a redirect I think.

[kinu]

Reg: Chrome's latest impl...

when we say "SW-level redirect" does it basically refer to 'reset the network request" after a service worker decides not to intercept' (as in https://github.com/whatwg/fetch/issues/576#issuecomment-322205164)? If so in the current code we don't count them as redirects. (Hopefully it matches with what https://github.com/whatwg/fetch/issues/576#issuecomment-588222721 says)

Reg: the question in https://github.com/whatwg/fetch/issues/576#issuecomment-588143486 about whether Chrome can get numbers for network-level redirects and others: now that Chrome more clearly splits network stack and other layers than before it's possible to count the # of network responses with a redirect status. One caveat is that it doesn't include redirect "responses" coming from service workers, because now the part runs outside the network stack.

[ddragana]

I was looking a bit through Firefox code.
non-internal redirects are http response redirects, service worker and HSTS. Anything else is internal.
@kinu My question was, if it will be possible to implement "only-network-redirect" limit if we decide to do so.

"only-network-redirect" would be a redirect resulting from a response with redirect status (suggested here).

Whether we should count HSTS as a redirect or not, I would say we should not include it, but I do not have a strong opinion. It cannot cause any loops.

[annevk]

@lbherrera found a new attack vector using this limit. In particular, because there is this limit one can figure out how many redirect hops an endpoint goes through.

Before redirecting to the victim endpoint you would first redirect limit - 1, then limit - 2, etc. on an attacker endpoint and observe differences in the handling of the response (network error vs not).

This means that response's URL list's size is exposed.

Questions:

1. Should we attempt to tackle this?
2. Do people have proposals for how to tackle this?

[valenting]

2\. Do people have proposals for how to tackle this?

We could have a random number of redirects in the 20-25 range? :)

[sleevi]

@lbherrera found a new attack vector using this limit. In particular, because there is this limit one can figure out how many redirect hops an endpoint goes through.

I thought we’d already generalized this to being that any resource limit (memory, response size, bandwidth, connections, ports, etc), whether per-origin or system wide, can be used as an XSLeak, but it is always interesting to see examples of it.

I’m not sure we should tackle it in this issue, since short of removing all limits, we’ll still need some value for a limit, even if that expression needs to change (which is unclear to me if it does / what the risk of the URL list size is). Perhaps we should tackle it separately?

[wanderview]

If its a no-cors request then how can they tell the difference between an opaque response and a network error? If its a cors request, then hasn't the cross-origin server opted in to leaking this information?

[lbherrera]

@wanderview A network error will throw an exception which you can catch, whereas an opaque response won't.

[wanderview]

Well, we could make the redirect limit not reject and return an error response instead. Then it would be indistinguishable for opaque responses.