Fetch: investigate data: URL quoted parameters

[bmeck]

Currently the data: URL processor splits the MIME from the body on the first ,. However, MIME parameters, do parse , within parameter values. While it a collision would continue to exist for unquoted parameters that is unlikely to be fixed, investigation into if supporting quoted parameters with , within their values is potentially useful.

[domenic]

In particular, the question is what the data: URL processor should do for cases like data:fizz/buzz;lang="foo,bar",hi. The current spec would give a body of bar",hi and a lang of foo, but you might expect a lang of foo,bar and a body of hi.

[bmeck]

This might affect any attempt to create a MIME API in JS such as https://github.com/bmeck/node-proposal-mime-api since assigning parameter values containing , could not properly round trip through data: URLs even using the quoted form of parameters

[gibson042]

IMO, the Fetch spec is wrong to silently alter data:… semantics specified by RFC 2397. That document did not encourage quoted-string media type parameter values, but it did not prohibit them either, instead deferring to RFC 2396 (which disallows " in _URIs_, as does its replacement 3986). So there are no grounds for interpreting the body of data:application/example;lang="foo,bar",baz as bar",baz, although rejecting it at the _URL level_ for being invalid would make sense if not for the tolerant behavior there (e.g., data:application/json,"" is already accepted). If at all possible (:crossed_fingers:), https://fetch.spec.whatwg.org/#data-url-processor step 5 should be changed to accommodate media type parameters with quoted-string values that contain commas.

As for round-tripping, «_parameter values should use the URL Escaped encoding instead of quoted string if the parameter values contain any "tspecial"_» anyway, so the canonical representation for my example would be data:application/example;lang=foo%2Cbar,baz. But proper treatment of that also requires changing https://fetch.spec.whatwg.org/#data-url-processor , specifically updating step 13 to be aware of percent-encoding in _mimeType_.

[annevk]

Pretty sure this was considered as there's test coverage for this in https://github.com/web-platform-tests/wpt/blob/master/fetch/data-urls/resources/data-urls.json, e.g., "data:text/plain;a=\",\",X". It's unfortunate, but in practice this hasn't been a problem.

[domenic]

According to https://wpt.fyi/results/fetch/data-urls/processing.any.html?label=master&label=experimental&aligned it looks like Safari matches the expectation and other browsers give a result of text/plain, whereas none of them give something like text/plain;a=",".

[annevk]

Sure, they don't pass the MIME type assert, but I'm pretty sure they pass the contents assert.

[bmeck]

@gibson042 it seems across the board pre-processing and doing a URI encoding allows round tripping to work:

fetch(
  `data:application/json;charset=${encodeURIComponent(`"has,comma"`)},0`, {}
).then(
  async r => console.dir({
     contentType: r.headers.get('content-type'),
     body: await r.text()
  })
);

this appears to match the recommendation in RFC2397 section 3.

Given that this matches the original RFC and allows round tripping I'm not sure we actually need to do anything.

[gibson042]

I don't think that conforms with the RFC, which states that «parameter values should use the URL Escaped encoding instead of quoted string if the parameter values contain any "tspecial".» But that does leave some ambiguity about how to actually use quoted strings , and it's certainly more consistent to keep and percent-encode the double quote characters as you have done. I think I'm going to submit an erratum.

[bmeck]

@gibson042 we might also want an explicit note that things should be URL percent decoded in https://fetch.spec.whatwg.org/#data-url-processor

[gibson042]

Yes, that was my second suggestion above.

[gibson042]

Section 5 makes pretty clear that the intent was to excluded quoted-string as a concept from "data:" URLs, since "_it would not easily yield valid URLs without additional %xx encoding, which itself is sufficient_". So I retract the first part of my comment above, but remain convinced that the canonical representation of a media type parameter value including a comma is foo=bar%2Cbaz rather than foo=%22bar%2Cbaz%22 and have submitted https://www.rfc-editor.org/errata/eid5923 to clarify.

[bmeck]

@gibson042 it seems like adding that to the algorithm itself would need to put the decoding in https://mimesniff.spec.whatwg.org/#parse-a-mime-type somehow, which would be a new flag or something and needs to be checked/tests added to handle a few odd cases:

encoded =

data:text/javascript;charset%3DUTF-8;

quoted encoded characters

data:text/javascript;charset="UTF%5C-8";

mismatched quote encoding

data:text/javascript;charset="charset=UTF-8%22;

This mostly seems an order of operations clarification but it feels like the URL encoding is intertwined with the mime parsing here.

[annevk]

I'm inclined to close this as I don't think there's an issue with Fetch. If there's a test demonstrating otherwise please do say.

[gibson042]

data:text/javascript;charset%3DUTF-8,

The ABNF is parameter := attribute "=" value (i.e., with a literal =), so that's invalid and specifically distinct from data:text/javascript;charset=UTF-8;. Percent-decoding in data: URLs applies between defined delimiters, not before identifying them.

data:text/javascript;charset="UTF%5C-8",

If https://www.rfc-editor.org/errata/eid5923 is accepted, that will also be invalid. But right now, an argument could be made that it corresponds to Content-Type: text/javascript; charset="UTF\-8" (which, per the rules of quoted-pair, has a charset parameter value of "UTF-8" anyway).

data:text/javascript;charset="charset=UTF-8%22,

Likewise here, except that the argument before https://www.rfc-editor.org/errata/eid5923 would be a correspondence with Content-Type: text/javascript; charset="charset=UTF-8".

I'm inclined to close this as I don't think there's an issue with Fetch. If there's a test demonstrating otherwise please do say.

@annevk Given how its algorithms seem to ignore these media type parameters for all purposes other than echoing them back (and in particular uses UTF-8 regardless of charset), I think Fetch can be left as-is and the necessary processing applied in higher levels (albeit redundantly so). However, there certainly are browser-inconsistent bugs from failure to properly percent-decode per RFC 2397: https://jsfiddle.net/7m5dtuk6/

[annevk]

I'm confused. Browsers don't follow any RFCs for any of this. I'm not sure how they're relevant.

[masinter]

If you were following RFC 3986

content-type: application/example;lang="foo,bar"

body

would turn into

data:application/example;lang=%22foo%2Cbar%22,body

but

data:application/example;lang=foo%2Cbar,body

would be allowed, because the MIME specification defines quoted strings as equivalent to the string with the initial and final quote omitted.

Percent-encoding is used for multiple purposes: it is used to escape (RFC3986) reserved characters to avoid confusion when it is not used for the reserved purpose (such as in this case where the comma is used to delimit the body) and also to encode characters that are (in RFC 3986) not allowed.

To fit into the WHATWG URL standard, though, if double quote is allowed, then

data:application/example;lang="foo%2Cbar",body

would also be equivalent.

The content-type string is itself a linearization of a data structure which allows surrounding double quotes for parameter values, and requires them when the parameter value contains any character in (RFC 2045) tspecial, which includes
double quote.

data:application/example;lang=%22foo%22baz%22,body

would correspond to

content-type: application/example;lang="foo"bar"

since quoted string allows embedded " marks.

Section 5 of RFC2397 is "History" and the discussion of "quoted-printable" is about an entirely different topic, explaining the use of base64 value for content-transfer-encoding (see RFC 2045 section 6.1). I imagined you might consider adding some kind of compression values instead of base64, for example.

[gibson042]

data:application/example;lang=%22foo%22baz%22,body

would correspond to

content-type: application/example;lang="foo"bar"

since quoted string allows embedded " marks.

The latter is clearly invalid per both RFC 2045 and the more recent (and more directly Fetch-relevant) RFC 7231, so does that mean the former is also invalid? And further, does that mean that it would be valid with double encoding as data:application/example;lang=%22foo%2522baz%22,body (which has its own undesirable consequences), or is it just impossible to include " in a parameter?

It seems like your mental model requires %xx to sometimes be interpreted as a delimiter and other times to be interpreted as content, even when all of the preceding data is identical. Does it also include quoted-pair backslash escapes (and if so, can the backslash itself be encoded as %5C)? Either way, it's bringing in a whole lot of parsing complexity that wouldn't be necessary if the commentary of section 5 about "%xx encoding, which itself is sufficient" were embraced for media type parameter values as proposed by https://www.rfc-editor.org/errata/eid5923 .

[annevk]

Closing per above. Let me know if there is a problem with Fetch after all.