Manifest: Why does obtaining not check a MIME type?

[annevk]

We should really enforce MIME types for new formats.

[marcoscaceres]

“New” is kinda relative: The format has been in browsers for quite a few years (~5 years), but we could definitely enforce it. However, we might have to check if there would be a lot of breakage. I suspect there would be.

[annevk]

Yeah, I still consider WebVTT new. 😊

[marcoscaceres]

@foolip, just wondering if you might know, are HTTP headers returned from <link rel="manifest"> something we could query from HTTP Archive? Or would we need to set up browser telemetry instead?

Trying to ascertain if it's feasible to enforce the MIME type for Web Manifest at this stage.

[foolip]

@marcoscaceres do you have an example of a site that has a manifest? Given that I could check what it looks like in httparchive. If it’s fetched by the browser during page load I’d expect it to be recorded with headers, but I predict the problem will be telling which fetches are for manifests, they’re probably in a misc category...

[marcoscaceres]

@foolip, https://mobile.twitter.com has one. Some other ones, in case:
https://pwa.rocks

[marcoscaceres]

Irrespective of what we find, it fairly likely the majority of manifests are going to be served as "application/json", so why don't we just support both? We could then enforce a MIME check on either "application/json" or "application/manifest+json" with limited breakage.

[annevk]

I'd be okay with using https://mimesniff.spec.whatwg.org/#json-mime-type here, but perhaps being slightly stricter as you propose is better for forward compatibility.

[marcoscaceres]

JSON MIME type seems like a nice fit, tho I was taken a back a little bit about how liberal it is... but I kinda like it :) Ok, let's got with that and the stricter one... and then see if we can shift people over to "doing the right thing"™️.

[marcoscaceres]

Claiming the bug, but would really like @contrepoint to work on it when she can (as it's a really "good first issue" + requires some simple changes to browser code).

[benfrancis]

Did anyone manage to find any data on MIME types currently being used in the wild?

I remember that when we enforced a new MIME type on mozApps on Firefox OS it caused a surprising amount of pain for developers confused about why their apps didn't work, e.g. on web hosting services where the developer didn't necessarily have control over the MIME type used to serve a resource (possibly less of a problem now?). This was previously discussed in the sysapps working group and I believe at the time Gecko would only enforce the MIME type for cross-origin app installs, but that was when there was an explicit install() API.

I realise the restriction in https://github.com/w3c/manifest/pull/858 is more lenient as it appears to accept application/json as well, but I'm curious how many manifests are currently served with the wrong MIME type (e.g. text/plain or text/html)? Flipboard appears to use text/plain to serve https://s.flipboard.com/webapp/images/favicon/site.webmanifest for example.

FWIW I don't think that enforcing a JSON MIME type is necessarily a bad idea, I'd just suggest there should probably be a warning in the specification that this is a newly enforced requirement and that some existing applications may not serve their manifest with the expected MIME type.

[marcoscaceres]

FWIW I don't think that enforcing a JSON MIME type is necessarily a bad idea, I'd just suggest there should probably be a warning in the specification that this is a newly enforced requirement and that some existing applications may not serve their manifest with the expected MIME type.

I don't mind adding a warning to the spec if you think it will help.

[dominickng]

Having discussed with @mgiuca, we are quite unlikely to add this check in Chromium.

1. If we enforce a MIME type, we can't see what the benefits are for imposing more pain on developers, or potentially putting them in a situation they can't easily fix if they don't have control over the served MIME type.
2. If we just add a warning, what is the path to removing that warning? Again, if developers can't easily fix the situation, we won't ever be able to move to enforcement, and we've just added noise.

[marcoscaceres]

@dominickng, @mgiuca, your position is totally reasonable. Think we (Mozilla) are willing to do strict content type checking, in conformance with "JSON Mime type". I've just sent a patch to enforce the check in Gecko.

What ultimately goes into the spec I guess now depends on what WebKit does (or wants to do), as they are the tie breaker. @rniwa?

[mikewest]

@dominickng, @mgiuca: Did we add any metrics to Chrome? Generally, we'd like to get away from MIME sniffing, both for aesthetic reasons and to create more robust security boundaries (CORB, for example, since AFAIR, Chromium parses the manifest file in the renderer). Before rejecting the suggestion out of hand, it would be helpful to see numbers.

[mgiuca]

@mikewest AFAIK we aren't "mime sniffing" here, we're just assuming that when an HTML file links to a manifest, it is a JSON file, without looking at the MIME type. (We aren't trying to "guess" the MIME type from its content or anything, we just ignore the MIME type.)

How would checking the MIME type help with security? We still have to parse a binary file that may or may not be valid JSON / a valid manifest.

I don't think we have metrics on the MIME type of the manifest files. @dmurph do you think it's worth adding this in Chrome?

[mikewest]

@mgiuca: I may be misunderstanding the architecture, but my concern here is that ignoring the MIME type allows an attacker to bring any resource into the renderer, and therefore exposes it to Spectre, etc (similar to the description in https://chromium.googlesource.com/chromium/src/+/master/docs/security/side-channel-threat-model.md#incompleteness-of-corb). This risk is reduced if we only bring resources into the renderer that claim to be reasonable for the kind of request that's being made; we could reject text/html and application/octet-stream out of hand, for instance, rather than attempting to parse them.

[mgiuca]

So is the concrete example here that a manifest that's served with application/manifest+json could not be fetched from an <img> element, for example, which means a site that allows users to create arbitrary <img> elements but not inject other HTML would be unable to trigger that manifest from being fetched cross-origin? But a manifest served with image/png could be fetched from an <img> element? Thus we want to make sure manifests served with image/png are not permitted to be used as manifests?

If so, I guess that makes sense. We should do a survey of the manifest MIME types. I think @dmurph is the right person to prioritize this. My suspicion is that since this has never been enforced, close to 0% of manifests will be served with application/manifest+json. Since most manifests have a .json file extension, I assume a large percentage of web servers will automatically serve them with the application/json MIME type, so it might be acceptable if we require one of those two types.

[mikewest]

The example you've provided is a good one, but I think the problem also exists in the manifest parser itself.

This is likely getting into "Chromium" vs "Standard" discussion, so I'm happy to continue it elsewhere, but, it looks to me like we're parsing the manifest file in the renderer (e.g. https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/renderer/modules/manifest/manifest_parser.h). If that's the case, then it appears to me that an attacker could add something like <link rel="manifest" href="https://super-secret-intranet-server/important-data.xml"> to a page, and thereby bring Important Intranet Data into a process they control. We'd parse the data, say "Huh. This isn't JSON!" and go about our business, but the data would have already entered the process, and can presumably be attacked from there.

(@anforowicz might also be interested in this conversation from a CORB perspective.)

[marcoscaceres]

To be clear, this is what the proposal is allowing:

A JSON MIME type is any MIME type whose subtype ends in "+json" or whose essence is "application/json" or "text/json".

[dominickng]

@mgiuca and I took a look in HTTPArchive, specifically the summary_requests.2020_03_01_mobile table with the following queries:

SELECT resp_content_type, COUNT(resp_content_type) AS total
FROM [httparchive:summary_requests.2020_03_01_mobile]
WHERE url LIKE "%webmanifest"
GROUP BY resp_content_type ORDER BY total DESC

SELECT resp_content_type, COUNT(resp_content_type) AS total
FROM [httparchive:summary_requests.2020_03_01_mobile]
WHERE url LIKE "%manifest.json"
GROUP BY resp_content_type ORDER BY total DESC

Note that this is a bit deficient: it's only capturing manifests that are named *manifest.json or *webmanifest, but hopefully it's representative enough. We found the following:

• only 201,867 out of 281,507 total requests (71.7%) are served with a "JSON essence MIME type". The rest use MIME types that don't end in +json at all
• only 13,405 out of 281,507 total requests (4.8%) are served with the _correct_ MIME type of application/manifest+json
  
  
  
  • only 11,356 out of 68,784 total requests (16.5%) for a *webmanifest file are served with the _correct_ MIME type
  
  
• 28,484 out of 68,784 total requests (41.4%) for a *webmanifest file are served with _no_ MIME type whatsoever

As written, the proposal breaks 28.3% of manifests in the query, which is a lot. It's also interesting that over 80% of folks who are using the recommended webmanifest file extension do not have their web server configured to also send the recommended MIME type for that extension.

@mikewest, achieving good MIME type hygiene is a great goal, and I agree with the aim of locking down as many file types here as possible. However, I think nearly 30% breakage is very high from a spec and implementation perspective, and I stated other reasons for doubting whether a warning / deprecation would be practical in https://github.com/w3c/manifest/issues/821#issuecomment-609539499

Based on these numbers, I think changing the spec is impractical at the current time. I'd welcome other people doing some analysis to make sure I didn't miss anything.

[marcoscaceres]

We might be able to address a large % of these if we can get .webmanifest recognized by the common web server software. For example, we are missing from:

https://github.com/nginx/nginx/blob/master/conf/mime.types

And:
https://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types

[dominickng]

Fixing .webmanifest completely would take us to 91.8% of manifests served with a JSON-like MIME type, with 8.2% still broken. The still broken ones are manifests served as *manifest.json but not with a JSON-like MIME type.

That is an absolute upper bound that assumes a perfect upgrade by devs to the up to date MIME type for .webmanifest. Is that realistic?

[marcoscaceres]

Is that realistic?

Could be... we could maybe broaden the acceptable types for legacy reasons if it captures a large enough %? (though that's a little eek, as the "JSON MIME Type" is already fairly broad).

It might also depend if those not serving with the correct mimetype are actual "web apps" worth installing. When I looked at this problem back in 2014, it turned out that the apps claiming to be "installable" were, in large part, not usable at all.

[dmurph]

Is that realistic?

Could be... we could maybe broaden the acceptable types for legacy reasons if it captures a large enough %? (though that's a little eek, as the "JSON MIME Type" is already fairly broad).

It might also depend if those not serving with the correct mimetype are actual "web apps" worth installing. When I looked at this problem back in 2014, it turned out that the apps claiming to be "installable" were, in large part, not usable at all.

I definitely buy that.

I'm also skeptical of relying on sites updating their software, especially for anything that isn't made by a large company. I would really hate to break a long tail of websites. I would guess that having .webmanifest not included in httpd and nginx now means that for the next... decade(?) at least a large amount of sites will not have .webmanifest included.

Maybe you have different data on how often website have their server updated. In my experience 1% is often, 5% sometimes, and 94% never or almost never. Maybe the world has changed though 🤷

@mikewest I think I understand the security concern, and it might be orthogonal to this - I'm guessing we could try to parse this in the browser or utility process if we need to? Perhaps we create a crbug?
Question for you: If we restrict on mime type, does this actually protect this from being parsed in the renderer? Is there some sort of network layer / browser layer validation?

I would personally like to not require enforcing this on the spec side.

[marcoscaceres]

Note, the pull request for this is now part of https://github.com/whatwg/html/pull/5581 instead.

[marcoscaceres]

closed via whatwg/html#5581

[mgiuca]

To recap here: it looks like the above proposal was adopted (in the HTML spec, not the manifest), which I must have missed at the time.

https://html.spec.whatwg.org/multipage/links.html#link-type-manifest

"If response's Content-Type metadata is not a JSON MIME type, then set success to false."

Was this implemented in any browsers? I must say, it's fairly surprising that this was changed, given the above comments by @dominickng and @dmurph ("As written, the proposal breaks 28.3% of manifests ...")

[marcoscaceres]

Was this implemented in any browsers?

I implemented it in Gecko, but haven't shipped it because... reasons. But I'm still planning to once I get unblocked.

I need to check if the WebKit folks will accept a patch.

"As written, the proposal breaks 28.3% of manifests ..."

Yeah, pretty unacceptable level of breakage... it might be worth figuring out an upgrade path, like a console warning when not type is given.