Manifest: Manifest processing should not be a function of document URL

[mgiuca]

The steps for processing a manifest are currently defined as a function of both manifest URL and document URL:

The steps for processing a manifest are given by the following algorithm. The algorithm takes a string text as an argument, which represents a manifest, and a URL manifest URL, which represents the location of the manifest, and a URL document URL.

This means that the entire identity of the app is uniquely determined by the tuple (text, manifest URL, document URL). Though practically, since text can be derived from manifest URL, it means just the pair (manifest URL, document URL).

The problem is document URL. The app can be interpreted differently depending on which document URL the manifest was referenced by. In reality, the document URL is only used in two places:

1. As the default value for "start_url".
2. To check that "scope" is same-origin as document URL.

I maintain that the processing of a manifest should not be a function of the document URL. This means the app will potentially behave differently depending on which page the user was on when they installed the app, and we cannot meaningfully index or install an app from outside of a document (as Microsoft is currently doing).

We should:

1. Remove the default value for "start_url", and require "start_url".
2. Move the check that "scope" is same-origin as document URL outside of the steps for processing the manifest (so if that check fails, it means that the manifest is not valid for that page, as opposed to being an invalid manifest).

This means that the semantics of the manifest becomes a pure function of the manifest URL itself (and of course the manifest text), and thus it will behave the same no matter where you install it from.

Making the above change 1 will technically be a breaking change, but since sites can't rely on the default "start_url" (since it's dynamic) this should not be able to break any expectations.

[ithinkihaveacat]

I think it would be useful to determine how common "versioned" manifests are before making this change, since this will cause problems for existing sites.

One example of a site that will be affected is weather.com, which has manifests at "versioned" URLs such as https://weather.com/weather/assets/manifest.507fcb498f4e29acfeed7596fe002857.json. If the manifest URL uniquely identifies a PWA then a "new" PWA will be created every time weather's manifest is deployed or moved.

More generally, I'm concerned that the realization that different manifest URLs correspond to different PWAs is not obvious. Also, no manifest validator/linter will be able to identify the problem with complete accuracy, and even heuristic based approaches are awkward (see e.g. https://github.com/GoogleChrome/lighthouse/issues/2570).

The failure mode is rather unfortunate too: sites unveiling a new PWA at an (incorrectly) versioned manifest URL will only discover the problem when the manifest is updated and deployed, a situation QA processes are likely to miss.

[mgiuca]

There are two problems. The versioned manifest is not the problem that we're dealing with here.

• Problem 1: Sites change their manifest URL every time they update. This means they effectively are publishing a new app every time they update.
• Problem 2: Sites do not specify start_url in their manifest. This means effectively you get a different app depending on which page you were on when you installed the site.

One example of a site that will be affected is weather.com, which has manifests at "versioned" URLs such as https://weather.com/weather/assets/manifest.507fcb498f4e29acfeed7596fe002857.json. If the manifest URL uniquely identifies a PWA then a "new" PWA will be created every time weather's manifest is deployed or moved.

This is already a problem (Problem 1), and my proposed change doesn't make it any worse. Right now, an app is uniquely identified by the tuple (manifest URL, document URL), so if the manifest URL changes, it's a new app. My proposal is to remove document URL from that tuple. It will still be a problem that if the manifest URL changes, it's a new app.

I don't want to talk too much about Problem 1 here. I agree it's a problem but we're talking about it elsewhere (GoogleChrome/lighthouse#2570, GoogleChromeLabs/gulliver#323).

For Problem 2: I noticed weather.com also doesn't have a start_url, which means according to the manifest spec, it should create a unique app for each page of the site. However it looks like Chrome, at least, considers the manifest invalid if it doesn't have a start_url:

"Site cannot be installed: the start URL in manifest is not valid"

(For my future reference, the Chromium code which does this is in installable_manager.cc where it sets START_URL_NOT_VALID).

Given that this is the case, it seems it would not be breaking any existing expectations if we made start_url mandatory in the manifest spec.

[kenchris]

I am OK with this change. I don't think this should be any issue for the Windows support either @boyofgreen

[mgiuca]

I must imagine (without checking) that Windows support also ignores both of the rules relating to document URL, since they wouldn't have a canonical document URL for their Microsoft Store PWAs. So this change would only be making the spec more in line with Microsoft's (and Chrome's) implementation. @boyofgreen

[marcoscaceres]

I'd be in favor of this change. @mgiuca, could you draft it up?

[mgiuca]

Working on it!

[ithinkihaveacat]

@mgiuca Ah, I didn't realize weather.com was no longer triggering any sort of A2HS prompt; that definitely worked in the past (they were showcased in the PWA tent at Google I/O last year) but either Chrome's behavior has changed, or start_url has been removed from their manifest such that it no longer triggers.

In any case, I agree with this change, and I take your point that this issue mostly separate to the question of how PWAs should be uniquely identified (#586).

[mgiuca]

I've put up a branch (but I can't make a PR until #670 lands; we should continue discussion on #669):
https://github.com/mgiuca/manifest/compare/start-url-required...no-document-url

This one is also a breaking change and may require some discussion. The break is that before, if the start URL did not match the document URL, the start URL (and likely scope) would be set to their default values, and manifest succeeds. Now, the entire manifest fails to load (in the "obtaining a manifest" steps).

This is necessary in order to make the same-origin-as-document check not intrinsic to processing the manifest (allowing the manifest to be processed independent of the document), but rather make it just part of the process of loading a manifest from an HTML page. We can't really just alter the start_url and scope after the manifest is processed; then we're back to the manifest values being dependent on the document URL.

I wonder what Chrome for Android does in this case (since the WebAPKs are built from the manifest, independent of the document URL, they clearly don't alter them to a default start_url if the origin doesn't match --- I wonder if they fail or if they simply allow a cross-origin PWA installation).

[mgiuca]

This (the dependence on the document) goes up a bit higher than I previously thought. I'm trying to update the spec to basically allow a web app to be installed directly through its manifest URL (not from within a document context at all), which is how Microsoft Store PWAs work, as well as Google's WebAPKs, and any future "install app directly from a store page" type of installation. Currently, this isn't really allowed.

In addition to the "steps for processing a manifest" (fixed above):

• The "steps to install the web application" uses:
  
  
  
  • The Window, to fire the appinstalled event (this isn't needed if there is no document).
  
  
  • Optionally using the Document to get metadata if there is no manifest (this is irrelevant when installing an app with a known manifest).
  
  
  • Registering a Service Worker with the Document (this is the hardest thing to fix, but it can surely be done against the start URL or scope instead?).
  
  
• The "steps for obtaining a manifest" uses:
  
  
  
  • The Document, for obtaining the manifest URL (this isn't needed if we're installing directly from a manifest URL).
  
  
  • The Document's origin (terminate if it is an "opaque origin") (this isn't needed if there is no document).
  
  
  • The Document, to check whether it is same-origin as the start_url (this isn't needed if there is no document).
  
  

I think the above algorithms can be rewritten in the following way:

1. The "steps for obtaining a manifest" should be split into two: one for obtaining a manifest URL from a document, and one for obtaining a manifest from a manifest URL.
2. The "steps to install the web application" should similarly have two versions: one that is applied within a top-level browsing context, and one that is given a manifest URL as input.

These direct-install user agents can thus use the non-document versions of these algorithms.

[benfrancis]

Just a note to say that I would strongly support this change.

I'm currently working on a use case for web app manifests, similar to the app store use case, where a web app is installed onto a digital signage display remotely.

In this case the digital signage runtime may not have a traditional browsing context from which the user can install a web application, it instead provides a remote API whereby an administrator can install a single web app to the display, which loads in a single application context.

This use case would be made much simpler if the manifest could be processed directly from its own manifest URL, rather than assuming an app is installed by a user agent fetching a manifest via a link relation in a separate document loaded in a browsing context.

[garygreen]

At it's core isn't the manifest file also to do with installability?... and you need a document to install an app, due to service workers, etc. So isn't this whole concept of seperation of document vs manifest simply impractical? Curious to know. I'm a total noob so feel free to shoot me down 😅

[marcoscaceres]

So isn't this whole concept of seperation of document vs manifest simply impractical?

Mostly, yes. Because things like the content security policy, etc. are governed by the client settings object. This affects, at a minimum, how icons are loaded. But it also affects other defaults (the ones where the document URL is used as a fallback).

On reflection, I think trying to decouple the manifest from the document might not work because they depend on each other too much... it be like decoupling a stylesheet from the documents the style sheet is applied to.

[dmurph]

I think an idea was that the "processing of the manifest" could be separated, not necessarily things like icon fetches, as this would allow entities like crawlers or stores to be able to process the manifest in a spec-conformant way (so that devs can expect this).

I think the main sticking point here for me is the verification of the document_url origin and the start_url origin matching. Based on the pull request by @mgiuca, this would still be done for user agents as they would always pass in document_url, and the processing would then consider it. But a crawler might not have a document, for example.

Going further in the store / indexing / crawler direction, I would imagine that there would be suggestions in the spec about 'verifying start_url' by fetching it to look for a manifest link back to the current manifest being parsed.

Anyways, just some thoughts here. 🤷

[marcoscaceres]

I think the main sticking point here for me is the verification of the document_url origin and the start_url origin matching.

Yeah :(

I'm inclined, for this spec (not for manifest-app-info!), to consider anything that's browser out of scope.

It might still be possible to crawl a manifest, but maybe just ignore browser specific stuff. The limitation being that one might not be able to create a store where a user installs apps directly from.

For App Store installs, I wonder if we should just come up with a new set of fields specific for that. Then we don't need to make the fields in this spec do any special gymnastics to get things to work?

[mgiuca]

You don't need a document to install an app though. Case in point: we have app stores that index an app based on its manifest URL (also enterprise admins can request the installation of an app by specifying its manifest URL).

While you can't register a service worker without loading a document, you can absolutely download icons, register an app with the OS, and prepare it for launching, without downloading any documents. The purpose of this work is that therefore user agents shouldn't be _required_ to modify their interpretation of the app based on the (possibly nonexistent) document URL.

There are two scenarios:

• Installation from a document. Document URL is available, but there could be any number of documents that link the same app.
• Installation without a document, e.g., from an app store. There is no document URL.

In either of these scenarios, the installed app should be the same. Also, I don't want Document X and Document Y to link the same manifest, but result in a different app experience.

The spec currently doesn't actually allow the second scenario at all, so it doesn't reflect reality.

[marcoscaceres]

While you can't register a service worker without loading a document, you can absolutely download icons, register an app with the OS, and prepare it for launching, without downloading any documents.

I don't know if that is entirely true, as it's the document's CSP that governs where icons can be fetched from. For example, this would prevent manifest icons being loaded from a remote origin:

Content-Security-Policy: img-src self

Or

<meta http-equiv="Content-Security-Policy" content="img-src self">

So, loading the icons without respecting the document's CSP wouldn't be amazing.... it could be done, but it violates the server's expectations and raises security concerns (or raises eyebrows, at least).

The purpose of this work is that therefore user agents shouldn't be required to modify their interpretation of the app based on the (possibly nonexistent) document URL.

Ok, but this then become a matter of scope (not manifest "scope"!) of this standardization effort. I.e., for the purpose of standardization of this specification, should the Working Group concern itself with anything but what web browsers do with the manifest?

My opinion has always been no. I'm not discounting the "App Store" use case (which is why we split out app-info), but we've been generally resistant to considering those requirements, as they can conflict with relationship between the Document and manifest as a sub-resource/link-relationship.

Also, I don't want Document X and Document Y to link the same manifest, but result in a different app experience.
The spec currently doesn't actually allow the second scenario at all, so it doesn't reflect reality.

But isn't this an author decision? If the author chooses to omit "scope", they are basically opting into that different experience (with the possible enhancement disambiguation via some identifier along the lines of #586).

[benfrancis]

I don't know if that is entirely true

I agree that the current manifest specification assumes installing a web app from a document but that was a design choice, it didn't have to be specified that way. That design choice unfortunately means that some implementations (e.g. app stores, mobile device management, kiosks) which already use web app manifests today can never be compliant with the specification.

should the Working Group concern itself with anything but what web browsers do with the manifest?

That's a good question. The web app manifest specification differs from other web specifications in that by its nature it can not be limited to just browsers, because it can not be implemented entirely inside a browser engine. It requires interactions between a browser engine, browser chrome, the underlying OS and in some cases external services like app stores and sync services.

My understanding of Progressive Web Apps is that they are all about web apps escaping web browsers. Restricting the scope of the specification to the context of a web browser would seem to fail to meet that primary use case.

I therefore agree in principle with the statement "manifest processing should not be a function of document URL", though in practice untangling the two at this stage would be challenging. Proposals like the proposed approach to unique IDs would further complicate this.

[aarongustafson]

This may be a naive question, but would it be possible to make the document optional so that the parser takes it into consideration if it's available but doesn't if it doesn't exist? Would that be too complicated?

[aarongustafson]

Also… in store/catalog contexts, I would think a start_url would be required, else the manifest would not be tied to a URL at all, right? We could make that a requirement in App Info if folx think it’s worth it.

[marcoscaceres]

This may be a naive question, but would it be possible to make the document optional so that the parser takes it into consideration if it's available but doesn't if it doesn't exist? Would that be too complicated?

The parsing aspect is not complicated, no. It's all the stuff that happens around it that gets complicated. That's why we presume this is coming form a document: because link relationships are from Document (and security/environment) -> linked thing (manifest), but never treated as independent.

We could make that a requirement in App Info if folx think it’s worth it.

Yes. I might make sense to do that. The stores can enforce that policy, while advising developers to also treat a manifest as an totally independent resource (not as a link relationship of a document).

[marcoscaceres]

@benfrancis wrote:

My understanding of Progressive Web Apps is that they are all about web apps escaping web browsers. Restricting the scope of the specification to the context of a web browser would seem to fail to meet that primary use case.

Note that this effort predates that write up, and I don't think there was every wholesale endorsement of PWAs as outlined in that blog post by the standards community (or browser vendors at large). As aspirational a guide as that blog post has been to the community, there are a lot of dissenting opinions, and in some cases, outright rejections, of what is proposes.

Without digressing too much, from the very beginning, the way we designed the "progressive" nature of this particular specification was that developers could add as few or as many manifest members as they wanted in their manifest, without any strict requirement on what they must include.

I therefore agree in principle with the statement "manifest processing should not be a function of document URL", though in practice untangling the two at this stage would be challenging. Proposals like the proposed approach to unique IDs would further complicate this.

Agree. And at this point in this nearly decade long journey, I'd really just like to get the unified base set of functionality standardized. It is, admittedly, a somewhat impoverished subset of the functionality web developers need - but I'm optimistic that we can continue to evolve this specification, along with implementations, in unison.

But I really just want to get the core of it "done" - so we can then incrementally standardize little enhancements to make everyone's lives better.

[garygreen]

@mgiuca

Installation without a document, e.g., from an app store. There is no document URL.
The spec currently doesn't actually allow the second scenario at all, so it doesn't reflect reality.

The reality is a document is required to install an app. Sure, you can INDEX an app (it's name, icons, etc) but to me that is not INSTALLING an app.

The installation of an app requires a document. It often requires a service worker, fetching of js, css, caching pages, numerous other stages during the install lifecyle of the service worker.

To really separate the document from the installation process - wouldn't the manifest file need to link to all the assets needed to install the app? i.e. is this issue trying to help address packaging of an app, and once packaged not really "needing" a document any longer? It seems to me the fact remains a document is needed to practically package and install an app. You could "index" it, but what does that really give you? You've indexed a bunch of icons, name, but not really done anything practical.

A document is an integral part of the manifest to describe where an app sits, how it can be accessed and installed? Maybe I'm misunderstanding the intention here though.

[aarongustafson]

The reality is a document is required to install an app. Sure, you can INDEX an app (it's name, icons, etc) but to me that is not INSTALLING an app.

The installation of an app requires a document.

I was thinking about this very thing. For digital catalogs, the start_url is really required because otherwise there’s no "there" there; you wouldn’t even have a domain to work from.

So, to go back to @mgiuca’s initial recommendation (and this PR), I’m all for making start_url required in Manifest (with a sensible upgrade plan in place, as Matt’s outlined in the PR). Moreover, I feel like the start_url (even the inferred one) should step in as the "document" in order to facilitate installation (see @marcoscaceres’ comment about CORS, etc.).

[marcoscaceres]

So, to go back to @mgiuca’s initial recommendation (and this PR), I’m all for making start_url required in Manifest (with a sensible upgrade plan in place, as Matt’s outlined in the PR).

I personally don't agree with making it required, for the reasons I already stated. But I'd be totally ok with making start_url and scope RECOMMENDED or including them as a "best practice".