Manifest: Make unbounded navigation scope less broad

[kenchris]

Check: https://bugs.chromium.org/p/chromium/issues/detail?id=688888

We don't necessarily want, say "start_url": "https://cnet.com/technology/" turn all of cnet.com into the PWA because scope is undeclared. We should probably restrict to closest surdir.

[pkotwicz]

Chrome currently sets the scope to the closest subdirectory if the scope is not defined.
For instance:
start_url
https://www.example.com/subdir/index.html
default scope
https://www.example.com/subdir/

start_url
https://www.example.com/subdir/?q=1
default scope
https://www.example.com/subdir/

start_url
https://www.example.com/start
default scope
https://www.example.com/

[kenchris]

Currently this is actually defined to be relative to the location of the manifest and not the start_url. I guess that is a discussion that we need to have

@RobDolinMS @Paul_Kinlan @slightlyoff

[pkotwicz]

kenchris:

For the sake of clarity:
Chrome currently uses the manifest URL as the base URL if the scope URL is specified. Thus if the Web Manifest specifies:
start_url="https://www.example.com/subdir/deeper/even_deeper/index.html"
manifest_url="https://www.example.com/subdir/manifest.json"
scope="deeper"

The resolved scope is
"https://www.example.com/subdir/deeper/"

I think that you are suggesting that if the Web Manifest specifies:
start_url="https://www.example.com/subdir/deeper/even_deeper/index.html"
manifest_url="https://www.example.com/subdir/manifest.json"
(no scope is specified)

The resolved scope should be
"https://www.example.com/subdir/"
instead of
"https://www.example.com/subdir/deeper/even_deeper/" which is what Chrome currently computes

[kenchris]

Right with scope declared

let resolved = new URL(scope = "deeper", manifest = "https://www.example.com/subdir/manifest.json").href
resolved === "https://www.example.com/subdir/deeper"

Without scope it would currently be unbounded, which means that the within-scope algorithm will return true (for same origin - just filed an issue about that)

Thus scope would become "/". That is how it is specced currently AFAICT.

My idea was to have the default scope be "." meaning that it would resolve like this:

let resolved = new URL(scope = ".", manifest = "https://www.example.com/subdir/manifest.json").href
resolved === "https://www.example.com/subdir/"

Currently scope is never relative to the start_url:

new URL(scope = ".", manifest = "https://www.example.com/subdir/manifest.json").href
=> "https://www.example.com/subdir/"
new URL(scope = "deeper", manifest = "https://www.example.com/subdir/manifest.json").href
=> "https://www.example.com/subdir/deeper"
new URL(scope = "/", manifest = "https://www.example.com/subdir/manifest.json").href
=> "https://www.example.com/"

so I think it is kind of weird that it suddenly is when undefined, and that that then is the only way to make it.

[pkotwicz]

A coworker ran a script over the internet's Web Manifests. There are:
~3000 Web Manifests where the start page's directory is more top level than the manifest's directory
~ 50 Web Manifests where the manifest's directory is more top level than the start page's directory

I suggest that if the Web Manifest does not have a scope, the scope should default to either:
Proposal A: The directory of the start page
OR
Proposal B: The directory of the start page or the directory of the manifest whichever is topmost. If the start page's directory does not contain the manifest's directory and manifest's directory does not contain the start page's directory, the scope is set to the start page's directory

Thus:

Manifest URL | Start URL | Computed Scope Proposal A | Computed Scope Proposal B
-------------------|---------------|----------------------------------------|-------------------------------------------
https://www.nfl.com/manifest/now.json | https://www.nfl.com/now?icampaign=nfl_now_progressiveap | https://www.nfl.com/ | https://www.nfl.com/
https://www.carthage.edu/manifest.json | https://www.carthage.edu/bridge/ | https://www.carthage.edu/bridge/ | https://www.carthage.edu/ (The manifest is more topmost than the start page)
https://m.wikipedia.org/w/api.php?action=webapp-manifest | https://m.wikipedia.org/wiki/Main_Page | https://m.wikipedia.org/wiki/ | https://m.wikipedia.org/wiki/ (https://m.wikipedia.org/w/ does not contain https://m.wikipedia.org/wiki/ and vice versa. Start page directory wins)

[kenchris]

Any input @marcoscaceres ?

[marcoscaceres]

Sorry, way behind :( trying to catch up this week... 😭🔥💥

[mgiuca]

Resurrecting this issue due to a discussion I had with Marcos in December, and now seems fairly urgent to resolve due to Safari launching manifest support on iOS.

This article explains:

Android browsers typically open URLs within the scope of the PWA context and other links in a browser or a Custom Tab. If you don’t specify the scope of the Web App Manifest, Android usually takes the folder of the manifest as the scope, which is generally the expected behavior.

Safari will not define a default scope if not specified and if that’s the case, then EVERY link in your PWA will open inside your App’s iOS window. What’s the problem? It’s iOS, and there is no back button and no back gesture, so the user might end up locked in an external website you link. If you specify the scope, everything works as expected similar to Android, destinations out of the scope will open in Safari — with a back button (the small one in the status bar) to your PWA.

So what Android (Chrome) does is actually a violation of the standard, while what Safari apparently does seems to match the standard. Still, I think there are big problems with this which is why we discussed changing the standard last year to match Chrome.

I wrote up a detailed analysis last year, a result of a long discussion with @dominickng, which I have now made public:
https://docs.google.com/document/d/1Nq3dqT7ULrztGe2hhCwU12uSc3JONJNteh7sHSlNJlQ/edit

Note: We did not start off from a standpoint of trying to justify Chrome's behaviour; instead worked through all the options and ended up arriving at the same conclusion as @pkotwicz and his team. Below I will summarize the doc:

The scope needs to be well-defined for two distinct purposes:

1. "App context navigation" (kicking you out of the app context if you navigate out of scope), and
2. "Deep linking" (sucking you into the app context if you happen to click a link into the app scope).

The current specification is that if "scope" is not present, it defaults to a special value called "unbounded scope" which is fairly non-sensical. For app context navigation, it's implementable but silly: it means you will never get kicked out of the app context, even if you navigate off-origin. As explained by this article:

Safari will not define a default scope if not specified and if that’s the case, then EVERY link in your PWA will open inside your App’s iOS window. What’s the problem? It’s iOS, and there is no back button and no back gesture, so the user might end up locked in an external website you link.

A counter-point to that is that the spec recommends that if you have unbounded scope and navigate off-origin, that the user be shown the location bar and possibly dropped into browser UI. But this recommendation essentially makes "unbounded" scope equivalent to the origin of the start_url. So why not get rid of the concept of unbounded scope, and just specify a proper default?

For deep linking, an unbounded scope is not implementable (it would imply that all links in the universe should be redirected into the app context of an app with unbounded scope).

Thus, we considered four alternatives:

1. Containing directory of the start_url.
2. Containing directory of the manifest URL.
3. Origin of the start_url.
4. Scope of the longest-prefix-match service worker controlling start_url.

You can read the pros and cons of each in the doc, but in short, we chose containing directory of start_url as it seems to be the least surprising, and some sites (e.g., CNET) already rely on this behaviour in Chrome.

Thus, the required changes to the manifest spec are:

• In the scope processing algorithm, replace all of the “undefined” error cases with returning the containing directory of start_url. This removes the concept of unbounded scope.
• The navigation scope security considerations for unbounded scope no longer apply. However, as an orthogonal change, allow out-of-scope navigations to occur within the app window but recommend the CCT-like UI that is currently recommended for off-domain navigations in unbounded-scope apps.
• The deep links section needs to be much better specified, explicitly specifying that the UA may instantiate an application context when it navigates to a URL in-scope.

I plan to put up a PR to change this shortly.

[mgiuca]

I've made the changes, shown in this diff.

No PR yet, as I'm waiting to land two (hopefully uncontroversial) preliminary changes first (#644 and #645).

[mgiuca]

I got a response from Apple. They say they'll update their implementation to match if the PR lands.

@marcoscaceres I think we just want an equivalent confirmation from Mozilla engineering and then we can go ahead.

[marcoscaceres]

I’ll make it happen on the Moz side. Cc @snorp.

[mgiuca]

FYI, this change has landed and is now live at https://www.w3.org/TR/appmanifest. Implementations should update to the new behaviour.

The normative changes are:

• If the "scope" member is omitted or invalid, it should now default to the start URL, with its filename (final path segment), query, and fragment removed, as specified in steps for processing the scope member.
• There is no longer a concept of "unbounded" scope.

Web developers should ensure that either a) the directory that the start_url is in encompasses the entire app, or b) they explicitly specify scope. This change may restrict the scope of the app compared to what it was previously. However, note that this has been the behaviour of Chrome for Android all along, so likely no change is required.

[socceroos]

This change has broken all OAuth workflows for PWAs that have been added to a user's home screen. The flow is opened in Safari and then control is never returned to the PWA.

[mgiuca]

@socceroos Noted. This is complex because it's partly a spec issue and partly a Safari issue. (We have seen similar reports on Safari iOS and I've been in contact with Apple about it.)

The spec issue is:

• Previously "unbounded scope" would allow navigations off-origin to proceed within the navigation context, but show some security UI. Now it is specced the way normal scope is: a navigation off-origin is supposed to open a new navigation context.
• This creates an issue for authentication flows where you are expected to navigate off site, then redirect back.
• We don't want to revert back, because the old behaviour only applied in the weird "unbounded scope" case. We want to fix it for all scopes.
• This is covered in the long issue thread #646.

The Safari issue is:

• While it respects the spec by creating a new navigation scope, it also isolates the cookies between the two navigation scopes.
• This means if your app redirects to a different scope, a new navigation context is created in the Safari browser (correct per current spec), you can sign in there (correct), and redirect back to the originating site but still within the Safari browser (bad but technically correct per current spec), but next time you go back to the app, your cookies will be missing (nothing in the spec says to isolate cookies).

So they could fix the above cookie isolation, but I do think we need to fix this case in the spec too. See #646.

[g-ortuno]

I wonder if this is a problem for Microsoft as well. IIRC, they also opened out-of-scope navigations in a new tab and didn't share the cookie jar.

cc: @boyofgreen

[marcoscaceres]

I wonder if this stuff will eventually be solvable with credential management API (haven't looked... seriouslyhonestly just wondering)?

[praneetloke]

@mgiuca just wondering if you've heard back from Apple about the issue stated by @socceroos? On Android, OAuth flows initiated by PWAs are handled by (automatically) opening a Custom Tab _within_ the context of the PWA. Chrome closes the login window when it detects a redirect back to the domain of the PWA and resumes the PWA.

Without a fix for this in Safari, specifically for the OAuth flow use-case, PWA on iOS is technically unusable for any web-app that has a user login. On top of this, because of the separate navigation context as you mentioned in your comment, there is not a nice workaround either. This issue is present on devices running iOS 11.3 and up. I have tested 11.4.1 Beta 2 and that still has the same issue. Not sure if this is being addressed in iOS 12 either.

EDIT: Fixed a typo.

[mgiuca]

I haven't heard anything about this issue from Apple.

[praneetloke]

For the moment I have a separate manifest for iOS devices where I set the display property to browser (ref: manifest on MDN). That way, Safari can still use most of the features of a PWA (service-worker, manifest, icons etc.) except for the chromeless webapp experience. I think it's a good compromise until there is some sort of a change or fix for this issue from Apple. I am still testing this, so far, it has been good.

[socceroos]

We've used a hacky, terrible solution that basically authenticates the user in the spawned Safari context and then saves the resulting token temporarily to a backend DB that it is then retrieved when the homescreen PWA is re-opened. Not a pleasant user experience.

[cpBurn]

My fix is similar to @socceroos. I had to redirect the user to a different domain so we would have a different context there. My solution to communicate between Safari tab and the PWA is basically using Cache API:

var content = { token: "{{ user.token }}"}; // a token provided by backend
var blob = new Blob([JSON.stringify(content, null, 2)], {type : 'application/json'});

var resp = new Response(blob,{ "status" : 200 , "statusText" : "ok" });

caches.open('/').then(function(cache) {
    cache.put('/token/', resp); // put this token on cache
});

When coming back to PWA I got this by reading the cache:

const req = new Request('/token/');
caches.match(req).then(response => response.json()).then(({ token }) => console.log(token));

It's not a pleasant experience, we're encouraging the user to install the PWA after he authorises in Safari. That should reduce the usage of this flow.