Manifest: Add offline_url option

[Snugug]

A proposal for a new, optional property for manifest files: offline_url. This should point to a URL like start_url. When present, if a PWA would serve the default browser offline page (like the Chrome offline dinosaur), it would instead serve the offline_url page. This should _not_ override the ability of Service Workers to serve pages when offline, it would _only_ be served if a URL isn't covered by a Service Worker and the browser would need inform users that they are offline.

[marcoscaceres]

That seems to replicate what service workers already provide. Can you help me understand why someone wouldn’t include a service worker to achieve what is being proposed?

[Snugug]

Sure, and yes I do understand that it's duplicative of what you can do with Service Workers.

Recently, I've found a number high-profile PWAs that present browser-default offline pages, which breaks the app-like experience that PWAs promise. The thinking here is that Service Workers are still hard to work with and configuring them in all the myriad ways one can _and_ ensuring that a custom offline page is always served appears to still be a challenge for developers. Having a stand-alone property for this that's as simple as passing a URL I believe would greatly increase the chance that PWAs provide the desired native-like offline experience, both by reducing the development cost of doing so and by making it highly visible.

In addition, I believe this could improve tooling for PWAs; currently, there isn't a good way to generally ensure that a PWA never shows the default offline page. Having this property would give tooling a way to suggest a way of supporting that.

[glennhartmann]

Would the idea be that offline_url points to a single self-contained file? Or would the browser be responsible for caching all of the ancillary style, script, and media files as well?

[Snugug]

I think I'm open to it going either way, although I think for simplicity sake it's likely better for the browser to only be responsible for caching the HTML result of offline_url, and then leave it as an exercise to the author to either make it entirely self-contained or use resources cached via their SW

[b1tr0t]

I like the simplicity of this. Dead simple for devs to understand and I think it would improve the % of sites that have a better than blank" experience, and closely mimic a fairly typical app offline experience in the form of a splash "you're offline" screen.

Implementation detail: when would the PWA re-sync the cached offline_url? Would it serve-stale but otherwise obey HTTP header caching rules for the object?

[reillyeon]

I don't think it makes sense to add yet another static content caching mechanism to the platform but I agree that this provides a safety net when the user hits a page not served by the Service Worker. offline_url itself should be served by the Service Worker and this is something implementations could check before offering installation.

[b1tr0t]

I agree with you in principle Reilly but worry the extra step of making sure this URL is correctly configured in SW will result in a lot of sites getting it wrong. :/

[Snugug]

@b1tr0t I was having a bit of a discussion around this, and I think either of the following scenarios for updating would be good:

• Serve stale, but otherwise obey HTTP caching rules (as you mentioned) (Stale-While-Revalidate in Workbox parlance)
• Always serve from cache, only refresh on Service Worker install event (Cache First in Workbox parlance)

I think the "Stale-While-Revalidate" method is likely least surprising for developers, even though it may lead to additional downloads if HTTP cache headers aren't set well, whereas the "Cache First" method wold ensure no additional downloads but may not feel intuitive for developers if they push updates to just that file.

[mgiuca]

I like the motivation for this (increase the number of sites that have a proper "you're offline" error), but I also share @reillyeon 's reticence about adding yet another offline caching mechanism (c.f.: appcache).

The additional concern is that this would essentially encourage devs to go just this far and no further.

We _do_ want to encourage developers to build proper offline support (and poor-network support) into their apps, but we view it as a spectrum, roughly as follows from worst to best:

1. No offline support (generic browser offline error message).
2. Show an offline error page when offline, otherwise serve content from the network.
3. Show the app's normal UI when offline, but rely on the network to do all tasks (i.e., the UI is basically empty when not connected to the internet, still letting you navigate around the app but lots of widgets say "Not available while offline").
4. Perform read caching so that stale content is available for viewing in the app, but the user can't really take any actions.
5. Perform write caching so that the user can do certain actions like compose posts, which are saved in local storage and synced to the server when a connection is found.

Right now, a lot of apps are at 1. We want to push developers as far along this spectrum as possible, because we think the further you are along, the better the experience for the user. (This isn't just about offline either; the further along in this spectrum, the better your app will work in poor network conditions as well, because the user will be able to interact with the pages of the app while the network is dropping in and out.)

While 2 is an improvement over 1, it isn't really _that_ much better. To go beyond 2, you need to implement a service worker and architect your app properly. This proposal basically gives developers an "easy" way to get to 2, but it doesn't at all help with going beyond 2. My concern is that if we give developers an easy way to get to 2, the leap from 2 to 3 is even larger because you haven't gotten even a basic service worker running to get to "good enough". So this may result in more apps settling at 2.

Ideally, we wouldn't be telling developers "start off by making your app show an offline error, and maybe improve it later." We would be telling them to architect their apps so they work as well offline and in poor network conditions as they possibly can.

[Snugug]

As it sounds like a lot of the disagreement is around having the manifest now being in charge of caching something. What if instead of assuming that offline_page will do some sort of caching, it's rather closer to how start_url works where it doesn't own caching but rather tells what _should_ be used. This I think would still solve most of the problems I see that had me propose this (increasing visibility of this UX aspect, providing a tooling hook) and reduce the overhead to developers by only requiring them to ensure that that particular URL is cached. Caching is still controlled by SW, but manifest lets browsers know what it _should_ use.

[marcoscaceres]

Consider, the code to do this in an SW would be something like:

self.addEventListener("fetch", ev => {
  ev.respondWith(aCachedResponse(ev));
});

async function aCachedResponse(ev) {
  const response = await caches.match(ev.request);
  if (response) {
    return response;
  }
  // go to network instead
  try {
    const netResponse = await fetch(ev.request);
    if (netResponse.ok) {
      return netResponse;
    }
  } catch (err) {
    console.error(err);
  }
  // just return the index if all goes bad.
  return await caches.match("/"); // OR 404.html
}

The above has the following advantages over offline_page, in that:

1. it already works in all browsers that already support service workers.
2. it gives the developer absolute control over the whole experience - including being easily debuggable in devtools.
3. it doesn't require any magic or violating/reinterpreting any HTTP cache semantics.
4. it doesn't introduce anything new to the spec - so less things to worry about.

I'd echo what both @reillyeon and @mgiuca have suggested - the motivation is right, but we already have a solution for this: service workers. Yeah, they are complicated and somewhat annoying, but they are what we have in the platform to do this today.

[mgiuca]

I had an offline chat about this today with some folks in Sydney. It's very complicated so I'll try to explain my thoughts here.

Firstly we've kind of got two parallel ideas going now:

1. A declarative mechanism for handling all the caching logic and serving it.
2. Just a declarative mechanism for _pointing at_ the offline page, but the caching is still handled by the service worker.

Re Idea 2:

@Snugug :

As it sounds like a lot of the disagreement is around having the manifest now being in charge of caching something. What if instead of assuming that offline_page will do some sort of caching, it's rather closer to how start_url works where it doesn't own caching but rather tells what should be used.

You're right that this new proposal separates concerns correctly, with the manifest in charge of the high-level layout of the app and the SW in charge of the network / caching. But if we did it this way, I don't really see what problems it solves. You'd still have to write a SW with a fetch handler that does caching, exposing your site to the performance and breakage concerns mentioned above. The only thing this offline_page option in the manifest would save you from doing is the last line of Marcos's service worker code ("just return the index if all goes bad"). I don't see the benefit of doing Idea 2 if the SW is still responsible for caching.

So turning back to Idea 1 (a declarative mechanism responsible for the caching logic):

I've been convinced that there _is_ an underlying need here that isn't really met by Service Workers: specifically that if you have a large business-critical website that's working just fine as it is, adding a fetch handler in front of all your requests is a pretty huge deal. Firstly because it may result in a performance regression due to the extra IPCs and JavaScript code running on every request, and secondly because it has the potential to break everything if there's a bug in the fetch handler (e.g., it could accidentally get stuck serving the main page from cache forever, making it impossible to push out an update). So it's a pretty scary thing to ask a large website to add, even if it is just about a dozen lines of code as @marcoscaceres wrote above.

If adding a manifest and an offline page is all a site needs to do to get their app to run as a PWA, they might do it, but if they need to put their site at risk, they might decide not to take the risk. So providing this "easy-path" offline page might be the thing that convinces many large web properties to become PWAs, and that could be a significant boon to the ecosystem. So this is something to consider.

Essentially we wouldn't be adding any capability that you can't already do with a SW, but we'd be making it much easier and safer to do it, and that alone could be worth it if it convinces people to adopt our tech.

Now, turning to what I said above (the "PWA spectrum" and pushing people further along). While I still think that would be good to not have sites simply stop at Step 2, I acknowledge that going past Step 2 requires a significant re-architecture of an existing site, something which big existing sites might never get around to (or at least, not until they do a full rewrite). Even though it's not the best experience to have an "app" that just shows a blank page while it's loading, and a generic error when it's offline, it's still better for our users to be able to choose to use such an app, rather than not have it at all (or, only be able to access it through a web browser). Overall, if the app's primary functionality requires network access, it doesn't really matter what the offline / poor network experience is.

So I think I'm convinced that if we can provide an easier/safer way to let websites get over the hurdle into Step 2, we should look into it.

Having said all that, I'm still not convinced that the manifest is the right place to do this. It definitely seems like it should be part of the service worker ecosystem (even if it isn't done with a fetch handler). The manifest is how we define the metadata for the app; the SW is how we define its interactions with the network.

As @glennhartmann pointed out, this would require some way to specify what all of the resources (like images) are needed by the offline page, and get them cached and served correctly too. That makes it non-trivial for a site to support this; you can't just throw a single "offline_page" line into your manifest, you also need to wire into your build system a way to generate the list of resources used by the offline page in order to cache them. So I don't think this is a "quick and easy" fix for lazy sites that just want to be a PWA without doing engineering work to integrate this properly.

Rather than doing this in the manifest, I think it would be best to introduce a declarative mechanism into the Service Worker itself. If sites are too scared to add a fetch handler, maybe you can still add a SW but set up the offline caching and serving by calling setup functions in the install event handler:

self.addEventListener('install', () => {
  self.addOfflineHandler("/offlinepage.html",
     {resources: ["/assets/heading.png", "/assets/style.css"]});
});

(Just a strawman proposal.) Basically a way of having service workers deal with this, since it really doesn't belong in the manifest.

The "addOfflineHandler" would tell the browser to automatically serve that page if any network request (or fetch handler) fails.

Now that we're talking about declarative routing in service workers, let's have a look at @jakearchibald 's declarative router proposal. I haven't had time to dig into this so I'm not sure if it solves the problem or not, but I'd be much more comfortable pushing this proposal into existing work being done by the Service Worker folks, rather than trying to come up with a run-around solution in the app manifest.

Those are my thoughts on this for now. Sorry this has been a very long essay; I have to run and don't have time to make it any shorter!

[jakearchibald]

I have a bad neck from nodding along to https://github.com/w3c/manifest/issues/774#issuecomment-515257608.

TL;DR: folks wanting to reanimate the corpse of appcache need stronger justification 😀

The "offline page" pattern still leaves you with the most frustrating experience, staring at a white screen while your device spends minutes figuring out it doesn't really have a connection at all. Of course, someone will suggest "let's add a timeout option!", but is that timeout based on first byte? What if the first byte arrives and then it hangs? "Let's add an option for that too!" What if the connection failed, not because of "offline", but because of some other fetch failure such as SSL, or an invalid redirect? "We'll add options for that as well!" But what if you want different offline pages for different parts of the app? "MOAR OPTIONS!" etc etc and welcome to appcache.

You can't put a debugger statement in a JSON file. Once you go beyond the very simplest of cases, figuring out the rule-order and quirks of your DSL becomes much harder than imperative code.

As @mgiuca points out, even if we ignore the option complexity, this solution is a dead-end. It leads you to a bare minimum, not-great offline experience, and it isn't a stepping stone to something better.

The declarative router proposal was about trying to meet somewhere in the middle. The routes are declared with JS, so you can write logic to express the conditions and order. But still, you'd need to make up for the lack of debugger;. You'd need to add strong devtools to show the routes in play, and which route was used for a particular request. It also allows you to specify different routes for different paths, and doesn't restrict you to 'offline' pages.

[kenchris]

Sounds sensible... I was also worrying about further requires resources and what would happen if some of these were covered by an existing service worker.

Existing tooling like Workbox CLI already generates a JSON like list of required resources, so it might make sure sense to come up with a solution that can reuse that instead of manually having to type in the required resources in an array as in the above example.

[marcoscaceres]

I’m going to go ahead and close this based on the rationale given by folks above. Thanks everyone for the in-depth discussion, and @Snugug for the initial proposal.

[mgiuca]

Looks like the Declarative routing (w3c/ServiceWorker#1373) would be able to support this use case without adding the overhead of a fetch handler.

So the example code I wrote above for the hypothetical addOfflineHandler:

self.addEventListener('install', () => {
  self.addOfflineHandler("/offlinepage.html",
     {resources: ["/assets/heading.png", "/assets/style.css"]});
});

Could be expressed with declarative routes (making use of the "Extensibility" extensions) as:

self.addEventListener('install', event => {
  router.add(
    new RouterOr(new RouterIfURL('/assets/heading.png'),
                 new RouterIfURL('/assets/style.css')),
    new RouterSourceCache());
  router.add(
    new RouterIfURLStarts('/'),
    [new RouterSourceNetwork(), new RouterSourceCache('/offlinepage.html')]);

  event.waitUntil(
    (async () => {
      const cache = await caches.open(CACHE_NAME);
      await cache.addAll([
        '/assets/heading.png',
        '/assets/style.css',
        '/offlinepage.html',
      ]);
    })()
  );
});

Obviously that's a lot more verbose, but it could be wrapped in a library pretty simply.

I don't like having to use the extremely verbose RouterOr just to do the same logic but with different exact URLs. Easily fixed with a RouterIfURLOneOf(['...', '...']) (or just allow RouterIfURL to take a list).

[jakearchibald]

@mgiuca take a look at the thread. We kinda concluded that an object-based approach was less verbose than a class-based approach.

[mgiuca]

@jakearchibald I skimmed it all but wasn't sure if the object-based stuff at the end was "final" or just some speculative alternative, so I wrote my example using the more solid syntax from above.

The syntax doesn't really matter though; do you agree that we could use declarative routing for the offline handler in roughly the way that I expressed it above?

[jakearchibald]

I think the route would be:

router.add([], [
  'cache', // look for a match in the cache
  'network', // else go to the network
  ['cache', { request: '/offlinepage.html' }] // else get this specific item from the cache
]);