Tools: handling caching issues

[jay8t6]

Is there a way to add revisions/versioning on polymer.json to handle caching issues?

Right now, when we make any changes to one of the subpages in the project and release it to prod, we had to refresh everytime to get the latest changes. how do we handle this issue? I was thinking about maybe adding ?v=randomsting next to every html dependency, but I am not sure if its possible to do in the json. If I add hashing in the index.html, its loading the file twice and throwing polymer errors...

    <script type="text/javascript">
      var link = document.createElement('link');
      var head = document.querySelector('head');
      var random = Math.random().toString(36).substring(7);
      link.setAttribute('rel', 'import');
      link.setAttribute('href', '/src/main-app.html?v=' + random);
      head.appendChild(link);
    </script>

and this in the lazy load:
      const random = Math.random().toString(36).substring(7);
         // Load page import on demand. Show 404 page if fails
       const resolvedPageUrl = this.resolveUrl(`main-${pageName}-page.html?v=${random}`);

Any thoughts?

[web-padawan]

In our team we ended up with using gulp-rev-all to generate fingerprints and rename files. This just works (after vulcanizing, minifying etc).

[jay8t6]

@web-padawan thats great to know, ill give it a try

[czellweg]

@web-padawan Do you have a sample script how you use gulp-rev-all?

[web-padawan]

This looks like this:

// Get the checksum of shared files, rename them and generate rev-manifest.json
function fingerprintGenerate () {

    // Add fingerprint prefix
    const fingerprintPrefix = function (file, hash) {
        var ext = path.extname(file.path);
        var name = path.basename(file.path, ext);
        return name + '-f9t-' + hash.substr(0, 8) + ext;
    };

    // Exclude files we don't want to rename
    // Don't rewrite references right now, to prevent wrong replacing of element id
    return gulp.src(GULP_CACHE + '/**/*')
        .pipe($.revAll.revision({
            dontRenameFile: ['index.html', 'favicon.ico'],
            dontUpdateReference: [/(.)+/g],
            includeFilesInManifest: ['.html', '.js', '.png', '.jpg', '.svg', '.woff'],
            transformFilename: fingerprintPrefix
        }))
        .pipe($.revDeleteOriginal())
        .pipe(gulp.dest(GULP_CACHE))
        .pipe($.revAll.manifestFile())
        .pipe(gulp.dest(GULP_CACHE));
}

// Rewrite references to file names which have been renamed by gulp-rev-all
function fingerprintReplace () {

    // We only need file name without directories
    const stripDirectories = function (filename) {
        return filename.split('/').slice(0).pop();
    };

    return gulp.src([GULP_CACHE + '/**/*', '!' + GULP_CACHE + '/rev-manifest.json'])
        .pipe($.revReplace({
            manifest: gulp.src('./' + GULP_CACHE + '/rev-manifest.json'),
            modifyUnreved: stripDirectories,
            modifyReved: stripDirectories
        }))
        .pipe(gulp.dest(DIST))
        .pipe($.size({
            title: 'dist size'
        }));
}

As you can see there are two tasks, and I'm using gulp-rev-replace to replace occurences. This is due to some weird bugs (e. g. gulp-rev-all replaces <dom-module id="my-element">)

Pay attention to that second task relies on unique file names (via stripping all directories), to allow using those names without full paths in the elements code.

[mgibas]

@web-padawan how does it work since index.html is also precached by service worker ?

[FredKSchott]

Since the service worker is regenerated for each build, the cache should be invalidated with each build.

[mgibas]

@FredKSchott should that happen automatically after I deploy new, regenerated, version of service worker ?

[FredKSchott]

Every time you generate a service worker via build, it is created with a new unique key. When you deploy that service worker, the browser sees that it has been updated and updates the service worker. The new key corresponds to a new cache, so the cache is effectively busted.

This isn't explained super clearly in the sw-precache README, but that has been my understanding of the behavior: https://github.com/GoogleChrome/sw-precache#dontcachebusturlsmatching-regex

[mgibas]

You are right - just made couple of tests locally and my service worker replaces all modified files as intended. Thing with deployed app is that service-worker.js gets cached by my CDN....stupid mistake. Turning cache off (I'm actually invalidating this file after every deploy) fix the problem and there is no need for any other tricks (like file versioning).

[cbfranca]

@web-padawan Thanks. One doubt: Is this approach very slow for you too?

[web-padawan]

@cbfranca the approach suggested by me is good as long as you handle file name changes during deployments.

Imagine the situation: one page used to be foo-page-fh7dgh.html and has changed to foo-page-34bvjd.html (while app shell stayed untouched) - your app should handle this properly to avoid 404 on importHref calls.

We have some kind of workaround relying on special HTTP request header called x-app-version which is injected based on Git commit hash, but that is generally DevOps stuff.

[davidmaxwaterman]

This is fine for service worker, but what about appcache?
Also, I wonder what happens with CDNs where files are copied one-at-a-time, and if the app is loaded during that time, it is in an inconsistent state...wouldn't it be better to version the app as a whole rather than individual files?

[web-padawan]

In our team, we detect case when fragment was changed during deployment and force reloading page (instead of showing toast) exactly to avoid inconsistent state.

[davidmaxwaterman]

Is there an example of this 'showing' toast (or even force reloading page, for that matter)?

[web-padawan]

Both webcomponents.org and polymer focs site show toast "reload to update", but they use service workers. To force reload, you could just do window.location.reload()

[davidmaxwaterman]

OK, thanks. I was more curious about the mechanism behind the toast/reload.

I see some clues here:

https://stackoverflow.com/questions/41502870/service-worker-reload-page-on-cache-update

where it is just about 'updatefound'->'statechanged' events, and then posting a message to the app, just like any other web worker.

Also, here:

https://github.com/GoogleChrome/sw-precache/blob/master/demo/app/js/service-worker-registration.js

I don't see any mention of this mechanism in the PSK service worker. That would have been nice.

[web-padawan]

@davidmaxwaterman the mechanism behind the reload used by us is as follows:

1. user browses the app served with the x-app-version header containing e. g. 0.0.1
2. user clicks on the link after we have deployed new version
3. router component receives 404 when loading view, due to fragment hash change
4. router component checks x-app-version header from 404 request and if it has changed as well (e. g. to 0.0.2) - then calls window.location.reload()

[beyondcreed]

main issue with service workers is that safari/edge still lack support, so you are hosed on some platforms/devices if you go with that, until they implement support.

[peterlauri]

@web-padawan we are trying to get this working. Started out by just running gulp with the file referenced here at https://github.com/PolymerElements/generator-polymer-init-custom-build/blob/master/generators/app/gulpfile.js. After that we stripped it down to be as simple as possible. As we never used gulp in our team, where in the pipe should one apply your two functions? Before or after bundling?

[web-padawan]

@peterlauri the idea of the approach I mentioned is to apply gulp-rev after bundling. Consider also taking a look at gulp-polymer-build-utils which is a set of gulp tasks containing the similar one.

[bendavis78]

For those stumbling on this thread, I've created gulp-polymer-build to help with this very issue. It borrows from polymer-cli build code, and supports your build configs in polymer.json. It allows you to modify your source stream before building, and then modify the forked streams for all builds you've configured in polymer.json. This makes it easy to use gulp-rev and gulp-rev-replace to do asset versioning.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.