Workbox: Task runner/bundler plugins for sw-build

[jeffposnick]

Library Affected:
sw-build, sw-lib

There are currently two distinct processes that need to be performed to produce a complete service worker that handles precaching:

1. Manifest generation (via, e.g., sw-build.generateFileManifest())
2. Bundling the manifest (and potentially the contents of sw-lib), inline to produce a final service-worker.js

One approach is to use sw-cli to handle both steps, and that should meet the need of some developers.

Other developers will want to automate both steps as part of a build process that uses an existing task runner/bundler. The immediate plans are to document how this could be done explicitly via "recipes" (https://github.com/GoogleChrome/sw-helpers/pull/258). Longer term, both these steps can be performed via a single plugin specific to a given task runner/bundling tool.

There are folks outside of this repo's contributors who might have thoughts about implementations, and I'm CC:ing a few of them:

• Webpack (CC: @goldhand @jescalan)
• Rollup (CC: ???)
• Browserify (CC: ???)

[gauntface]

Would we want a gulp plugin?

[jescalan]

I think as long as the API is clearly documented and there are instructions on how to do it, it should be easy to implement into any type of plugin, right?

[jeffposnick]

Just sharing some random thoughts while they're fresh in my mind:

We've put together a "recipe" for a very manual build process that uses Rollup as the bundler: https://github.com/GoogleChrome/sw-helpers/blob/master/docs/recipes/build/rollup.md

Those steps could be rolled-up into a... Rollup... plugin that takes in a single configuration object and handles both manifest generation and the bundling of the final sw.js.

Thinking about what that might look like for Webpack, but with the caveat that I don't really understand how Webpack plugins or loaders work, I came up with:

// Contents of src/sw.js:

import manifest from '/fake/path/to/manifest.js';
import swLib from 'sw-lib';

// This will precache the files in your manifest, and keep them up to date.
swLib.cacheRevisionedAssets(manifest);

// Any additional runtime service worker logic can go here.

// Contents of webpack.config.js:

module.exports = {
  entry: {
    sw: './src/sw.js',
    // Other entries for the rest of your web app.
  },
  rules: [{
    test: /manifest\.js$/,
    use: [{
      loader: 'sw-build-manifest-loader',
      options: {
      // Assume that the globPatterns will default to matching the output
      // of the Webpack build process, like SWPrecacheWebpackPlugin does?
      // But some other options would be configurable here.
    }]
  }],
  // Other config as appropriate...
};

The idea being that sw-build-manifest-loader would take care of using sw-build.generateManifestEntries(options) to create the manifest entries, and then inject that into the final build/sw.js.

My assumption is that Webpack would let you put in a "dummy" ES 2015 module path in your source's imports, which here I'm calling '/fake/path/to/manifest.js', which in turn would get translated by the loader into valid JavaScript, which then gets included in the final output. But maybe that's not how loaders work, and you need to use a valid path to an actual file in the file system in order to kick off the loader pipeline?

I also am not clear as to whether this automation makes sense within a loader, or whether it should be a Webpack plugin.

There's also the issue that some people might want a mode in which the entirety of the build/sw.js is generated for them based on a basic Webpack plugin configuration, and they don't want to create a src/sw.js at all. (That would be equivalent to what they get now via SWPrecacheWebpackPlugin.)

[prateekbh]

I guess a webpack plugin would be the apt thing which would act on generated chunks and modules after compilation is done, AFAIK loaders work on a specified files which are entry poibts and not generated files. Keen to hear from @jescalan

[jescalan]

Yeah so I'd probably do this as a plugin rather than a loader, like @prateekbh suggested. The sky's the limit with plugins, you can get a list of all assets and all their dependencies, add or remove any assets you need, inject variables and alias requires in the user's javascript file, etc.

If you're trying to tinker with a webpack plugin, the place to start would probably be the emit hook, where you get direct access to a list of all the assets that it has processed (via compliation.assets), and have the opportunity to build and inject your own additional assets if you want. It seems like exactly that's what you need to be doing here 😁

If you have any other thoughts for the ideal interface or developer experience with this, even if they seem impossible, feel free to post em here, there probably is a way to do it with webpack and I can try to point you in the right direction or build a little example.

[prateekbh]

Just curious, if anyone has started to get their hands on the webpack implementation? If not I'd be happy to pick this up

[goldhand]

Just sharing some ideas for the api:

One of the principles of SWPrecacheWebpackPlugin was to allow users access to the entire api from SWPrecache. We generated default values for SWPrecache using compilation.assets but always allowed users to override them and connect to the SWPrecache

I haven't use sw-build yet but I was looking at the instance methods in the documentation this would be tricky because it looks like there are 3 methods that take like-named arguments.

Would the value for generateFileManifest.globPatterns would ever be different than generateSW.globPatterns and so on for the other arugment?

Something simple like this would allow a variety of use cases and allow gradual development of additional webpack support:

...
plugins: [
  new SWBuildWebpackPlugin({
    generateFileManifest: {
      dest: './build/manifest.js'
      rootDirectory: './build/',
      globPatterns: ['**\/*.{html,js,css}'],
      globIgnores: ['admin.html'],
      format: 'iife', // alternatively, use 'es'
    },
    generateSW: {
      rootDirectory:  './build/manifest.js',
      ...
    },
  }),
]

[patrickhulce]

IMO, access to many of the sw-build options loses meaning in the context of a webpack plugin as the file paths are not as interesting and can, for the most part, be inferred from the output assets alone. It might be worth making the plugin more streamlined by default and just support these as a fallback.

Also happy to work on it :) I'd love to have consume this on the webpack side!

[prateekbh]

+1 for @patrickhulce, I guess defaults should be the picking the generated bundles and chunks.
Although overriding these with a user specified config object would be make the plugin more complete.

Although all of this sounds fine for sw-build, but I am curious if we can do something for plugins as well?

[goldhand]

@patrickhulce I kindly disagree. I can see some use for a few of the options at least. globPatterns and globIgnores for seem very useful if you don't want all of your webpack assets precached. Even though I don't see the use of every option, I wouldn't want to assume that a developer will never want to use them.

Sorry for being so verbose but consider the following:

I can see a lot of cases where a developer hear's about sw-build and wants to use it so she invests time studying sw-build docs / examples. She decides to integrate it into her project that happens to use webpack. Unsure where it fits in with her webpack project, she finds the sw-build-webpack-plugin. Rather than reading about another api, she just uses the plugin as a proxy to work with sw-build and passes suggested configurations from the sw-build docs in the sw-build-webpack-plugin and has no trouble, it works exactly as she expected it would.

[patrickhulce]

globPatterns and globIgnores for seem very useful if you don't want all of your webpack assets precached

I totally agree, and I didn't mean to drop support for all of these. I meant more that the API given by sw-build is entirely focused around files on the filesystem. How that translates to webpack is unclear to me (e.g. does this glob deal with my output assets or my source files which it tries to match to my output assets? must I respecify my output path in this plugin when I've already put it into my webpack config? must I respecify my public path in rootDirectory when I've already put it into my webpack config? etc). By providing an identical API I would expect the semantics of its arguments to not change. Given that the semantics of the original don't really hold up/are fairly ambiguous, I think there's a case for attempting to automatically fill these in with the information given, make explicit what maps to sw-build concepts, and have the user not need to use it in the same manner except in override cases.

[goldhand]

@patrickhulce I think we are on the same page, I'm sorry I wasn't clearer but I think automatically filling filling values like rootDirectory where possible while allowing developers to override these default values is a great idea.

[addyosmani]

Hey folks. Just catching up on the discussion. We're planning out the stable milestone atm and it would be great to nail down owners for the Webpack plugin work.

Wrt the current Webpack Precache plugin we are committing to ship a 1:1 API for sw-cli as sw-precache offers today and to some extent it might make the integration easier. I agreed with Patrick and @goldhand on populating values that are overridable where we can.

Beyond owners for this work, I am also interested in what more we can do to support Webpack users in this space other than calling out the plugin in our docs/site/etc. We flagged wanting a getting started recipe for Webpack in our weekly sync today and are happy for that too to be prominently listed in our docs.

[prateekbh]

@addyosmani @patrickhulce @goldhand I mentioned this somewhere above, I am kinda curious if we can extend this to support plugins as well e.g. bgQueue, images plugins...
I wonder if in conjunction with above approach if we can supply something like a sw-template file, inside which the populated assets value can be filled in too.

[goldhand]

@addyosmani regarding supporting webpack users:
I think some of the dissonance with webpack and the sw-* packages comes from the way they both want to write to the filesystem during the build process. If there was an easier way to access the populatedTemplate or the manifest as a string, we could add it to webpack's compilation assets.

On another note, it would be nice if these packages just had webpack support built in and you didn't need a separate webpack plugin. Not sure if it would work for all the packages in here but here's what I'm entertaining:
Make a magical decorator utility that returns a webpack plugin class for the decorated package.

Generally, sw-* packages need to be aware of what webpack is emiting.

webpack  -[1]-[2]--------[5]--->
sw-*     --------[3]-[4]------->

1. webpack begins its build process
2. webpack emits its compilation object with all the assets
3. sw-* gets the compilation object and uses it to populate a template string.
4. sw-* passes the generated string back to the webpack's compilation assets.
5. webpack writes files to system.

The decorator will would return a webpack plugin class with methods for connecting to webpack's compilation. Something like:

const SWWebpack = swApp => class SWWebpackPlugin {

  constructor(options) {
    this.options = options;
  }

  apply(compiler) {
    const {dest} = this.options;
    compiler.plugin('emit', (compilation, callback) => {
      const {chunks} = compilation.getStats();
      // get the sw generated string using the webpack chuncks
      return swApp(chunks)
        .then(generatedSW => {
          // add the generated string as a new webpack compliation asset
          compilation.assets[dest] = {
            source: generatedSW,
            size: Buffer.byteLength(generatedSW),
          };
          return callback();
        });
    });
  }
};

and in sw-build/webpack.js:

export default SWWebpack(generateSW);

And then one would use it in their webpack.config.js like:

import SWBuild from 'sw-build/webpack';

module.exports = {
   ...
  plugins: [
    new SWBuild({
      dest: 'my-sw.js',
    }),
  ],
}

[prateekbh]

@goldhand in step 3,
Are we also optionally taking a source template string so that all this process can sit with any custom logic already present?
This would really help devs to take advantage of this plugin, while combining it with any custom logic.

[prateekbh]

@goldhand I tried playing around with a webpack plugin, sw-build currently works on files on disk.
I think it'd be better to run this webpack plugin on after-emit and set default to webpack values
e.g.

const default = {
  rootDirectory = compilation.mainTemplate.getPublicPath({});,
  globPatterns: ['**\/*.{html,js,css}'],
  globIgnores: [],
  serviceWorkerFileName: 'sw.js',
  dest: rootDirectory + serviceWorkerFileName
}

This can be overridden with new config in webpack config like...

plugins: [
  new SwBuildWebpackPlugin({
    globPatterns: ['**\/*.js']
  });
]

Also @jeffposnick In case in above config we can add a key like sourceFile, which if present this plugin in will call getFileManifestEntries instead of generateSW and replace the output somewhere in the file and generate a custom serviceworker.

[goldhand]

You're right, @prateekbh. It needs to be on after-emit.
Just noting it won't be written to the disk as a webpack asset then, as that process is already completed by the time after-emit is fired.
Ran into this issue while trying to get sw-precache to work with webpack-dev-server.
It would be nice if there was a way to generate a service worker using files that don't exist yet on disk.

[prateekbh]

@goldhand @patrickhulce i have added a webpack plugin in the repo, please let me know if you have any views about it

[goldhand]

Looks good to me. I like how simple it is, it will be easy to add behaviors later as its uses are discovered.

[prateekbh]

@addyosmani i guess we should close this issue and open a separate one for other plugins.