Tools: De-duping dependencies across polymer-bundler passes

[govis]

I'm trying to bundle an application in 4 bundles: Shell, Bundle1, Bundle2 and Bundle3 with Shell containing dependencies shared between Shell.html and Bundle1.html and shared_bundle_1.html containing dependencies shared between Bundle2.html and Bundle3.html less what has already been bundled in Shell.html and Bundle1.html.

The following 2 ommands work independently, but some dependencies get duplicated - is it possible to de-dup them somehow?

polymer-bundler --root rootDir --manifest-out manifest.json --out-dir outDir --shell "shell.html" --in-html "shell.html" --in-html "bundle1.html"

polymer-bundler --root rootDir --manifest-out manifest.json --out-dir outDir --in-html "bundle2.html" --in-html "bundle3.html"

[justinfagnani]

The bundle is meant to be run once to produce all the bundles you need. Is there a reason your not configuring multiple fragments?

[govis]

Thanks @justinfagnani for your response. It's my goal to try keeping the shell slim. It contains splash screen and only bundles Polymer and some mixins. Bundle 1 is lazy-loaded upon shell initialization and contains most frequently used pages and core set of elements. Bundle 2 & 3 contain other less used pages and can be fat, so would rather not bundle their dependencies into the shell. I need to bundle as I have a lot of custom elements and can't leverage server push in my server environment.

As I understand SPA fragments are individual pages and if I were to bundle them(with their dependencies) individually - I still wouldn't be able to de-dup the dependencies whould I?

[justinfagnani]

The Bundler doesn't de-dupe dependencies because it never duplicates them in the first place, which is why it's important to only do one run of the Bundler, configured properly.

Fragments are any dynamically loaded dependencies, they could be pages are anything else loaded with importHref or lazy-imports: https://www.polymer-project.org/2.0/docs/tools/polymer-json

If you configure a shell for shell.html, and a fragment for each of the bundle.html files, it should work fine.

[govis]

I see, that might actually work for me, so I'll try to setup my bundles as fragments. From reading the docs it appears that it'll generate a single shared dependencies bundle for my bundles 1, 2 & 3, which will get imported along with Bundle 1 when shell loads.

With other bundles being fat and rarely loaded - do you think it would make sense for a future enhancement to have a fragment spawn a new shared dependencies bundle to make it possible for more then one shared bundle (I imagine previously created bundles would have to be linked as well as the order of import may not be guaranteed)?

On a separate note - is it possible to feed polymer.json to polymer-bundlerdirectly or does it have to be invoked by the polymer build? Thanks.

[govis]

I see that bundle groups idea was brought up in #511 and agree that it could get hairy with managing imports order, but something as simple as incremental creation of new shared bundles (as required) along with linking previously created bundles as per build sequence might be a compromise.

[web-padawan]

@govis here is the example of how you can configure bundle merge strategy with polymer-build:

const polymer = require('polymer-build');
const bundler = require('polymer-bundler');
const polymerJSON = require('./polymer.json');
const project = new polymer.PolymerProject(polymerJSON);

var buildStream = merge(project.sources(), project.dependencies())
    .pipe(project.bundler({
        strategy: bundler.generateShellMergeStrategy(polymerJSON.shell, 4)
   }))

This results in the following workflow:

1. If the component is only imported by 3 fragments, it will be placed to separate bundle
2. If the component is imported by 4 fragments or more, and / or shell, it will be placed to shell

Any other tricks, like manual excluding etc, are unreliable, can cause code duplications and runtime errors.

[govis]

Thanks @web-padawan for your suggestion. Given your experience with generating optimal bundles I'd like to give it a try - is it possible to run it directly with node? I saved your snippet as my_build_script.js but when I try executing it with node I get

Cannot find module 'polymer-build'

    at Function.Module._resolveFilename (module.js:470:15)

    at Function.Module._load (module.js:418:25)

    at Module.require (module.js:498:17)

    at require (internal/module.js:20:19)

    at Object. (my_build_script.js)

    at Module._compile (module.js:571:32)

    at Object.Module._extensions..js (module.js:580:10)

    at Module.load (module.js:488:32)

    at tryModuleLoad (module.js:447:12)

    at Function.Module._load (module.js:439:3)

My current build is done with running polymer-bundler and polymer build commands and I'm a bit new to the gulp setup.

[web-padawan]

@govis you can find an example gulpfile here. If you are not inclined to use gulp, you could just use Vinyl streams since polymer-build supports them as well.

[govis]

Finally have made some progress. I created polymer.json as follows:

{
  "entrypoint": "index.html",
  "shell": "shell.html",
  "fragments": [
      "bundle1.html",
      "bundle2.html",
      "bundle3.html"
  ],
  "lint": {
    "rules": [ "polymer-2-hybrid" ]
  },
  "builds": [
    { "preset": "es6-bundled" }
  ]
}

For some reason polymer build would not generate any shared bundles - it looks like shared deps get bundled into the shell by default, which is not ideal.

So I created custom gulpfile.js with generateSharedDepsMergeStrategy(3) and executed it with gulp build - now my shell stays slim and I get a few shared bundles created. Unfortunately documentation is fairly dry on how to get it setup from scratch, thanks @web-padawan for the link. Both generateSharedDepsMergeStrategy(3) and generateShellMergeStrategy(project.shell, 3) appear to generate identical output in my case.

[govis]

Actually before I close it - could you please explain why polymer build does not generate any shared bundles but generateSharedDepsMergeStrategy and generateShellMergeStrategydo and if it's possible to feed the bundle strategy/config via polymer.json or some other way?

Having to resort to a custom gulpfile.js also requires dealing with minification and other things already offered by polymer-build.

And even with shared dependencies bundles generated it appears that some (the fattest one in my case) get linked into the shell. My goal is to keep the shell as slim as possible and not include any dependencies from the fragments in it - is it possible?

[web-padawan]

@govis see my comment here for some clarification.

Regarding your question about how to keep the shell slim: this depends on how much fragments import your heavy dependencies. In our project we defer loading some app-level stuff, e. g. i18n library, withe the help of polymer-lib-loader element.

This enables us to reduce shell size and to use better caching, since libraries like this are not expected to change really often. As a drawback, we have to ensure i18n is available before doing some date, time or currency formatting.

The number passed to generateShellMergeStrategy as a maybeMinEntrypoints parameter is up to you. We configured it to 4 and now we have 49 shared bundles, while also having 65 fragments, and our shell is still big (174 Kb gzipped) due to moment library which is imported from 6 fragments. But at least we managed to split other libraries like d3, ace and masonry out to shared bundles.

I'd also have to say that, the more you increase this humber, the lesser shell becomes (and thus first paint faster), but this also means more requests later on, e. g. our pages can import from 6 to 10 shared bundles each. This makes situation closer to an unbundled approach.

[govis]

Thanks @web-padawan I am afraid those issues will keep coming until there's more info around available bundling strategies and other options such as minification. I agree with @vdegenne that it seems strange when those tools produce different output even though one uses the other. From poking at the code it looks like polymer-build uses generateShellMergeStrategy when shell is provided and polymer-bundler falls back to generateSharedDepsMergeStrategy by default.

I still don't get why shell should get any dependencies from lazy-loaded fragments bundled into it or linked via any shared deps bundles - it doesn't need them and it will only slow it down at startup. Polymer doc for PRPL pattern states that "The shell is responsible for dynamically importing the fragments as needed" and the first diagram shows no linkages between the shell and shared dependencies. It says that it's to load bundled and unbundled fragments, but what if there're lot of fragments and/or dependencies - will it not make the shell bloated unnecessary.

[vdegenne]

@govis not sure if that will be of any use for you but I spent few days trying to understand polymer-build behaviors and tools behind because I really needed to find a way to output only one minimized file giving a shell as the input, and I made a concatenation of tools inside a yarn package with an example case you can find on this git page, hope it will help : https://github.com/vdegenne/polymer-micro-build-toolbox

[govis]

Thanks @vdegenne for sharing this - I'll have a look. I've spent quite a bit of time as well trying to come up with an optimal packaging approach. We ended up converting to ES6 modules syntax. This allows for bundling with rollup and/or webpack. There's also polymer-webpack-loader plugin for Webpack that converts dom-module markup to ES6 module as well, so that it can be bundled together with the underlying javascript of the app itself - I think this is going to be direction moving forward. More info in Rob Dodson article here