Gatsby: Allow to create a script tag with src attribute for chunk manifest file

[szimek]

Currently the chunk manifest is always inlined into generated html files. It's ok if there are not that many pages/chunks, but in my prototype with ~500 pages the uncompressed chunk-manifest.json file has already size of ~30kB (12kB gzipped) and I can assume that with ~5000 pages that we got in total, it will be ~10x larger.

That's why it might be a good idea to add an option to generate a chunk-manifest.js file and a script tag with src attribute pointing to it.

So far I came up with something like this:

This generates chunk-manifest.js file based on contents of chunk-manifest.json file:

// /src/utils/webpack.config.js
new ChunkManifestPlugin({
  filename: `chunk-manifest.json`,
  manifestVariable: `webpackManifest`
}), 
{
  apply: function (compiler) {
    compiler.plugin(`emit`, function (compilation, callback) {
      compilation.assets[`chunk-manifest.js`] = new ConcatSource(
        `window.webpackManifest = `,
        compilation.assets[`chunk-manifest.json`],
        `;`
      );
      callback();
    });
  }
},

This decides if it should create script tag with inlined contents of chunk-manifest.json file or with src attribute pointing to chunk-manifest.js file:

// /cache-dir/static-entry.js
if (some condition) {
  const chunkManifest = require(`!file?name=[name].[hash].[ext]!../public/chunk-manifest.js`)

  headComponents.unshift( <
    script id = "webpack-manifest"
    key = "webpack-manifest"
    src = {chunkManifest}
    />
  )
} else {
  const chunkManifest = require(`!raw!../public/chunk-manifest.json`)

  headComponents.unshift( <
    script id = "webpack-manifest"
    key = "webpack-manifest"
    dangerouslySetInnerHTML = {
      {
        __html: `
            //<![CDATA[
            window.webpackManifest = ${chunkManifest}
            //]]>
            `,
      }
    }
    />
  )
}

It's just an initial version, which probably has some issues, e.g. require(!file...) will probably emit chunk-manifest.[hash].js as many times as there are pages (though maybe node caching mechanism kicks in here for require calls and it will do it just once), pathPrefix is not taken into account (though it should be an easy fix) etc.

Please let me know if it's something you'd be interested in and what's the best way to specify an option to decide if chunk manifest should be inlined or not and I'll prepare a PR.

[KyleAMathews]

Since the webpack manifest is loaded only once per session and is on the critical path for getting to interactive (better to inline things on critical path as we need them ASAP), we don't want to defer loading the script.

We do want however to move the inlining after the body as the manifest isn't on the critical path for the initial paint so putting it in the head is slowing down the initial paint some. Lemme know if you want to work on that!

https://github.com/gatsbyjs/gatsby/issues/2541

[szimek]

@KyleAMathews But wouldn't that mean that:

1. The same user who opens different pages (in different sessions) from our mobile app or marketing campaigns will have to load 300KB of _the same data_ over and over again? If it was extracted, it could be cached and loaded instantly on next sessions, until we deploy a new version. It will suck on mobile devices over 3G.
2. At 5K pages, every page size of 300KB+, the total size of generated pages would be 1.5GB+?

I'm not asking for it to be the default behavior, just an option that might be useful in cases where there are so many pages. Could it be reconsidered once the inlined manifest is moved to the end or or after the body element?

[KyleAMathews]

1. The total size of this manifest will shrink soon as we won't be managing the results of GraphQL queries through webpack like we are now
2. we'll also soon be splitting and lazy loading manifests so we'll only inline what's necessary to bootstrap webpack on the initial page

[KyleAMathews]

Hopefully you won't be building 5k page sites before then :-P

[barraq]

Thanks @KyleAMathews for you support, that is awesome news!

[KyleAMathews]

@barraq np! Yeah the goal has always been to make Gatsby work for any sized site e.g. 10s of millions of pages even — actually, really want to rebuild Wikipedia on Gatsby someday haha :-)

But yeah, had to ship something first and then go from there but there's straightforward ways to remove all the current scaling problems.

[KyleAMathews]

Will be putting together a new roadmap for this stuff in coming weeks.