Gatsby: Split page metadata so can lazy load it and reduce the initial JS

[chmac]

Summary: Allow the data.json file to be loaded in shards for sites with large numbers of pages.

History

Once #4715 lands:

• data.json contains a map of all paths to data file names
  
  
  
  • File names include a content hash
  
  
• data.json is loaded after the page has finished rendering

Sharding was also discussed in #4626.

Changes

Rather than loading data.json as a single file, it could be split into multiple files based on a simple sharding mechanism, for example:

const calculateShardNameForId(shardLength: int, id: string) => md5(id).substr(0, shardLength)

@KyleAMathews mentioned:

Ideally you'd shard by something like path names so a shard for /blog/*, as those are likely to be needed together.

@pieh shared:

Just to give more info - when I run my tests against https://github.com/freeCodeCamp/guides (~2800 pages) - gzipped "webpackified" data.json is 141KB - this surely won't scale very nice for 100 000 pages sites, but up to 10 000 I think this shouldn't be that much of a issue

Rationale

• On very large sites (>1k pages, or >10k pages) the data.json file can become large.
• It must be fully loaded before prefetching can begin.
• Splitting it into chunks could allow faster prefetching, and require less data over the wire, on very large sites.

This is a low priority enhancement.

[KyleAMathews]

I have an open PR for this https://github.com/gatsbyjs/gatsby/pull/6651

[KyleAMathews]

Definitely needs done for Gatsby to scale to larger sites.

[gatsbot[bot]]

Old issues will be closed after 30 days of inactivity. This issue has been quiet for 20 days and is being marked as stale. Reply here or add the label "not stale" to keep this issue open!

[KyleAMathews]

gatsbyjs.org's page metadata chunk is up to 451kbs yikes 😬

[m-allanson]

Related to #13004

[m-allanson]

Fixed in #13004, and available in [email protected] 🎉 Huge thanks to @Moocar and everyone involved in reviewing and testing this feature 💜