Gatsby: Why is my app-[hash].js file so large?

[bengray]

Summary

I work for a company that's running a website using Gatsby. There are 60k product pages, and that's causing our app-[hash].js file to be 10MB. Is there any optimization we can make to reduce the size of this app-[hash].js file?

Relevant information

I work on a fairly large e-commerce site that was built using Gatsby. Up until now we have been dynamically loading all our product data at runtime so we only had 1 product page template and on load, we would fetch the necessary JSON to populate that template. So for the whole site we had a total of about 3k pages.

However, now we are trying to statically build all our product pages and that adds about 60k pages to the website.

After we did this our app-[hash].js file has increased from 140kb to 10mb and general page performance has tanked. Hovering links causes page to hang for a half a second (due to prefetching?)

We are creating the product pages by passing the product data as page context when calling create pages:

const productsRaw = await fetch(productsUrl)
  const products = await productsRaw.json()

  /* Build a page for each product */
  if (products) {
    products.forEach(product => {
      const productPage = {
        path: `/product${product.route}`,
        component: path.resolve(`./src/templates/product.jsx`),
        context: product,
      }
      createPage(productPage)
    })
  }

We believe the bad page performance is related to the app-[hash].js file jumping to about 10mb. Is there any optimization we can make to reduce the size of this file? Can we somehow "exclude" certain pages? Can we even "exclude" our 60k product pages from this file? Or is this just unavoidable when increasing the page count?

Environment (if relevant)

System:
    OS: macOS 10.14.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 10.16.0 - ~/.nvm/versions/node/v10.16.0/bin/node
    Yarn: 1.22.1 - /usr/local/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.16.0/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 80.0.3987.163
    Firefox: 75.0
    Safari: 13.1
  npmPackages:
    gatsby: 2.19.39 => 2.19.39
    gatsby-cli: ^2.7.2 => 2.7.2
    gatsby-image: ^2.3.2 => 2.3.2
    gatsby-plugin-alias-imports: ^1.0.5 => 1.0.5
    gatsby-plugin-create-client-paths: ^2.0.5 => 2.0.5
    gatsby-plugin-google-tagmanager: ^2.0.15 => 2.0.15
    gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12
    gatsby-plugin-remove-trailing-slashes: ^2.0.11 => 2.0.11
    gatsby-plugin-sass: ^2.0.11 => 2.0.11
    gatsby-plugin-sitemap: ^2.1.0 => 2.1.0
    gatsby-plugin-styled-components: ^3.1.19 => 3.1.19
    gatsby-source-contentful: ^2.0.67 => 2.0.67
    gatsby-source-filesystem: ^2.0.38 => 2.1.38
    gatsby-transformer-json: ^2.1.11 => 2.1.11
    gatsby-transformer-remark: ^2.3.12 => 2.3.12

File contents (if changed)

gatsby-config.js: N/A
package.json: N/A
gatsby-node.js:

const contentfulPage = require('./src/lib/generators/contentful-page.jsx')
const stores = require('./src/lib/generators/stores.jsx')
const externalJson = require('./src/lib/generators/external-json.jsx')
const financeFeed = require('./src/lib/generators/finance-feed.jsx')
const staticPDPPages = require('./src/lib/generators/static-pdp-pages.jsx')

exports.onPreInit = async ({ reporter }) => {
  externalJson.storeFile({ reporter }, `${process.env.GATSBY_S3_MISC_URL}/abc123.json`, `abc123`)
}

exports.createPages = async ({ graphql, actions }) => {
  await financeFeed.storeFile({ graphql })
  await contentfulPage.addPages({ graphql, actions })
  await staticPDPPages.addPages({ graphql, actions })
  await stores.addPages({ graphql, actions })
}

exports.onCreateWebpackConfig = ({ stage, getConfig, actions: { replaceWebpackConfig } }) => {
  switch (stage) {
    case 'build-javascript':
      const config = getConfig()

      const app = typeof config.entry.app === 'string' ? [config.entry.app] : config.entry.app

      config.entry.app = ['@babel/polyfill', ...app]
      replaceWebpackConfig(config)
  }
}

gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[KyleAMathews]

That doesn't sound right — have you looked at what data is in the app.js file? Can you try adding this plugin? https://www.gatsbyjs.org/packages/gatsby-plugin-webpack-bundle-analyser-v2/

Data added to the pageContext is added to the page-data.json file.

[bengray]

@KyleAMathews Thanks for that advice to use the webpack bundle analyser. I'm uploading an image of the report here (if you'd like the actual html report I can upload it). Looks like all our size is a result of match-paths.json.

[starrett67]

I wanted to update this thread (@bengray and I work together). So we found that all the new product pages we were building were getting added to match-paths.json. However, we don't think they are needed. I wrote a script to reduce the match-path.json file to only have paths with regex patterns. That script runs before webpack consumes the file. The site performance is great after I introduced this script and everything seems to work fine.

Is there a way we can exclude our new product pages from getting added to the file in the first place?

[KyleAMathews]

You're not setting matchPath ever when creating pages?

[starrett67]

We are not passing it as an argument to the createPage function so I'm not sure how its getting into the file.

[wardpeet]

That's odd, do you have any routes with matchPath set? Would it be possible to share your match-paths.json or a part of it? If it's difficult to share publicly would you be able to send me an email at [email protected]

[LekoArts]

Can you please also post your gatsby-config.js as I see that you use gatsby-plugin-create-client-paths

[starrett67]

Here is the config for client paths, we just use it for certain pages but it should not affect our product pages:

    {
      resolve: `gatsby-plugin-create-client-paths`,
      options: { prefixes: [`/search/*`, `/order/success/*`, `/cart/*`] },
    },

[starrett67]

I figured this out. So we actually do create another dynamic product page that contains a matchPath. This page is used just in case we dont properly build out all the products and a user tries to access a product thats not statically generated. In this case, it would hit a match path that would dynamically load product data. This page also uses the same template as the other product page. When I remove this page from getting created all the other products get removed from match-path.json but when it is included all the other products get added to the match-path.json. Here is the code:

/* Build a page for each product */
  if (products) {
    products.forEach(product => {
      const productPage = {
        path: `/product${product.route}`,
        component: path.resolve(`./src/templates/product.jsx`),
        context: { product, banners: PDPBannerData ? PDPBannerData.contentfulProductAdditionalContent : null },
      }
      createPage(productPage)
    })
  }

  const productPage = {
    path: '/product',
    component: path.resolve(`./src/templates/product.jsx`),
    context: { banners: PDPBannerData ? PDPBannerData.contentfulProductAdditionalContent : null },
    matchPath: `/furniture/product/*`,
  }
  createPage(productPage)

I'm assuming it does this for a reason, any thoughts on why this is necessary? We were talking about removing this dynamic page so its probably not a big deal for us to just deleted it and move on.

[coreyward]

I believe this is the same issue: https://github.com/gatsbyjs/gatsby/issues/21701

[wardpeet]

@starrett67 it's the same bug as #21701. I'll close this one and move the conversation there.