Gatsby: Building JS and CSS bundles for 5 pages takes 88.374s

[mikheevm]

Description

After updating gatsby-*dependencies to the latest versions the cold build started to take really long compared to what it was before (like 4x or even more). After the cold one passes, succeeding ones will take 20-25 secs.
Approximately it became so after this commit: https://github.com/gnosis/dappcon-website/commit/56be14cc789c16355d81b2cc4be0d181adabc789

BUILD LOG

$ npm run clean && gatsby build
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1559199574589-0.9665285201595042/node but npm is using /usr/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.
> rimraf .cache public
success open and validate gatsby-configs - 0.013 s
success load plugins - 0.483 s
success onPreInit - 0.012 s
success delete html and css files from previous builds - 0.006 s
success initialize cache - 0.009 s
success copy gatsby files - 0.029 s
success onPreBootstrap - 0.007 s
success source and transform nodes - 0.163 s
success building schema - 0.244 s
success createPages - 0.074 s
success createPagesStatefully - 0.036 s
success onPreExtractQueries - 0.005 s
success update schema - 0.037 s
success extract queries from components - 0.179 s
success run static queries - 0.135 s — 2/2 15.57 queries/second
success run page queries - 0.552 s — 7/7 12.86 queries/second
success write out page data - 0.004 s
success write out redirect data - 0.003 s
success onPostBootstrap - 0.008 s
info bootstrap finished - 8.466 s
success Building production JavaScript and CSS bundles - 88.374 s
success Building static HTML for pages - 2.633 s — 7/7 11.08 pages/second
info Done building in 99.507033731 sec
Done in 100.05s.

Steps to reproduce

• Clone this repo: https://github.com/gnosis/dappcon-website
• Install the dependencies with yarn
• Run yarn build

Expected result

The project has only 5 static pages, so I expect it to build production bundle in less than 100 seconds.

Actual result

The build takes 100 seconds approximately. I could live with it, but on the CI environment it makes the build fail with error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Additional info

Sometimes I also get this error when trying to build:

error GraphQL Error Field "image" must not have a selection since type "String" has no subfields.

  file: /home/mmv/work/dappcon-website/src/pages/speakers.js

   1 |
   2 |   query {
   3 |     speakers: allMarkdownRemark(filter: { frontmatter: { templateKey: { eq: "speaker" } } }) {
   4 |       edges {
   5 |         node {
   6 |           frontmatter {
   7 |             name
   8 |             company
   9 |             position
> 10 |             image {
     |                   ^
  11 |               childImageSharp {
  12 |                 # Specify the image processing specifications right in the query.
  13 |                 # Makes it trivial to update as your page's design changes.
  14 |                 fluid(maxWidth: 134) {
  15 |                   ...GatsbyImageSharpFluid_withWebp
  16 |                 }
  17 |               }
  18 |             }
  19 |           }
  20 |         }

I tried to debug it but no success. It is solved by just re-running the build.

Environment

  System:
    OS: Linux 4.18 Ubuntu 18.04.2 LTS (Bionic Beaver)
    CPU: (12) x64 AMD Ryzen 5 1600 Six-Core Processor
    Shell: 5.4.2 - /usr/bin/zsh
  Binaries:
    Node: 11.15.0 - /usr/bin/node
    Yarn: 1.16.0 - ~/.yvm/shim/yarn
    npm: 6.9.0 - ~/.npm-packages/bin/npm
  Languages:
    Python: 2.7.15 - /usr/bin/python
  Browsers:
    Chrome: 74.0.3729.169
    Firefox: 67.0
  npmPackages:
    gatsby: 2.7.5 => 2.7.5
    gatsby-background-image: 0.7.0 => 0.7.0
    gatsby-image: 2.1.2 => 2.1.2
    gatsby-plugin-layout: 1.0.15 => 1.0.15
    gatsby-plugin-netlify: 2.0.17 => 2.0.17
    gatsby-plugin-netlify-cms: 4.0.1 => 4.0.1
    gatsby-plugin-react-helmet: 3.0.12 => 3.0.12
    gatsby-plugin-resolve-src: ^2.0.0 => 2.0.0
    gatsby-plugin-sass: ^2.0.11 => 2.0.11
    gatsby-plugin-sharp: 2.1.2 => 2.1.2
    gatsby-plugin-styled-components: ^3.0.7 => 3.0.7
    gatsby-remark-copy-linked-files: 2.0.13 => 2.0.13
    gatsby-remark-images: 3.0.14 => 3.0.14
    gatsby-remark-relative-images: 0.2.2 => 0.2.2
    gatsby-source-filesystem: 2.0.37 => 2.0.37
    gatsby-transformer-remark: 2.3.12 => 2.3.12
    gatsby-transformer-sharp: 2.1.20 => 2.1.20
  npmGlobalPackages:
    gatsby: 2.7.5

[m-allanson]

Thanks for the repro @mikheevm - it seems this extra build time is coming from gatsby-plugin-netlify-cms v4. You can test this with the default Gatsby starter using the following steps:

gatsby new netlify-cms-test
cd netlify-cms-test
gatsby build

... more output
success Building production JavaScript and CSS bundles - 6.790 s
success Building static HTML for pages - 0.680 s — 4/4 18.12 pages/second
info Done building in 12.437 sec

yarn add netlify-cms-app gatsby-plugin-netlify-cms

(add `gatsby-plugin-netlify-cms` to `gatsby-config.js`)

gatsby clean && gatsby build

... more output
success Building production JavaScript and CSS bundles - 41.253 s
success Building static HTML for pages - 0.655 s — 4/4 19.51 pages/second
info Done building in 46.047 sec

@erquhart is this something you've seen elsewhere or is it something to do with how Gatsby and Netlify CMS are interacting with each other?

[erquhart]

@talves have you seen anything like this with the new plugin setup?

[mikheevm]

An update:
An error I provided in Additional info section of the issue also happens when running gatsby develop. After trying to run it multiple times (2-4), the error is gone.

[erquhart]

It really seems like this only impacts the very first deploy. Using the method @m-allanson suggested to isolate the issue, I'm seeing 13s deploys without Netlify CMS, and 18s with. Don't see anything in local or CI builds that would suggest a possibility of running out of memory. I don't know exactly why the initial build takes so long, but Netlify CMS does have it's own isolated build process, and it's now building from ES modules, I'm guessing that's what it comes down to.

[mikheevm]

Are you saying that the reason for this is the build of the CMS app?
I just tried to remove the plugin from the repo from issue description and got 11s (vs almost 100), the bundle is ok.
Seems the plugin is the case. Is it possible to disable the CMS app build when running gatsby build?

[talves]

@mikheevm I am looking at your repository now. I already see some things that are an issue. Let me review and I will get back to you here.

[talves]

First thing we need to fix is the dependencies for netlify-cms. See PR https://github.com/gnosis/dappcon-website/pull/92

This brought the bundle size from 6M back down to the expected ~3M and cut the build time of the cms bundle in half for me.

The new version (^4.0.0) of the plugin now requires netlify-cms-app@^2.9.1 rather than netlify-cms now. Let's start from here and see if you get back on track with where you were before.

[mikheevm]

@talves Thank you! It cut the build time but it's still far away from what it used to be 🙃

Before updating (v3):

Now (v4 with your fix):

[talves]

I am not sure what has changed that would make that difference.

BTW: my bootstrap on your repo took 67s, not anywhere near 14s and my javascript build was 51s not 91s the first run

On my second run I get:

[mikheevm]

@talves It's not only my repo, @m-allanson posted steps to reproduce the issue on newly created gatsby project by just adding gatsby-plugin-netlify-cms to the config.

About build times - ofc they depend on a configuration, but even 51s sounds a lot for 5 static pages

[talves]

@erquhart the only explanation is that v3 was running the build in development mode and there was no optimization of the bundle size prior to v4. If you went back and ran v3 vs v4 there is a drastic size difference in the bundle size. If you run gatsby develop on v4, you get a quick bundle build also.

I will run some builds on a clean react app without gatsby and see if we get the same build times to make sure this is what is expected.

[talves]

@mikheevm the build time for netlify-cms would have no bearing on the number of pages or content in the gatsby project. There is essentially two javascript builds going on in gatsby when you have the netlify-cms plugin installed.

[mikheevm]

@talves I think now I get it. Thank you for your help :)

[talves]

@mikheevm Also, you might want to ask the gatsby community about running a clean (rimraf .cache public) on each build and development. The public should be ok, but if I bypass the clean on subsequent builds there is a big drop on the gatsby side.

[mikheevm]

@talves We'll try to remove it and see how it goes. The docs say it should work: https://www.gatsbyjs.org/docs/debugging-cache-issues/
Thank you!

[talves]

@mikheevm I am only seeing 12s build times on my gatsby projects with javascript using this same plugin, so I am not sure this is 100% the netlify-cms plugin. I even have over 14 previews on the one I am bench-marking against.

[DanielRuf]

Is this related to some antivirus scanner or how the content is fetched (local or through network, SSD or HDD)?

[talves]

@mikheevm I just realized my build times are against my gatsby plugin (@talves/[email protected]) which have the changes prior to the changes made by @wardpeet prior to publishing the new version.

Would you mind testing against that plugin version to see what results you get?

All you have to do is change:

• in package.json to "@talves/gatsby-plugin-netlify-cms": "2.3.4"
• change plugin in gatsby-config.js to resolve: '@talves/gatsby-plugin-netlify-cms'

When I use my plugin version, the build drops from 44s to 11s

[talves]

@DanielRuf not that those would not affect it, but that is not the core issue at this time.

[mikheevm]

@talves I changed plugins and it made no difference for me

[talves]

@mikheevm if you have time. I will be in the netlify-cms chat for awhile. @talves

[mikheevm]

@talves it’s too late for me, would be really glad to chat another day though 🙂

[talves]

Using default gatsby starter (gatsby new)

Build without netlify-cms

success Building production JavaScript and CSS bundles - 2.187 s
success Building static HTML for pages - 0.463 s — 4/4 19.44 pages/second
info Done building in 6.252 sec

Build with gatsby-plugin-netlify-cms

success Building production JavaScript and CSS bundles - 8.752 s
success Building static HTML for pages - 0.470 s — 4/4 18.60 pages/second
info Done building in 12.879 sec

Build with @talves/gatsby-plugin-netlify-cms

success Building production JavaScript and CSS bundles - 8.660 s
success Building static HTML for pages - 0.475 s — 4/4 18.57 pages/second
info Done building in 13.025 sec

You are right @mikheevm , plugins are the same and when using the cache builds, I am getting the build times above which match what I see on all my Gatsby builds.

@erquhart I am seeing these times for the build and on a CRA build I am seeing a build in about 11s, so I would say everything is working as expected. Each individual system is going to have different results and should benchmark against that same system. I am not saying these build times are acceptable for a static site generation, just that netlify-cms is going to add this overhead due to it's size and configuration.

[mikheevm]

@talves does it make sense to add an option to disable CMS app build when building prod bundle? In our case our production website lives on AWS, when the cms is on netlify (we didn't understand if it was possible to migrate it to AWS). In our case it would be helpful, not sure for others though.

[chrfritsch]

I am having the same issue. I have all the latest version of gatsby and gatsby-plugin-netlify-cms and without gatsby-plugin-netlify-cms the "Building production JavaScript and CSS bundles" step takes 12 seconds and with the plugin it takes over 70 seconds.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!