Gatsby: [email protected] Build failure (JavaScript heap out of memory)

[moonmeister]

Description

Gatsby 2.11 breaks site build. I'm guessing it's the new functionality around building node_modules as well as the site (#14111).

i was updating dependencies on master when build started breaking. I've updated to 2.10.5 on an update branch with no troubles but as soon as I move Gatsby to 2.11 the build breaks.

Steps to reproduce

1. Clone master or update branches from https://github.com/moonmeister/moonmeister.net
2. update gatsby dep to [email protected]
3. run build => see breakage.

Expected result

build should be successful.

Actual result

[BABEL] Note: The code generator has deoptimised the styling of /home/ajmoon/code/moonmeister/moonmeister.net/node_modules/netlify-cms-app/dist/netlify-cms-app.js as it
⠦ Building production JavaScript and CSS bundles

<--- Last few GCs --->

[19712:0x2dbca30]    45370 ms: Scavenge 1338.2 (1423.9) -> 1337.3 (1423.9) MB, 2.9 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45374 ms: Scavenge 1338.4 (1423.9) -> 1337.5 (1423.9) MB, 2.8 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45378 ms: Scavenge 1338.6 (1423.9) -> 1337.7 (1424.4) MB, 2.6 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x34eb54adbe1d]
    1: StubFrame [pc: 0x34eb54a875c2]
Security context: 0x2caa7a21e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x23804dd52ac9] [/home/ajmoon/code/moonmeister/moonmeister.net/node_modules/@babel/core/lib/transformation/file/merge-map.js:~155] [pc=0x34eb5534b963](this=0x01389b5022b1 <null>,m=0x1c2e6adbae29 <Object map = 0x11c6eb590b11>)
    3: arguments adaptor frame: 3->1
    4: forEach...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 2: 0x8dc55c  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 5: 0xec7bf2  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
12: 0x34eb54adbe1d 
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

  System:
    OS: Linux 5.0 Ubuntu 19.04 (Disco Dingo)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.15.2 - ~/.nvm/versions/node/v10.15.3/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Browsers:
    Chrome: 75.0.3770.100
    Firefox: 67.0.4
  npmPackages:
    gatsby: 2.11.0 => 2.11.0 
    gatsby-image: ^2.0.34 => 2.2.3 
    gatsby-plugin-eslint: ^2.0.4 => 2.0.5 
    gatsby-plugin-manifest: ^2.0.24 => 2.2.0 
    gatsby-plugin-netlify: ^2.0.13 => 2.1.0 
    gatsby-plugin-netlify-cms: ^4.1.0 => 4.1.0 
    gatsby-plugin-offline: ^2.0.25 => 2.2.0 
    gatsby-plugin-react-helmet: ^3.0.10 => 3.1.0 
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0 
    gatsby-plugin-sass: ^2.0.11 => 2.1.0 
    gatsby-plugin-sharp: ^2.0.30 => 2.2.1 
    gatsby-plugin-sitemap: ^2.0.11 => 2.2.0 
    gatsby-remark-images: ^3.0.10 => 3.1.2 
    gatsby-remark-relative-images: ^0.2.2 => 0.2.2 
    gatsby-source-filesystem: ^2.0.27 => 2.1.1 
    gatsby-transformer-remark: ^2.3.8 => 2.5.0 
    gatsby-transformer-sharp: ^2.1.17 => 2.2.0 
  npmGlobalPackages:
    gatsby-cli: 2.7.2
    gatsby-dev-cli: 2.5.0

[moonmeister]

Okay, so this is definitely a node issue. It seems that the extra transpiling being done is just now making node hit it's default limit in my case. I ran export NODE_OPTIONS=--max_old_space_size=4096 to expand the defaults and build is now happy.

this seems strange that node doesn't just request more memory if it's available, also I'm a little concerned how this might affect folks build servers.

[moonmeister]

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:

[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"

[moonmeister]

This seems like it might negatively affect a lot of people. Any way we can make this more memory efficient?

[vickylance]

@moonmeister OMG thank you so much I just updated my package.json file and suddenly this issue crept up. And yes increasing the size solved the issue.

[moonmeister]

@vickylance figured I wouldn't be the only one. Glad you figured it out.

[vickylance]

However netlify does not seem to build even after increasing the node size. Its not throwing an error but its just staying building forever.

EDIT: it got published. But it took a lot of time to build

[Zellement]

I've got the same issue I think, @vickylance and @moonmeister - I can test the environment variable fix if someone can guide me how? :) Not done them before - is it just as here in Netlify?

[moonmeister]

Not sure you need the quotes but that looks correct.

[Zellement]

OK - didn't work with the quotes initially. I'll try without.

EDIT: Works without quotes in Netlify

[moonmeister]

Yeah, build times have definitely increased. I'm not sure what value this adds, I'm sure it does but it seems to have some significant downsides too.

[vickylance]

Add in netlify like this.

[vickylance]

@moonmeister Yes my build time was less than a min and now its more than 6 mins

[Zellement]

Same here; environment variable fixed it, but it took an age to deploy.

Not sure about you guys but I also got a lot of lines of...

10:04:46 PM: Pkg: FLAGS_MISMATCH

Not seen that before myself.

EDIT: In Netlify
EDIT 2: Gone from 4 mins to 7 mins.

[moonmeister]

@gatsbyjs/core Can we get some input here? This isn't strictly a Gatsby issue but definitely caused by a decision the Gatsby project made. IS there a way to disable this for folks that don't want it, or maybe a way to optimize the process to increase performance/build times (or at least not break node)?

[wardpeet]

It only took 20-40s longer on gatsbyjs.org when transpiling node_modules so it's weird that it's so heavy for your sites. I'll have a look at your repro! @moonmeister

[moonmeister]

@wardpeet Thanks! My site is only page (aside from the cms), so it wasn't much to start with...but it probably still doubled.

If anyone else has repositories they can share that may be helpful.

[homearanya]

Same problem on my side. This has been driving me bananas. I'm glad to find out it's an issue on Gatsby's last version

[wardpeet]

Seems like gatsby-netlify-cms being the issue. I'll cook up a fix.

Sorry for the inconveniences

[wardpeet]

https://github.com/gatsbyjs/gatsby/pull/15191 seems to be ready to get merged!

[moonmeister]

@wardpeet can you explain why this plugin specifically is an issue? Is it purely its size?

[erquhart]

gatsby-plugin-netlify-cms runs an independent Webpack build within Gatsby's Webpack build (insert "yo dawg" reference here) using a modified version of the Gatsby Webpack config. Gatsby 2.11.0 no longer excludes node_modules, which causes the plugin's Webpack build to attempt processing the Netlify CMS modules with Babel. The modules are massive and also prebuilt, so it will (and should) crash your system.

The PR I raised just adds that exclusion back in to the plugin's Webpack config. Please don't override your node max file size limit to fix this!

[bscs-dev-team]

Hi all, this had us stumped for a minute.

Thanks, @moonmeister for posting a temporary fix.

Yes, it increased our build time quite a bit too but at least it is building again.

@erquhart - are you suggesting we shouldn't use the temp fix?

[erquhart]

Sorry, no - definitely good as a temp fix, just recommending folks revert back once a proper fix is released.

[Jamamuuga]

Will #15191 be sufficient for a developer who uses gatsby-plugin-netlify-cms & uses Yarn's PnP feature?

[hendra-go]

I also experienced the same thing, high memory usage on gatsby build.

after investigated a little further, i think this block of code from https://github.com/gatsbyjs/gatsby/pull/14111/commits/4210885b605f61d49fec786509441bb28dc0570e commit that caused the problem:

    let js = ({ exclude = [], ...options } = {}) => {
      const excludeRegex = [
        `core-js|event-source-polyfill|webpack-hot-middleware/client`,
      ].concat(exclude)

      return {
        test: /\.(js|mjs|jsx)$/,
        exclude: new RegExp(excludeRegex.join(`|`)),
        type: `javascript/auto`,
        use: [loaders.js(options)],
      }

I succesfully running gatsby build after revert webpack-utils.js back to this https://github.com/gatsbyjs/gatsby/pull/14111/commits/d46f17b8fe7a9eb566f257b0c8174ae273660875 commit

  {
    let js = (options = {}) => {
      return {
        test: /\.(js|mjs|jsx)$/,
        exclude: /core-js|event-source-polyfill|webpack-hot-middleware\/client/,
        type: `javascript/auto`,
        use: [loaders.js(options)],
      }
    }

    rules.js = js
  }

I suspect the RegExp causing this memory inefficiency

[erquhart]

@hendra-go is the project you're seeing this issue in using the Netlify CMS plug-in?

_{Sent with GitHawk}

[hendra-go]

@hendra-go is the project you're seeing this issue in using the Netlify CMS plug-in?

@erquhart No, i dont use Netlify CMS plug-in, i try to build locally, using [email protected]
even using node --max-old-space-size=4096 when running gatsby build did not work on my laptop with 8GB memory.

[erquhart]

Interesting, maybe Gatsby itself is having the same issues the Netlify CMS plug-in build is having. It's likely due to the commit you mentioned (from #14111), but I don't think it's a RegExp perf issue - seems more probable that it's due to Babel now processing modules that used to be excluded, especially in node_modules. I could easily be wrong though, @wardpeet should have more insight.

_{Sent with GitHawk}

[ssijak]

I am too experiencing this problem with Gatsby 2.11.x versions and not with 2.10. I downgraded for now.

[SpicyPete]

Also getting this issue with Gatsby 2.11.x while using styled-components. Sticking with 2.10 for now.

[hupe1980]

I have the same issue. A downgrade to 2.10 solved the problem

[wardpeet]

@hendra-go mind sharing your repo?

[wardpeet]

new version of gatsby-netlify-cms got published 4.1.1, it should fix this issue.

[pogo19]

It seems fixed for me. Thanks!

[SpicyPete]

Still broken for me when using styled-components 🤔

[rwieruch]

I ran into this today as well:

[BABEL] Note: The code generator has deoptimised the styling of undefined 
as it exceeds the max of 500KB.

It happens for a large markdown file (exactly this long read blog post which I want to transfer from my old Hugo website to my new Gatsby website). In Gatsby I use PrismJS and MDX. The 1600th line of markdown kills it.

So I am not entirely sure whether it is related, because I am not using Netlify. If not related, I am happy to open a new issue.

---

My dependencies:

  "dependencies": {
    "@mdx-js/mdx": "^1.0.21",
    "@mdx-js/react": "^1.0.21",
    "axios": "^0.19.0",
    "disqus-react": "^1.0.6",
    "dotenv": "^7.0.0",
    "gatsby": "^2.12.0",
    "gatsby-image": "^2.2.3",
    "gatsby-link": "^2.2.0",
    "gatsby-mdx": "^0.6.3",
    "gatsby-plugin-catch-links": "^2.1.0",
    "gatsby-plugin-feed": "^2.3.1",
    "gatsby-plugin-google-analytics": "^2.1.1",
    "gatsby-plugin-manifest": "^2.2.0",
    "gatsby-plugin-offline": "^2.2.0",
    "gatsby-plugin-react-helmet": "^3.1.0",
    "gatsby-plugin-sentry": "^1.0.1",
    "gatsby-plugin-sharp": "^2.2.1",
    "gatsby-plugin-sitemap": "^2.2.1",
    "gatsby-plugin-styled-components": "^3.1.0",
    "gatsby-remark-autolink-headers": "^2.1.0",
    "gatsby-remark-copy-linked-files": "^2.1.0",
    "gatsby-remark-images": "^3.1.2",
    "gatsby-remark-prismjs": "^3.3.0",
    "gatsby-source-filesystem": "^2.1.1",
    "gatsby-transformer-remark": "^2.5.0",
    "gatsby-transformer-sharp": "^2.2.0",
    "prismjs": "^1.16.0",
    "react": "next",
    "react-dom": "next",
    "react-facebook-pixel": "^0.1.1",
    "react-helmet": "~5.2.1",
    "react-quora-pixel": "0.0.5",
    "react-twitter-conversion-tracker": "^1.0.0",
    "react-youtube": "^7.9.0",
    "styled-components": "^4.3.2"
  },

Before I had Gatsby 2.3.25 and ran into the same issue (in case we want to exclude a regression bug). Then I updated full of hope that a new version would resolve it.

[SpicyPete]

Gatsby 2.12.x seemed to of fixed all of my issues

[wernerglinka]

I am following the Gatsby tutorial from https://www.netlifycms.org/docs/gatsby/
I am running Gatsby 2.13.2, build on Netlify crashed but the fix NODE_OPTIONS = "--max_old_space_size=4096" did the job.

[moonmeister]

@wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure.

[jamesgrubb]

Having same issue here.

[erquhart]

This was fixed - are you sure it's the same exact issue? Also make sure you have the latest version of both Gatsby and the Netlify CMS Gatsby plugin.

_{Sent with GitHawk}

[wernerglinka]

Thanks Alex,
I updated to netlify-cms-app and everything works.
Werner

Sent from my iPad

On Jul 4, 2019, at 19:20, Alex Moon [e-mail] wrote:

@wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[jamesgrubb]

I added the environment variable --max-old-space-size=4096 to netlify and managed a successful build. I deleted lock files deleted node_modules and re-installed. Same issue as far as I can `see.

"dependencies": {
    "gatsby": "^2.13.4",
    "gatsby-plugin-netlify-cms": "^4.1.1",
    "netlify-cms-app": "^2.9.5",
    "react": "^16.8.6",
    "react-dom": "^16.8.6"
  }

[erquhart]

Can you help us reproduce? Repo we can clone?

_{Sent with GitHawk}

[flamebeard]

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:

[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"

This worked for me <3

[rdela]

Added "(JavaScript heap out of memory)" to end of issue title just now (in addition to reopening yesterday) to try and prevent more duplicate issues like #15540

[wardpeet]

I'll be closing this one in favor of #15256

Please move the discussion to that thread. I have created a PR that hopefully fixes the OOM issue https://github.com/gatsbyjs/gatsby/pull/15588

[rdela]

#15588 #15591 has done the trick for me thanks @wardpeet!

[vientang]

I just got hit with this. Gatsby version 2.13.31, Node 10.15.3 via npx and npm 6.4.1

I had imported a file that was 8mb, which was working just a few days ago. @moonmeister 's solution worked for me - increase memory with export NODE_OPTIONS=--max_old_space_size=4096