Gatsby: Gatsby V2 can't handle the generation of BIG sites.

[rametta]

Description

Gatsby V2 can't handle the generation of very big sites. We build around 2000 pages (however, the data is those pages is pretty big, e-commerce...), and the build step ends up failing.
We narrowed it down to this line of code: https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/redux/index.js#L82
stringify seems to crash with error:

const dataStr = JSON.stringify(data, null, 2)
                     ^
RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at Object.<anonymous> (/Users/xxx/Code/stringy-tests/index.js:17:22)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3
error Command failed with exit code 1.

Note that this error comes from the reproducible snippet here: https://codepen.io/rametta/pen/xyQpRb

Steps to reproduce

Use this Codepen to reproduce the JSON.stringify crashing: https://codepen.io/rametta/pen/xyQpRb
Open the codepen, change the 5 to 500000 then open the devtools console to see the error message.
We we're able to build our website by using the streamed JSON.stringify technique instead. PR coming soon.

Expected result

The site should build for extra large JSON objects going through JSON.stringify()

Actual result

JSON.stringify can't handle it and crashes.

Environment

```
System:
OS: macOS High Sierra 10.13.6
CPU: x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 8.11.1 - /usr/local/bin/node
Yarn: 1.6.0 - /usr/local/bin/yarn
npm: 5.6.0 - /usr/local/bin/npm
Browsers:
Chrome: 69.0.3497.100
Firefox: 52.0.2
Safari: 12.0
npmPackages:
gatsby: ^2.0.5 => 2.0.9
gatsby-image: ^2.0.5 => 2.0.10
gatsby-plugin-accessibilityjs: 1.0.3 => 1.0.3
gatsby-plugin-google-tagmanager: 2.0.5 => 2.0.5
gatsby-plugin-manifest: 2.0.5 => 2.0.2
gatsby-plugin-react-helmet: 3.0.0 => 3.0.0
gatsby-plugin-robots-txt: 1.3.0 => 1.3.0
gatsby-plugin-sass: 2.0.1 => 2.0.1
gatsby-plugin-sharp: ^2.0.5 => 2.0.5
gatsby-plugin-styled-components: 3.0.0 => 3.0.0
gatsby-source-filesystem: ^2.0.1 => 2.0.1
gatsby-transformer-sharp: ^2.1.1 => 2.1.2
npmGlobalPackages:
gatsby-cli: 2.4.3

[DSchau]

@rametta would you be able to try out the latest version of Gatsby (at least 2.0.18)? We recently worked on some cache improvements, which led to some pretty significant improvements as it relates to building out large sites.

See #8435 for additional context

[Whipgit]

Hey @DSchau we have the same issue with Gatsby v2.0.30.

[DSchau]

This sounds like it's specific to the redux state being serialized, rather than anything to do with caching then (which we've seen most frequently before!).

I think we can follow a similar approach, and if we can segment the redux state to separate, smaller files/pieces (and then re-assemble from cache on startup) that would solve this issue! Not sure of the best approach on this yet, but in general, if we're able to accomplish a few of the following:

• Move off of JSON.stringify for _entire_ redux state--which seems to be the bottleneck here
• Chunk the redux state into disparate pieces, rather than the whole state at once

Accomplishing one or both of those would have some pretty big wins, here, I'd think.

[rametta]

@DSchau I opened a PR with a fix, let me know what you think. #9370

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[lannonbr]

@rametta Is this problem still coming up for you or can we close this issue?