Nuxt.js: Nuxt runs out of memory while generating thousands of pages

[uhyo]

Version

v2.4.5

Reproduction link

https://github.com/uhyo/gengou-yosou/tree/flyio

Steps to reproduce

Run npx nuxt generate.

What is expected ?

Nuxt should generate 4889 static pages (4 from pages and 4885 as instructed in nuxt.config.js) without exhausting memory.

What is actually happening?

JavaScript heap out of memory error occurs (on my PC with 8GB of memory).

ℹ Production build                                                           10:45:02
✔ Builder initialized                                                        10:45:02
✔ Nuxt files generated                                                       10:45:02

✔ Client
  Compiled successfully in 8.73s

✔ Server
  Compiled successfully in 3.71s


Hash: 2b064fae45a5ab220fa8
Version: webpack 4.29.5
Time: 8730ms
Built at: 2019-02-28 10:45:13
                         Asset       Size  Chunks             Chunk Names
../server/client.manifest.json   6.32 KiB          [emitted]  
       105b44c228445b1c9246.js   39.7 KiB       1  [emitted]  app
       1b7b70fde28ccc903939.js   2.97 KiB       7  [emitted]  pages/search
       3257a673f63e2552412d.js    155 KiB       2  [emitted]  commons.app
       6065dc6a071d304c8a5c.js    2.7 KiB       5  [emitted]  pages/list/_id
       686fc8e50276ec882be9.js   3.65 KiB       3  [emitted]  pages/_gengou/index
       6a105754fe9556767f41.js   3.14 KiB       8  [emitted]  pages/template/gengou
       6d633a92122a24985760.js   51.3 KiB       0  [emitted]  
       7572a37206b342264d67.js   1.29 KiB       4  [emitted]  pages/index
       7e424e0b065df14e8d3f.js  466 bytes       6  [emitted]  pages/random
                      LICENSES  486 bytes          [emitted]  
       f10fff28c29091d8d8ee.js    2.6 KiB       9  [emitted]  runtime
 + 2 hidden assets
Entrypoint app = f10fff28c29091d8d8ee.js 3257a673f63e2552412d.js 105b44c228445b1c9246.js

Hash: c7341f34eb9da355820a
Version: webpack 4.29.5
Time: 3712ms
Built at: 2019-02-28 10:45:16
                  Asset       Size  Chunks             Chunk Names
0e34725d5dff9dc18fcc.js   53.6 KiB    3, 7  [emitted]  pages/list/_id
1b8a280131b10484a5cf.js   53.6 KiB    5, 7  [emitted]  pages/search
1c342de4b10ae6a9765c.js  463 bytes       4  [emitted]  pages/random
426e227990f2f5736949.js   50.9 KiB       7  [emitted]  
4b13172a835655cba71b.js   3.08 KiB       6  [emitted]  pages/template/gengou
8d222bf0036c3f3690c2.js   3.42 KiB       1  [emitted]  pages/_gengou/index
ad89d28763f905c28785.js   1.43 KiB       2  [emitted]  pages/index
              server.js   28.9 KiB       0  [emitted]  app
   server.manifest.json  963 bytes          [emitted]  
 + 8 hidden assets
Entrypoint app = server.js server.js.map
ℹ Generating pages                                                           10:45:16

<--- Last few GCs --->

[6544:0x423ff70]    40055 ms: Mark-sweep 1359.9 (1446.8) -> 1348.4 (1449.3) MB, 805.1 / 0.0 ms  (average mu = 0.151, current mu = 0.071) allocation failure scavenge might not succeed
[6544:0x423ff70]    41139 ms: Mark-sweep 1362.1 (1449.3) -> 1350.7 (1451.8) MB, 1026.7 / 0.0 ms  (average mu = 0.098, current mu = 0.053) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x37fd8cedbe1d]
Security context: 0x1c3fe9a9e6e1 <JSObject>
    1: formatRaw(aka formatRaw) [0xa0c0c18fe09] [internal/util/inspect.js:~474] [pc=0x37fd8da125d7](this=0x2487ff4826f1 <undefined>,ctx=0x2c820b2c42d1 <Object map = 0x1e1f3c7c7591>,value=0x1c3fe9a8faa9 <JSFunction String (sfi = 0x139b35691519)>,recurseTimes=0)
    2: inspect(aka inspect) [0xa0c0c189bf9] [internal/util/inspect.js:~154] [pc=0x37fd8cf6cd39](thi...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8daaa0 node::Abort() [node]
 2: 0x8daaec  [node]
 3: 0xad73ce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad7604 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xec4c32  [node]
 6: 0xec4d38 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xed0e12 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xed1744 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xed43b1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe9d834 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x113cf9e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x37fd8cedbe1d 
zsh: abort (core dumped)  npx nuxt generate

Additional comments?

Sorry that I haven't managed to make a minimal reproduction.

_{This bug report is available on Nuxt community (#c8746)}

[pimlie]

@uhyo Please ignore my previous remarks

The problem is that you use a _lot_ of memory, each list page creates an array of 1000 items. While that isnt an issue normally, nuxt batch'es generating pages with a default concurrency of 500. It does this by chaining the generating of those routes in a Promise.all, but it seems the gc cannot run while that Promise has not finished. So node needs to keep 500 x 1000 items in memory. Which apparently is a problem.

The solution for you in this case is to set a lower generate.concurrency so the gc can run more often to free memory. Eg if I run with a concurrency of 10 the memory usages doesnt exceed ~400MB.

[uhyo]

@pimlie I didn't notice the generate.concurrency config; sorry about that.
I admit that it's not a bug as Nuxt has appropriate config for my case.
Thank you very much.