pm2 dump by JavaScript heap out of memory

[arainyu]

Your issue may already be reported!
Please search on the issue tracker before creating one.

What's going wrong?

How could we reproduce this issue?

Supporting information

<--- Last few GCs --->

[9144:0x3cfd9f0] 516170092 ms: Mark-sweep 1031.4 (1039.5) -> 611.8 (619.9) MB, 140.2 / 0.0 ms (+ 103.1 ms in 1 steps since start of marking, biggest step 103.1 ms) allocation failure GC in old space requested
[9144:0x3cfd9f0] 516187836 ms: Mark-sweep 1155.3 (1163.5) -> 551.4 (559.5) MB, 234.1 / 0.0 ms allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2bf63dc0d31
2: onconnection [cluster.js:~133] [pc=0x38fd5ecf2b8d](this=0x272468ad7151 ,err=-24,handle=0x2bf63d04311 )

==== Details ================================================

// optimized frame
--------- s...

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
1: node::Abort() [PM2 v2.1.6: God Daemon (/root/.pm2)]
2: 0x126492c [PM2 v2.1.6: God Daemon (/root/.pm2)]
3: v8::Utils::ReportOOMFailure(char const, bool) [PM2 v2.1.6: God Daemon (/root/.pm2)]
4: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [PM2 v2.1.6: God Daemon (/root/.pm2)]
5: v8::internal::Heap::AllocateUninitializedFixedArray(int) [PM2 v2.1.6: God Daemon (/root/.pm2)]
6: v8::internal::Factory::NewUninitializedFixedArray(int) [PM2 v2.1.6: God Daemon (/root/.pm2)]
7: 0xe072ef [PM2 v2.1.6: God Daemon (/root/.pm2)]
8: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object*, v8::internal::Isolate) [PM2 v2.1.6: God Daemon (/root/.pm2)]
9: 0x38fd5e4063a7
2017-07-25 20:27:19: ===============================================================================
2017-07-25 20:27:19: --- New PM2 Daemon started ----------------------------------------------------
2017-07-25 20:27:19: Time : Tue Jul 25 2017 20:27:19 GMT+0800 (HKT)
2017-07-25 20:27:19: PM2 version : 2.1.6
2017-07-25 20:27:19: Node.js version : 7.2.0
2017-07-25 20:27:19: Current arch : x64
2017-07-25 20:27:19: PM2 home : /root/.pm2
2017-07-25 20:27:19: PM2 PID file : /root/.pm2/pm2.pid
2017-07-25 20:27:19: RPC socket file : /root/.pm2/rpc.sock
2017-07-25 20:27:19: BUS socket file : /root/.pm2/pub.sock
2017-07-25 20:27:19: Application log path : /root/.pm2/logs
2017-07-25 20:27:19: Process dump file : /root/.pm2/dump.pm2
2017-07-25 20:27:19: Concurrent actions : 2
2017-07-25 20:27:19: SIGTERM timeout : 1600
2017-07-25 20:27:19: ===============================================================================

Please run the following command (available on PM2 >= 2.6)

$ pm2 report

[vmarchaud]

Not related to PM2, we can't control how much RAM your system is using.

[arainyu]

find the solution : https://github.com/nodejs/node/issues/10137

thanks

[fiftin]

@vmarchaud This issue related to PM2. I have server with 32Gb memory with only PM2 with 4 apps. And I received this error. PM2 crashed with all apps.

PM2 version: 2.10.3
Node: 8.7
OS: Amazon Linux AMI

<--- Last few GCs --->

[3325:0x37500c0] 4096090348 ms: Mark-sweep 1378.9 (1456.9) -> 1378.9 (1456.9) MB, 986.6 / 0.0 ms  allocation failure GC in old space requested
[3325:0x37500c0] 4096091351 ms: Mark-sweep 1378.9 (1456.9) -> 1378.9 (1407.9) MB, 1002.9 / 0.0 ms  last resort GC in old space requested
[3325:0x37500c0] 4096092329 ms: Mark-sweep 1378.9 (1407.9) -> 1378.9 (1407.9) MB, 978.3 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2ca5d37a5e71 <JSObject>
    2: clearBuffer(aka clearBuffer) [_stream_writable.js:476] [bytecode=0xb76337fd111 offset=41](this=0xe1ffa582311 <undefined>,stream=0x18336496dee1 <Socket map = 0x3679f363ecc1>,state=0x18336496e331 <WritableState map = 0x2db5a2ab4b89>)
    3: onwrite(aka onwrite) [_stream_writable.js:~422] [pc=0x14762a043ff](this=0xe1ffa582311 <undefined>,stream=0x18336496dee1 <Socket map = 0x3679f363ecc1>,er...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 2: 0x1217ecc [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 5: v8::internal::Factory::NewUninitializedFixedArray(int) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 6: 0xe1b6fc [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 7: 0xe35a78 [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 8: 0xe35fdb [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
 9: v8::internal::JSArray::SetLength(v8::internal::Handle<v8::internal::JSArray>, unsigned int) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
10: v8::internal::ArrayConstructInitializeElements(v8::internal::Handle<v8::internal::JSArray>, v8::internal::Arguments*) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
11: v8::internal::Runtime_NewArray(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v2.10.3: God Daemon (/home/ec2-user/.pm2)]
12: 0x1476220463d

3325 - PM2 PID

[tsabbay]

i dont think its PM2 releated. V8 not using your ram without you specifying it..
max_old_space_size should be set to apx 80% of what your system have.
for example
you mention that you have 32gb..
i would use 25000 as a value

[lucaswxp]

@fiftin Have you solved this? I have the same issue, apparently the PM2 God daemon is consuming too much memory, then when it reaches 1.5GB it gets killed. The problems are not the apps, the problem is pm2 daemon itself. @tsabbay How would I go about setting max_old_space_size to the daemon, and not the apps?

[tsabbay]

@fiftin i dont know, but I dont think that the PM2 deamon is your problem...

[Unitech]

please upgrade to latest pm2

$ npm install pm2 -g
$ pm2 update

[fiftin]

@lucaswxp reboot server, I no longer encountered this problem.

[lucaswxp]

@tsabbay In my case, it is the daemon. I have a crash dump reporting it. And it is not the kernel killing it for out of memory, I have plenty.

@Unitech Upgraded! Will wait and see. Thanks.
@fiftin If upgrade don't work, will try that. Thanks.

[lucaswxp]

The problem happened again. The daemon is occupying more than 1.5GB and node is killing it. Here is the crash report:

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x28052cedbe1d]
Security context: 0x00fd66d1e6e1 <JSObject>
    1: encode [0x38c551210e29] [/opt/node-v10.15.0-linux-x64/lib/node_modules/pm2/node_modules/amp/lib/encode.js:~16] [pc=0x2805311a16e5](this=0x1ed1de40aae9 <Object map = 0x9d2638ec9a1>,args=0x0a5b74506039 <JSArray[2]>)
    2: /* anonymous */ [0x157e9dd7fb09] [/opt/node-v10.15.0-linux-x64/lib/node_modules/pm2/lib/Daemon.js:~429] [pc=0x28053119e14d](this=0x3...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8db900 node::Abort() [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 2: 0x8db94c  [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 3: 0xad6c1e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 4: 0xad6e54 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 5: 0xec44e2  [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 6: 0xec45e8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 7: 0xed06c2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 8: 0xed0ff4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
 9: 0xed242f v8::internal::Heap::HandleGCRequest() [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
10: 0xe717aa v8::internal::StackGuard::HandleInterrupts() [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
11: 0x113c7c5 v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v3.5.0: God Daemon (/home/reduza/.pm2)]
12: 0x28052cedbe1d 

I know how to increase the memory space of my running apps, but how do I increase the memory used by PM2 daemon itself? I think there's a memory leak somewhere, this will eventually happen again even if I increase memory. The only plugin I'm using is log-autorotator.

[lucaswxp]

Any way to debug this further? It's crashing almost every day now.

[databu]

I have the same problem. It happened just _after_ one of my apps was restarted because of memory usage. I don't use any plugins.