Pm2: If PM2 is killed, the dump.pm2 file is empty

Created on 10 Apr 2017 · 35Comments · Source: Unitech/pm2

Hello.

~# pm2 start process.json && pm2 save

process.json

{
  "apps" : [{
    "name" : "zal.su",
    "script" : "/home/zal.su/app.js",
    "cwd" : "/home/zal.su",
    "error_file" : "/home/zal.su/log/err.log",
    "out_file" : "/home/zal.su/log/out.log",
    "node_args" : "--max-old-space-size=768",
    "max_memory_restart" : "768M",
    "watch" : true,
    "ignore_watch" : [
      "config",
      "log",
      "node_modules",
      "themes/*/public",
      "themes/.*",
      ".*"
    ],
    "watch_options" : {
      "followSymlinks": false
    },
    "vizion" : false
  }]
}

My log:

2017-04-10 19:30:25: pm2 has been killed by signal, dumping process list before exit...
2017-04-10 19:30:26: Deleting process 1
2017-04-10 19:30:26: Stopping app:zal.su id:1
2017-04-10 19:30:26: App [pm2-logrotate] with id [2] and pid [3817], exited with code [0] via signal [SIGTERM]
2017-04-10 19:30:26: App [zal.su] with id [1] and pid [11334], exited with code [0] via signal [SIGTERM]
2017-04-10 19:30:26: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:2
2017-04-10 19:30:26: App name:pm2-logrotate id:2 online
2017-04-10 19:30:26: pid=11334 msg=process killed
2017-04-10 19:30:26: Deleting process 2
2017-04-10 19:30:26: Stopping app:pm2-logrotate id:2
2017-04-10 19:30:26: App [pm2-logrotate] with id [2] and pid [12030], exited with code [0] via signal [SIGINT]
2017-04-10 19:30:27: pid=12030 msg=process killed
2017-04-10 19:30:27: [PM2] Exited peacefully

I reboot server, and my App zal.su not start.

2017-04-10 22:44:31: ===============================================================================
2017-04-10 22:44:31: --- New PM2 Daemon started ----------------------------------------------------
2017-04-10 22:44:31: Time                 : Mon Apr 10 2017 22:44:31 GMT+0300 (MSK)
2017-04-10 22:44:31: PM2 version          : 2.4.2
2017-04-10 22:44:31: Node.js version      : 6.10.2
2017-04-10 22:44:31: Current arch         : x64
2017-04-10 22:44:31: PM2 home             : /root/.pm2
2017-04-10 22:44:31: PM2 PID file         : /root/.pm2/pm2.pid
2017-04-10 22:44:31: RPC socket file      : /root/.pm2/rpc.sock
2017-04-10 22:44:31: BUS socket file      : /root/.pm2/pub.sock
2017-04-10 22:44:31: Application log path : /root/.pm2/logs
2017-04-10 22:44:31: Process dump file    : /root/.pm2/dump.pm2
2017-04-10 22:44:31: Concurrent actions   : 2
2017-04-10 22:44:31: SIGTERM timeout      : 1600
2017-04-10 22:44:31: ===============================================================================
2017-04-10 22:44:32: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-04-10 22:44:32: App name:pm2-logrotate id:0 online

Why did not the application start after reboot?

Need More Data Pending Release

Source

extensionsapp

👍2

Most helpful comment

I still had this issue, is this behaviour has anything to do with how the Linux system work? i am using Ubuntu 16.04 after reboot the dump file is empty.

nooballday on 18 Apr 2019

👍4

All 35 comments

Can you share the service unit file (/etc/systemd/system/pm2.service with systemd) ?

vmarchaud on 20 Apr 2017

cat /etc/systemd/system/pm2-root.service

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

extensionsapp on 20 Apr 2017

The dump does not exists after running pm2 start process.json && pm2 save?

After running this command can you run ll ~/.pm2/dump.pm2?

Unitech on 21 Apr 2017

# pm2 start process.json && pm2 save
[PM2][WARN] Applications menopo.space not running, starting...
[PM2] App [menopo.space] launched (1 instances)
┌──────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
│ App name     │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ watching │
├──────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
│ menopo.space │ 1  │ fork │ 1378 │ online │ 0       │ 0s     │ 50% │ 20.5 MB   │ enabled  │
└──────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 33.324 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app
[PM2] Saving current process list...
[PM2] Successfully saved in /root/.pm2/dump.pm2
# ls -l ~/.pm2/dump.pm2
-rw-r--r-- 1 root root 3189 Apr 26 18:28 /root/.pm2/dump.pm2
# pm2 logs
-bash: fork: Cannot allocate memory
# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬──────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu  │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼──────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 100% │ 31.711 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴──────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

after reboot, my app not start.

# cat .pm2/dump.pm2
[]
# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 32.301 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

extensionsapp on 26 Apr 2017

<--- Last few GCs --->

1207491733 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.5 / 0.0 ms [allocation failure].
1207492861 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].
1207497444 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.5 / 0.0 ms [allocation failure].
1207500246 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].
1207505179 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x20aa14ecfb39 <JS Object>
    1: stringify [native json.js:178] [pc=0x275ddf61c9b7] (this=0x20aa14ec90f9 <a JSON with map 0x1dd74670a0e1>,E=0x33c5ad3ff401 <an Object with map 0x390cd35a4ef9>,F=0x20aa14e04381 <undefined>,S=0x20aa14e04381 <undefined>)
    2: arguments adaptor frame: 1->3
    3: send [/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:~38] [pc=0x275ddfb181aa] (this=0x14677e13c649 <a PubSock...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 2: 0x10c9a4c [PM2 v2.4.4: God Daemon (/root/.pm2)]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 7: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 8: v8::internal::IncrementalStringBuilder::Extend() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 9: v8::internal::BasicJsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
10: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
11: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
12: v8::internal::BasicJsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
13: v8::internal::Runtime_BasicJSONStringify(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
14: 0x275dde8092a7
2017-05-01 11:50:37: ===============================================================================
2017-05-01 11:50:37: --- New PM2 Daemon started ----------------------------------------------------
2017-05-01 11:50:37: Time                 : Mon May 01 2017 11:50:37 GMT+0300 (MSK)
2017-05-01 11:50:37: PM2 version          : 2.4.4
2017-05-01 11:50:37: Node.js version      : 6.10.2
2017-05-01 11:50:37: Current arch         : x64
2017-05-01 11:50:37: PM2 home             : /root/.pm2
2017-05-01 11:50:37: PM2 PID file         : /root/.pm2/pm2.pid
2017-05-01 11:50:37: RPC socket file      : /root/.pm2/rpc.sock
2017-05-01 11:50:37: BUS socket file      : /root/.pm2/pub.sock
2017-05-01 11:50:37: Application log path : /root/.pm2/logs
2017-05-01 11:50:37: Process dump file    : /root/.pm2/dump.pm2
2017-05-01 11:50:37: Concurrent actions   : 2
2017-05-01 11:50:37: SIGTERM timeout      : 1600
2017-05-01 11:50:37: ===============================================================================
2017-05-01 11:50:38: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-05-01 11:50:38: App name:pm2-logrotate id:0 online
2017-05-01 11:50:38: Stopping app:pm2-logrotate id:0
2017-05-01 11:50:38: App [pm2-logrotate] with id [0] and pid [27004], exited with code [0] via signal [SIGINT]
2017-05-01 11:50:38: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-05-01 11:50:38: pid=27004 msg=process killed
2017-05-01 11:50:38: ===============================================================================
2017-05-01 11:50:38: --- Stopping PM2 --------------------------------------------------------------
2017-05-01 11:50:38: Time                 : Mon May 01 2017 11:50:38 GMT+0300 (MSK)
2017-05-01 11:50:38: ===============================================================================
2017-05-01 11:50:38: RPC closed
2017-05-01 11:50:38: PUB closed
2017-05-01 11:50:38: PM2 successfully stopped
2017-05-01 13:46:00: ===============================================================================
2017-05-01 13:46:00: --- New PM2 Daemon started ----------------------------------------------------
2017-05-01 13:46:00: Time                 : Mon May 01 2017 13:46:00 GMT+0300 (MSK)
2017-05-01 13:46:00: PM2 version          : 2.4.4
2017-05-01 13:46:00: Node.js version      : 6.10.2
2017-05-01 13:46:00: Current arch         : x64
2017-05-01 13:46:00: PM2 home             : /root/.pm2
2017-05-01 13:46:00: PM2 PID file         : /root/.pm2/pm2.pid
2017-05-01 13:46:00: RPC socket file      : /root/.pm2/rpc.sock
2017-05-01 13:46:00: BUS socket file      : /root/.pm2/pub.sock
2017-05-01 13:46:00: Application log path : /root/.pm2/logs
2017-05-01 13:46:00: Process dump file    : /root/.pm2/dump.pm2
2017-05-01 13:46:00: Concurrent actions   : 2
2017-05-01 13:46:00: SIGTERM timeout      : 1600
2017-05-01 13:46:00: ===============================================================================
2017-05-01 13:46:00: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-05-01 13:46:00: App name:pm2-logrotate id:0 online

cat /root/.pm2/dump.pm2
[...

...]

But if I reboot dump.pm2 empty!

cat /root/.pm2/dump.pm2
[]

pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 32.777 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

I need websites to be always online! Please do something.

extensionsapp on 1 May 2017

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory

Unitech on 1 May 2017

@Unitech There was a splash in the load, memory refused. But why PM2 does not restart the processes? And /root/.pm2/dump.pm2 - empty []?

extensionsapp on 1 May 2017

I have error ENOMEM - OK;
All process stop - OK;
I read file /root/.pm2/dump.pm2 - OK;
I reboot server - OK;
I read file /root/.pm2/dump.pm2 - EMPTY FILE [], Why?

2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: --- PM2 global error caught --------------------------------------------                                                                                                -------
2017-06-12 11:50:31: Time                 : Mon Jun 12 2017 11:50:31 GMT+0300 (MSK)
2017-06-12 11:50:31: spawn ENOMEM
2017-06-12 11:50:31: Error: spawn ENOMEM
    at exports._errnoException (util.js:1018:11)
    at ChildProcess.spawn (internal/child_process.js:319:11)
    at exports.spawn (child_process.js:378:9)
    at /usr/lib/node_modules/pm2/lib/TreeKill.js:33:14
    at buildProcessTree (/usr/lib/node_modules/pm2/lib/TreeKill.js:79:12)
    at module.exports (/usr/lib/node_modules/pm2/lib/TreeKill.js:32:5)
    at Object.God.killProcess (/usr/lib/node_modules/pm2/lib/God/Methods.js:223:7)
    at Object.God.stopProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:265:9)
    at Object.God.restartProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:362:11                                                                                                )
    at /usr/lib/node_modules/pm2/lib/God/ActionMethods.js:395:20
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: [PM2] Resurrecting PM2
internal/child_process.js:319
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at exports._errnoException (util.js:1018:11)
    at ChildProcess.spawn (internal/child_process.js:319:11)
    at exports.spawn (child_process.js:378:9)
    at /usr/lib/node_modules/pm2/lib/TreeKill.js:33:14
    at buildProcessTree (/usr/lib/node_modules/pm2/lib/TreeKill.js:79:12)
    at module.exports (/usr/lib/node_modules/pm2/lib/TreeKill.js:32:5)
    at Object.God.killProcess (/usr/lib/node_modules/pm2/lib/God/Methods.js:223:7)
    at Object.God.stopProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:265:9)
    at Object.God.restartProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:362:11                                                                                                )
    at /usr/lib/node_modules/pm2/lib/God/ActionMethods.js:395:20
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: --- New PM2 Daemon started ---------------------------------------------                                                                                                -------
2017-06-12 11:50:31: Time                 : Mon Jun 12 2017 11:50:31 GMT+0300 (MSK)
2017-06-12 11:50:31: PM2 version          : 2.4.4
2017-06-12 11:50:31: Node.js version      : 6.10.2
2017-06-12 11:50:31: Current arch         : x64
2017-06-12 11:50:31: PM2 home             : /root/.pm2
2017-06-12 11:50:31: PM2 PID file         : /root/.pm2/pm2.pid
2017-06-12 11:50:31: RPC socket file      : /root/.pm2/rpc.sock
2017-06-12 11:50:31: BUS socket file      : /root/.pm2/pub.sock
2017-06-12 11:50:31: Application log path : /root/.pm2/logs
2017-06-12 11:50:31: Process dump file    : /root/.pm2/dump.pm2
2017-06-12 11:50:31: Concurrent actions   : 2
2017-06-12 11:50:31: SIGTERM timeout      : 1600
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:32: Starting execution sequence in -fork mode- for app name:pm2-logrotate id                                                                                                :0
2017-06-12 11:50:32: App name:pm2-logrotate id:0 online
2017-06-12 11:50:32: Stopping app:pm2-logrotate id:0
2017-06-12 11:50:32: App [pm2-logrotate] with id [0] and pid [19674], exited with code [0] vi                                                                                                a signal [SIGINT]
2017-06-12 11:50:32: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-06-12 11:50:32: pid=19674 msg=process killed
2017-06-12 11:50:32: ========================================================================                                                                                                =======
2017-06-12 11:50:32: --- Stopping PM2 -------------------------------------------------------                                                                                                -------
2017-06-12 11:50:32: Time                 : Mon Jun 12 2017 11:50:32 GMT+0300 (MSK)
2017-06-12 11:50:32: ========================================================================                                                                                                =======
2017-06-12 11:50:32: RPC closed
2017-06-12 11:50:32: PUB closed
2017-06-12 11:50:32: PM2 successfully stopped

Please, tell me why after the reboot the /root/.pm2/dump.pm2 file becomes empty []?

After reboot I have memory

~# free
             total       used       free     shared    buffers     cached
Mem:       1024416     237764     786652      14168      19252     104068
-/+ buffers/cache:     114444     909972
Swap:            0          0          0

but app`s not start.

extensionsapp on 12 Jun 2017

This is probably the biggest problem with PM2, since I can not be sure that my sites will have 99.99% uptime.

extensionsapp on 12 Jun 2017

👍3

2017-07-31 00:00:14: Change detected on path get.js for app test.website.org - restarting
2017-07-31 00:00:14: Stopping app:test.website.org id:12
2017-07-31 00:00:14: App [test.website.org] with id [12] and pid [6444], exited with code [0] via signal [SIGINT]
2017-07-31 00:00:14: pid=6444 msg=process killed
2017-07-31 00:00:14: Starting execution sequence in -fork mode- for app name:test.website.org id:12
2017-07-31 00:00:14: App name:test.website.org id:12 online

<--- Last few GCs --->

4172772419 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 1.1 / 0.0 ms [allocation failure].
4172787059 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 0.5 / 0.0 ms [allocation failure].
4172800147 ms: Scavenge 20.6 (27.0) -> 19.7 (27.0) MB, 0.9 / 0.0 ms [allocation failure].
4172813613 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 1.1 / 0.0 ms [allocation failure].
4172814008 ms: Scavenge 20.7 (27.0) -> 19.8 (27.0) MB, 1.5 / 0.0 ms [allocation failure].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1520f60cfb39 <JS Object>
    1: parse [native json.js:~61] [pc=0x3bd85db7ad21] (this=0x1520f60c90f9 <a JSON with map 0x2037fb90a0e1>,C=0x94d94dfff59 <String[91]: {"type":"axm:monitor","data":{"Loop delay":{"value":"0.52ms","agg_type":"avg","alert":{}}}}>,w=0x1520f6004381 <undefined>)
    2: arguments adaptor frame: 1->2
    3: onread [internal/child_process.js:457] [pc=0x3bd85db3880e] (this=0x24bedbc649d1 <a Pipe with ...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 2: 0x10c9a4c [PM2 v2.4.4: God Daemon (/root/.pm2)]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 7: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 8: v8::internal::Handle<v8::internal::String> v8::internal::JsonParser<false>::ScanJsonString<true>() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 9: v8::internal::JsonParser<false>::ParseJsonObject() [PM2 v2.4.4: God Daemon (/root/.pm2)]
10: v8::internal::JsonParser<false>::ParseJsonValue() [PM2 v2.4.4: God Daemon (/root/.pm2)]
11: v8::internal::JsonParser<false>::ParseJson() [PM2 v2.4.4: God Daemon (/root/.pm2)]
12: v8::internal::JsonParser<false>::Parse(v8::internal::Handle<v8::internal::String>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
13: v8::internal::Runtime_ParseJson(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
14: 0x3bd85d4092a7
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: --- New PM2 Daemon started ----------------------------------------------------
2017-07-31 00:00:19: Time                 : Mon Jul 31 2017 00:00:19 GMT+0300 (MSK)
2017-07-31 00:00:19: PM2 version          : 2.4.4
2017-07-31 00:00:19: Node.js version      : 6.10.2
2017-07-31 00:00:19: Current arch         : x64
2017-07-31 00:00:19: PM2 home             : /root/.pm2
2017-07-31 00:00:19: PM2 PID file         : /root/.pm2/pm2.pid
2017-07-31 00:00:19: RPC socket file      : /root/.pm2/rpc.sock
2017-07-31 00:00:19: BUS socket file      : /root/.pm2/pub.sock
2017-07-31 00:00:19: Application log path : /root/.pm2/logs
2017-07-31 00:00:19: Process dump file    : /root/.pm2/dump.pm2
2017-07-31 00:00:19: Concurrent actions   : 2
2017-07-31 00:00:19: SIGTERM timeout      : 1600
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-07-31 00:00:19: App name:pm2-logrotate id:0 online
2017-07-31 00:00:19: Stopping app:pm2-logrotate id:0
2017-07-31 00:00:19: App [pm2-logrotate] with id [0] and pid [6545], exited with code [0] via signal [SIGINT]
2017-07-31 00:00:19: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-07-31 00:00:19: pid=6545 msg=process killed
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: --- Stopping PM2 --------------------------------------------------------------
2017-07-31 00:00:19: Time                 : Mon Jul 31 2017 00:00:19 GMT+0300 (MSK)
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: RPC closed
2017-07-31 00:00:19: PUB closed
2017-07-31 00:00:19: PM2 successfully stopped
2017-07-31 00:01:43: ===============================================================================
2017-07-31 00:01:43: --- New PM2 Daemon started ----------------------------------------------------
2017-07-31 00:01:43: Time                 : Mon Jul 31 2017 00:01:43 GMT+0300 (MSK)
2017-07-31 00:01:43: PM2 version          : 2.4.4
2017-07-31 00:01:43: Node.js version      : 6.10.2
2017-07-31 00:01:43: Current arch         : x64
2017-07-31 00:01:43: PM2 home             : /root/.pm2
2017-07-31 00:01:43: PM2 PID file         : /root/.pm2/pm2.pid
2017-07-31 00:01:43: RPC socket file      : /root/.pm2/rpc.sock
2017-07-31 00:01:43: BUS socket file      : /root/.pm2/pub.sock
2017-07-31 00:01:43: Application log path : /root/.pm2/logs
2017-07-31 00:01:43: Process dump file    : /root/.pm2/dump.pm2
2017-07-31 00:01:43: Concurrent actions   : 2
2017-07-31 00:01:43: SIGTERM timeout      : 1600
2017-07-31 00:01:43: ===============================================================================
2017-07-31 00:01:43: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-07-31 00:01:43: App name:pm2-logrotate id:0 online

Add automatic start sites after the fall.

extensionsapp on 30 Jul 2017

Please correct one single problem.

!!!Do not clear the dump file after rebooting the server!!!

If I have error "Error: spawn ENOMEM", my dump file:

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

After reboot:

~# cat /root/.pm2/dump.pm2
[]

Why is it empty []?

extensionsapp on 24 Sep 2017

@Unitech, @vmarchaud please, read my message, step-by-step.

Start all process and pm2 save;
In a few hours ERROR: spawn ENOMEM or process out of memory;

~# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┬──────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │ user │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┼──────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 37.957 MB   │ root │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┴──────┘
 Use `pm2 show <id|name>` to get more details about an app

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

cp /root/.pm2/dump.pm2 /root/.pm2/dump2.pm2
reboot

~# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┬──────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │ user │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┼──────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 37.957 MB   │ root │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┴──────┘
 Use `pm2 show <id|name>` to get more details about an app

~# cat /root/.pm2/dump.pm2
[]

dump2 > dump. cp /root/.pm2/dump2.pm2 /root/.pm2/dump.pm2
resurrect. pm2 resurrect

[PM2] Resurrecting
[PM2] Restoring processes located in /root/.pm2/dump.pm2
[PM2] Process /test1/app.js restored
[PM2] Process /test2/app.js restored
[PM2] Process /test3/app.js restored
[PM2] Process /test4/app.js restored
[PM2] Process /test5/app.js restored
[PM2] Process /test6/app.js restored
[PM2] Process /test7/app.js restored
[PM2] Process /test8/app.js restored
[PM2] Process /test9/app.js restored
[PM2] Process /test10/app.js restored
...
NICE

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

Why is it empty [] dump after reboot?

extensionsapp on 26 Sep 2017

Steps to reproduce:

systemctl start pm2-node.service
pm2 start someapp
pm2 save

Now dump.pm2 is okay.
Then kill pm2:

sudo kill -9 $(pgrep PM2)

Now dump.pm2 is empty and the process list is lost, no way to resurrect.
That's insane and totally makes pm2 unreliable!

radkodinev on 22 Oct 2017

@radkodinev @extensionsapp Hello guys! Did you solve problem ? i've shocked about exist it and i faced it with it too.

denisenkoaj on 21 Feb 2018

I've tried to set flag +i for dump file, it is solve problem , but i have next - i use keymetrics monitoring service, and if file have +i flag - keymetrics cannot write to it and cannot monitoring my server. It's problem if you use this service.

denisenkoaj on 21 Feb 2018

will be fixed via: https://github.com/Unitech/pm2/pull/3510

Unitech on 2 Mar 2018

If you want to get the fix now you can use the dev branch:

$ npm install Unitech/pm2#development -g

Unitech on 9 Apr 2018

@Unitech

If you want to get the fix now you can use the dev branch:

$ npm install Unitech/pm2#development -g

it seems the problem is still here.

shixukai on 20 Apr 2018

I also found this issue.

jazarja on 18 May 2018

We're also experiencing this. Is this planned to be added soon? Because this is a major bug imo.

dreaganluna on 2 Jul 2018

Indeed, it is a major bug IMO as well. I cannot believe so much time has passed and it still hasn't been resolved. I strongly dislike products that have many fancy features but cannot get the basics right. I really hope this gets fixed very soon.

radkodinev on 2 Jul 2018

I've stoped paying to keymetrics due this bug, after i wrote them on mail about it, they wrote comment

If you want to get the fix now you can use the dev branch:
npm install Unitech/pm2#development -g

but it's yet here, and i think it's really major bug

denisenkoaj on 3 Jul 2018

PM2 3.0.0 has been released:

$ npm install pm2@latest -g
$ pm2 update

Be aware of breaking changes, especially drop support for node 0.12.
Full list here : https://github.com/Unitech/pm2/blob/master/CHANGELOG.md

This is an important issue we really want to solve.
Any feedback and information are welcome.

Feel free to reopen if version 3.0.0 doesn't solve it.

wallet77 on 4 Jul 2018

This caused problem on one of our production server, dump file gets emptied. Exact same problem described as the first post. I should probably re create a new issue, right?

@wallet77

entrptaher on 30 Dec 2018

were you able to solve it

marko-mlinarevic on 31 Mar 2019

@marko-mlinarevic unfortunately I do not think so. But sometimes when we did pm2 update, it would get stuck and no process would generate. After that the pm2 list would be empty. We had to do pm2 resurrect and sometimes it would stay empty, sometimes it would work.

Really curious about such behavior.

entrptaher on 31 Mar 2019

I still had this issue, is this behaviour has anything to do with how the Linux system work? i am using Ubuntu 16.04 after reboot the dump file is empty.

nooballday on 18 Apr 2019

👍4

Running PM2 version 3.5.1 and this issues is still occurring on Ubuntu. I suggest reopening this issue.

Steps to reproduce. Start App and save process list.

pm2 start someAppName
pm2 save

Then show running processes

pm2 status

Then kill PM2 to simulate some type of crash

sudo kill -9 $(pgrep PM2)

Dump file is now gone and trying to resurrect or start the app does not work.

This fails

pm2 start someAppName

As does this

pm2 resurrect

What is going on? Is there a fix in sight for us unlucky souls?

tomchomiak on 12 Dec 2019

👍3

current issue on [email protected] is still here
Ubuntu 18.04 LTS (GNU/Linux 4.15.0-20-generic x86_64)

2020-01-25T00:50:40: PM2 log: App [mongodb:1] starting in -fork mode-
2020-01-25T00:50:40: PM2 log: App [mongodb:1] online
2020-01-25T00:50:40: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:40: PM2 log: App [parser:2] starting in -cluster mode-
2020-01-25T00:50:40: PM2 log: App [parser_dev:3] starting in -fork mode-
2020-01-25T00:50:40: PM2 log: App [parser_dev:3] online
2020-01-25T00:50:41: PM2 log: App [parser:2] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:4] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:4] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:5] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:5] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:6] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:6] online
2020-01-25T00:51:13: PM2 log: Stopping app:pm2-logrotate id:0
2020-01-25T00:51:13: PM2 log: Stopping app:mongodb id:1
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:2
2020-01-25T00:51:13: PM2 log: Stopping app:parser_dev id:3
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:4
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:5
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:6
2020-01-25T00:51:14: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App name:parser id:6 disconnected
2020-01-25T00:51:14: PM2 log: App [parser_dev:3] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:4] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:6] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:2] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:5] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App name:parser id:4 disconnected
2020-01-25T00:51:14: PM2 log: App name:parser id:5 disconnected
2020-01-25T00:51:14: PM2 log: App name:parser id:2 disconnected
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=415 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=724 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=717 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=710 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=703 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=735 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: App [mongodb:1] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: pid=702 msg=process killed
2020-01-25T00:51:14: PM2 log: PM2 successfully stopped
2020-01-25T00:51:39: PM2 log: ===============================================================================
2020-01-25T00:51:39: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2020-01-25T00:51:39: PM2 log: Time                 : Sat Jan 25 2020 00:51:39 GMT+0300 (Moscow Standard Time)
2020-01-25T00:51:39: PM2 log: PM2 version          : 4.2.3
2020-01-25T00:51:39: PM2 log: Node.js version      : 13.3.0
2020-01-25T00:51:39: PM2 log: Current arch         : x64
2020-01-25T00:51:39: PM2 log: PM2 home             : /root/.pm2
2020-01-25T00:51:39: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2020-01-25T00:51:39: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2020-01-25T00:51:39: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2020-01-25T00:51:39: PM2 log: Application log path : /root/.pm2/logs
2020-01-25T00:51:39: PM2 log: Worker Interval      : 30000
2020-01-25T00:51:39: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2020-01-25T00:51:39: PM2 log: Concurrent actions   : 2
2020-01-25T00:51:39: PM2 log: SIGTERM timeout      : 1600
2020-01-25T00:51:39: PM2 log: ===============================================================================
2020-01-25T00:51:41: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2020-01-25T00:51:41: PM2 log: App [pm2-logrotate:0] online

content of /etc/systemd/system/pm2-root.service

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/root/.nvm/versions/node/v13.3.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
Restart=on-failure

ExecStart=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Pashted on 24 Jan 2020

Up the issue. Empty dump file after pm2 (4.2.0) process shutdown.

batrudinych on 27 Mar 2020

in 4.4.0 version my problem was solved. thx

Pashted on 12 Jul 2020

Had this issue on 4.4.0. Solve it by removing pm2-logrotate, which was deleting the file for no reason

realSaddy on 16 Jul 2020

👍2

seems that it won't be fixed :D

чт, 16 июл. 2020 г. в 10:36, Jack notifications@github.com:

Having this issue in 4.4.0

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Unitech/pm2/issues/2827#issuecomment-659152969, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AEJDCQ2ZPLGFNOVT3ZKV54DR3Z7UTANCNFSM4DHFEMCA
.