pm2 Memory Leak

Created on 13 Jun 2018  路  15Comments  路  Source: Unitech/pm2

What's going wrong?

pm2 slowly (over the course of an hour uses LOTS of RAM)!! And often runs at near 100% of CPU.

How could we reproduce this issue?

Use pm2 2.10.4 to run a node Process and wait.

Supporting information

npm 5.6.0
pm2 2.10.4
Centos v6.9
Node v8.9.4

```

--- PM2 REPORT (Wed Jun 13 2018 09:15:24 GMT+0100 (BST)) ----------------------

--- Daemon -------------------------------------------------
pm2d version : 2.10.4
node version : 8.9.4
node path : /usr/bin/pm2
argv : /root/.nvm/versions/node/v8.9.4/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0 : node
user : root
uid : 0
gid : 0

uptime : 70min

--- CLI ----------------------------------------------------
local pm2 : 2.10.4
node version : 8.9.4
node path : /usr/bin/pm2
argv : /root/.nvm/versions/node/v8.9.4/bin/node,/usr/bin/pm2,report
argv0 : node
user : root
uid : 0

gid : 0

--- System info --------------------------------------------
arch : x64
platform : linux
type : Linux
cpus : AMD Opteron(tm) Processor 6380
cpus nb : 8
freemem : 4429713408
totalmem : 16717795328
home : /root

Please copy/paste the above report in your issue on https://github.com/Unitech/pm2/issues

In progress

Most helpful comment

I have just updated and the problem persists....
As you can see in the "~/.pm2/agent.log", something makes the agent to crash and "re-initiate" over and over again..... i had to run "pkill -f Agent" to avoid the server crash, but whitout the agents no data is sent to pm2.io website 馃憥 , but its better no data then the a server crash....

===============================================================================
--- PM2 REPORT (Tue Aug 14 2018 22:59:15 GMT+0000 (Coordinated Universal Time))
===============================================================================
--- Daemon --------------------------------------------------------------------
pm2d version         : 3.0.3
node version         : 10.8.0
node path            : not found
argv                 : /usr/local/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 69min
===============================================================================
--- CLI -----------------------------------------------------------------------
local pm2            : 3.0.3
node version         : 10.8.0
node path            : not found
argv                 : /usr/local/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info ---------------------------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
cpus nb              : 2
freemem              : 1918013440
totalmem             : 4142145536
2018-08-14T22:38:51.986Z interactor:daemon Launching Interactor RPC server (bind to /home/ubuntu/.pm2/interactor.sock)
2018-08-14T22:38:53.109Z interactor:daemon Send to process daemon is started
2018-08-14T22:38:53.112Z interactor:reverse Reverse interactor is listening
2018-08-14T22:38:53.112Z interactor:daemon Interactor daemon started
-- FATAL EXCEPTION happened --
2018-08-14T22:38:53.113Z
Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
    at process.target.send (internal/child_process.js:628:16)
    at verifyEndpointCallback (/usr/lib/node_modules/pm2/node_modules/@pm2/agent/src/InteractorDaemon.js:338:17)
    at async.each (/usr/lib/node_modules/pm2/node_modules/@pm2/agent/src/TransporterInterface.js:79:7)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:473:16
    at iterateeCallback (/usr/lib/node_modules/pm2/node_modules/async/dist/async.js:992:24)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:969:16
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:3888:9
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:473:16
    at iteratorCallback (/usr/lib/node_modules/pm2/node_modules/async/dist/async.js:1064:13)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:969:16
Re-initiating Agent
[PM2 Agent] Using (Public key: 1rvp3hqt572s982) (Private key: xxx) (Info node: https://root.keymetrics.io)
2018-08-14T22:38:53.123Z interactor:pm2:interface PM2Interface instantiate
2018-08-14T22:38:53.259Z interactor:daemon Succesfully launched new agent
2018-08-14T22:38:53.259Z interactor:daemon Exiting Interactor
2018-08-14T22:38:53.260Z interactor:reverse Reverse interactor is no longer listening

All 15 comments

I have observer the same error for version 3.x and 2.9.x. pm2 start a lot of pm2 list process. Each of them uses about 0.7 % of total memory. When I tried to kill them using ps aux|grep -i pm2|awk '{print $2}'| xargs kill -9 but npm start it over and over again. If anyone knows workaround please shere with me.

me too

We inspect pm2 under heavy load before every release and we have not noticed any leak.

To report a potencial leak:

$ pm2 install pm2-probe

When PM2 is "leaking memory" please send a screenshot of htop filtered on PM2 + the pm2 ls command

me too @ 3.0.2
bug-pm2

htop:
screen shot 2018-07-30 at 14 20 56
screen shot 2018-07-30 at 14 21 31

pm2 ls:
screen shot 2018-07-30 at 14 21 54

CPU and RAM on that machine. I installed pm2 3.0.2 on Thuesday
screen shot 2018-07-30 at 14 22 52

The Camparound app is a sails.js server.

Can you upgrade to latest, some fixes have been landed in latest agent:

Before run:
pkill -f Agent

Then

npm install pm2@latest -g
pm2 update

It has been fixed, just kill all agent before updating pkill -f Agent.
Checked again in some production server, the previous agent was trying to resurrect PM2.

i have same problem,
my step:
i kill instance "pm2"
-npm install pm2@latest -g
-pm2 update

After 15 min i have multi instance node and pm2 agent with only one app

i run pm2 version 3.0.3
image

Again:

pkill -f Agent
npm install pm2@latest -g
pm2 update

Same problem..

image

Your service works well "PM2 v3.0.3: God" to kill all "PM2 Agent v0.5" instances, but Node instances are still multiplying

image

this problem happens when I turn on the monitor for "app.pm2.io"

Can you run:

pm2 kill

Then check if there are Agent still running?

Also @swoofer could you please display the content of the logfile ~/.pm2/agent.log thanks

Fixed & Published, just update:

npm install pm2@latest -g
pm2 update

I have just updated and the problem persists....
As you can see in the "~/.pm2/agent.log", something makes the agent to crash and "re-initiate" over and over again..... i had to run "pkill -f Agent" to avoid the server crash, but whitout the agents no data is sent to pm2.io website 馃憥 , but its better no data then the a server crash....

===============================================================================
--- PM2 REPORT (Tue Aug 14 2018 22:59:15 GMT+0000 (Coordinated Universal Time))
===============================================================================
--- Daemon --------------------------------------------------------------------
pm2d version         : 3.0.3
node version         : 10.8.0
node path            : not found
argv                 : /usr/local/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 69min
===============================================================================
--- CLI -----------------------------------------------------------------------
local pm2            : 3.0.3
node version         : 10.8.0
node path            : not found
argv                 : /usr/local/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info ---------------------------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
cpus nb              : 2
freemem              : 1918013440
totalmem             : 4142145536
2018-08-14T22:38:51.986Z interactor:daemon Launching Interactor RPC server (bind to /home/ubuntu/.pm2/interactor.sock)
2018-08-14T22:38:53.109Z interactor:daemon Send to process daemon is started
2018-08-14T22:38:53.112Z interactor:reverse Reverse interactor is listening
2018-08-14T22:38:53.112Z interactor:daemon Interactor daemon started
-- FATAL EXCEPTION happened --
2018-08-14T22:38:53.113Z
Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
    at process.target.send (internal/child_process.js:628:16)
    at verifyEndpointCallback (/usr/lib/node_modules/pm2/node_modules/@pm2/agent/src/InteractorDaemon.js:338:17)
    at async.each (/usr/lib/node_modules/pm2/node_modules/@pm2/agent/src/TransporterInterface.js:79:7)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:473:16
    at iterateeCallback (/usr/lib/node_modules/pm2/node_modules/async/dist/async.js:992:24)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:969:16
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:3888:9
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:473:16
    at iteratorCallback (/usr/lib/node_modules/pm2/node_modules/async/dist/async.js:1064:13)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:969:16
Re-initiating Agent
[PM2 Agent] Using (Public key: 1rvp3hqt572s982) (Private key: xxx) (Info node: https://root.keymetrics.io)
2018-08-14T22:38:53.123Z interactor:pm2:interface PM2Interface instantiate
2018-08-14T22:38:53.259Z interactor:daemon Succesfully launched new agent
2018-08-14T22:38:53.259Z interactor:daemon Exiting Interactor
2018-08-14T22:38:53.260Z interactor:reverse Reverse interactor is no longer listening
Was this page helpful?
0 / 5 - 0 ratings

Related issues

alexpts picture alexpts  路  3Comments

morfies picture morfies  路  3Comments

chaos-git picture chaos-git  路  3Comments

jubairsaidi picture jubairsaidi  路  3Comments

phra picture phra  路  3Comments