Pm2: PM2 using too much RAM and CPU

[frontierpsycho]

Now, I know this issue is potentially very case-specific, but I'd like to get your thoughts on it, and also let you know of it.

I have written a nodejs service that is supposed to run forever, listen to a queue, and act upon the messages. The actions taken include appending to a file and writing to a database. The entire thing is asynchronous.

Since I was going to run this on a dual-core machine, and potentially even more cores in the future, I wanted to run more than one instance. I wanted to use PM2's clustering, because it seemed very convenient, and PM2 in general seems to come with lots of goodies which I liked (the web interface, clustering, log aggregation, etc). I also considered forever, which has only the basic of PM2's features.

However, after running two instances of the service for a day with each of forever and PM2, the results are:

• forever: ~250mb RAM, ~20% CPU usage
• PM2: ~1,250mb RAM, ~100% CPU usage

The machines are virtual, and exactly the same. I tried with both node 0.10 and 0.12, and the results were the same.

Because under PM2, the instances are draining the machine's resources, the service performs atrociously (sometimes even consuming 10 times more slowly).

My question is mainly, are you aware of any performance weaknesses of PM2? Are there cases where PM2 is not the correct solution? Do you think you could fix/optimize the problem? It seems likely that this is a memory leak (probably not in my code, as forever seems to run it quite well).

I would have really liked to use PM2 for this.

[jshkurti]

Hello, can you try it with npm i -g git://github.com/Unitech/PM2#development version please ?
I removed a feature which I suspect to be the cause of the memory leak.

[jshkurti]

Also, which version of PM2 are you using ?
Are your apps in fork mode or cluster mode ?

[frontierpsycho]

My PM2 reports version 0.12.15, and I've been using cluster mode.

I'm going to try to use the development version as soon as possible, although that might not be very very soon. Thanks for the reply, though.

[Unitech]

When you report such apparently critical things please attach a screenshot of pm2 resources usage with htop.

We use PM2 under heavy load on 6 differents servers, with cluster mode with all keymetrics features and we never seen that.

[frontierpsycho]

Unfortunately, I was working on vagrant machines which I've now destroyed.

To be clear, I'm not suggesting PM2 is always leaking memory and that it's useless, but I've seen a couple of mentions of this online, so I believe under certain circumstances this might be true. Hopefully the bugfix jshkurti mentioned will fix those.

[Unitech]

Ok we keep this issue open for the week if there is someone else experiencing that issue.

Which modules were you using? (queue etc)

[frontierpsycho]

No modules.

[Unitech]

Could you send us a gist of your app so we can try to reproduce the issue?

Thanks

[frontierpsycho]

Unfortunately I can't, I wrote it at my work and it's not open-source. Plus you probably can't reproduce it, as it isn't standalone, it interacts with several systems in our architecture, a queue, the DB, etc. If I have time, I can try to present you with a pared down version. I'm more likely to just try the dev version and see how that works, though, to be honest.

[frontierpsycho]

Alright, so I tried the development version, left it to work for a day (although without much traffic), and it seems to work okay! I got a memory usage of ~250mb, and reasonable CPU usage.

So, what is the fix? Is it going to be released soon?

[Unitech]

Thanks for trying it!

We are still deep inspecting the issue, would it be possible to have a sample of your code that uses the http or the https module?

[elijahchancey]

+1.

I'm experiencing a similar issue. I've installed PM2 and have not yet configured any apps (haven't yet configured fork vs cluster). Using NVM to install node v0.12.4 and then using npm the latest pm2. Here's an htop screenshot:
https://www.dropbox.com/s/j0v4p1d9kui0pg2/Screenshot%202015-06-07%2022.19.24.png?dl=0
Ubuntu 14.04.2
PM2 0.12.16

Let me know what other information I can provide.

[Unitech]

Just spawned a Ubuntu 14.04 on AWS, with 2 cores and 4go, with Node.js 0.12.4:

What did you do before to get such a high CPU/Memory usage?

[Unitech]

Bug reproduced - inspecting

[frontierpsycho]

BTW, I don't really use the http or https modules directly, I use this:

https://github.com/almost/sqs-readable-stream

and this:

https://github.com/felixge/node-mysql/

[askhogan]

@frontierpsycho @elijahchancey

Add this to your project and retry

require('pmx').init({
    http: false, 
    errors: false,
    custom_probes: false,
    network: false, 
    ports: false  
});

My memory issue was solved after setting http to false. By default http is on.

https://github.com/keymetrics/pmx/issues/23

[frontierpsycho]

Wait, I wasn't using keymetrics. Does PM2 by default send data to keymetrics?

Also, I won't be able to try it, as I opted for a different solution right now (forever). Moreover, since the development version fixed it for me, I guess after that's released, it will no longer be an issue.

[askhogan]

If you weren't using keymetrics, then it shouldn't send data. If you setup keymetrics it does.

Glad to hear the dev version solved it for you. Will give that a shot.

[6a68]

@Unitech I see you landed some commits related to this issue. Have you verified that it's resolved?

[ajschlosser]

I am having this issue using PM2 0.14.7 on Ubuntu 14.04 with Node.js 4.2.1.

We have four cores; PM2 always uses >100% of one of them, and ~50% of another. It also uses >90% of our available RAM.

We have tried various combinations of Node.js and PM2 versions to no avail.

[qspencer]

Same problem - Ubuntu 14.04, 8 GB, 4 CPUs (on AWS EC2 if that matters), Node.js 4.1.2, PM2 0.15.7

[soyuka]

Actually it also uses a lot of cpu, can we have last 100 lines of ~/.pm2/pm2.log? The json configuration might be interesting too, thx.

[qspencer]

Not using json configuration (unless there's a default). Log attached.

On Thu, Oct 29, 2015 at 2:55 PM, Antoine Bluchet [email protected]
wrote:

Actually it also uses a lot of cpu, can we have last 100 lines of
~/.pm2/pm2.log? The json configuration might be interesting too, thx.

—
Reply to this email directly or view it on GitHub
https://github.com/Unitech/pm2/issues/1314#issuecomment-152286442.

[soyuka]

Do you use options (like watch)?

[ratsbomb]

+1

I'm using CentOS 6, pm2 version 0.15.10(node v4.2.2, npm v.3.3.12)
before pm2 start, TOP

top - 14:40:03 up 151 days, 31 min,  4 users,  load average: 1.52, 1.87, 1.67
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.0%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1016228k total,   867120k used,   149108k free,     9340k buffers
Swap:  4095996k total,  1507392k used,  2588604k free,    87068k cached

after pm2 start, TOP

top - 14:40:45 up 151 days, 32 min,  4 users,  load average: 0.95, 1.65, 1.60
Tasks: 124 total,   2 running, 122 sleeping,   0 stopped,   0 zombie
Cpu(s): 57.4%us, 12.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 30.2%st
Mem:   1016228k total,   927696k used,    88532k free,     9364k buffers
Swap:  4095996k total,  1507448k used,  2588548k free,    87048k cached

[kurenov]

I am also experiencing same problem. Ubuntu 14 x64 Single Core.
nodejs 0.12.7
npm 2.11.3
pm2 1.0.0

[Fantus]

node v5.4.1
pm2 1.0.0
system: aws linux, small instance single core with 2 gig memory
processes to manage: 6

memory rising over the day until it reaches over a gigabyte. the system dies then as pm2 grepped all the memory.

we are using pm2 to manage our worker processes. atm the only solution for us is to kill pm2 on a regular basis and then restart all worker.

[ldijkstra]

I seem to be having an issue with monit.
The monit reports 100% usage for all processes,
https://www.dropbox.com/s/0392ucq8p41bzid/Screenshot%202016-03-15%2015.29.29.png?dl=0
whereas htop reports something completely different:
https://www.dropbox.com/s/o8m1eyxl0txb6hx/Screenshot%202016-03-15%2015.33.19.png?dl=0

[kerberjg]

I can confirm the same problem with:

• Ubuntu Server 14.04 LTS x64
• Node.js v4.4.3
• PM2 v 1.1.1

Memory keeps piling up and reaches as high as 300MB. This is true for both the PM2 God Daemon and the PM2 Keymetrics.io Agent

EDIT: I wasn't able to confirm the 100% usage with my eyes, but htop reports as much as 3~5h of total CPU time after a week (also both for PM2 and the Keymetrics Agent).

I'm running 2-4 Node apps with it

[Tomino2112]

Same problem here. PM2 is maxing out CPU. When running app manually it uses 0.1% of cpu...

After some debugging, it seems that the files watching is the problem. Once I remove watch config from ecosystem file, pm2 runs fine.

Here is our watch config:

"watch": true,
    "ignore_watch": ["./**/*.ts","./**/*.map",".git",".idea","./**/*.log"],
    "watch_options": {
      "usePolling": true,
      "useFsEvents": false,
      "atomic": false,
      "followSymLinks": false
    },

[Tomino2112]

Discard what I have said about the watch option above, problem is still there even if not watching.

This is fresh install of pm2 (and whole system to that matter), linked to keymetrics no app running

[Unitech]

CPU usage has been reduced a LOT with pm2 v2:

$ npm install Unitech/pm2#development -g
$ pm2 update

[Tomino2112]

I can confirm that so far it works great, but this is development version, is it stable? when will it be officially released ?

[Unitech]

Awesome! Yes it's stable, it's been 2 months that we are working on PM2 V2. Right now we are doing monitored production checks and stress tests and it works great so far! I'm quite happy about this new release

[kzjeef]

I still have the issue with latest pm2...
pm2 --version
2.0.0-rc3

It will happens when fork mode and cluster mode.

[soyuka]

@kzjeef can you give more informations?

[kzjeef]

Do you need more log or information dump ?

I tried upgrade pm2 to latest by:

npm i -g git://github.com/Unitech/PM2#development

my nodejs:
node -v
v0.12.7

My server:
cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

[soyuka]

I'd like more informations about what you're running. You say fork and
cluster, can we have an extract of your package.json? Are you running any
specific framework ? What is wrong with pm2 memory/cpu consumption in your
case?
Thanks.
Le jeu. 7 juil. 2016 à 08:28, kzjeef [email protected] a écrit :

Do you need more log or information dump ?

I tried upgrade pm2 to latest by:

npm i -g git://github.com/Unitech/PM2#development

my nodejs:
node -v
v0.12.7

My server:
cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Unitech/pm2/issues/1314#issuecomment-230990651, or mute
the thread
https://github.com/notifications/unsubscribe/ABQr89kib2phd09d8gnu9_QpC9eqhMNCks5qTJxxgaJpZM4EtSJP
.

[kzjeef]

@soyuka

framework is: Nodejs + express

My issue is the memory usage is increase very fast.
I run my server with forever, the memory of server is about 97M and for long time.
but when I use pm2 run my server, it increase very fast from 97M to 100, and 200M.

[...]

And the whole npm list
[...]

[soyuka]

How do you start the app, do you have a pm2 json declaration?

After doing npm i -g git://github.com/Unitech/PM2#development did you launch pm2 updatePM2?

Btw, you should not stay on node 0.12, better use LTS (v4 or v0.10), I don't think 0.12 was ever considered stable.

[kzjeef]

yes, I did the pm2 update to restart pm2.

what you mean pm2 json declearation ?

Yes, I want to upgrade, but these a lot of dependency. I will do the upgrade to LTS version soon.
:-(

[soyuka]

How do you start your app with pm2?

[kzjeef]

start with this command pm2 start bin/api.js -i 0

[Unitech]

@kzjeef to help you in this, follow this steps:

$ npm install Unitech/pm2#dev-v8 -g
$ pm2 update

This special dev-v8 branch has v8-profiler embedded. To start to CPU profile PM2 do:

$ pm2 profile start
# Wait some time, open HTOP look at usage resources
$ pm2 profile stop

A cpu profiling file will be generated.
Once you have this file send it to us or inspect it with the chrome developer tool to see where the CPU is used.

Once we have this info we will be able to go further

Thank you

[Tomino2112]

Here you go https://www.dropbox.com/s/vpp8iscpd10bulq/Fr-15%253A52%253A15.cpuprofile?dl=0

[Unitech]

Thanks,

95,60% (program) ?

[Unitech]

Ok before digging further on the (program) mystery would you mind to link your pm2 with keymetrics.io and install the probe:

$ pm2 install pm2-probe

If you could share with me your bucket name at: contact AT keymetrics.io that would be great, thanks

[Unitech]

Just popped up an Amazon EC2 m4.large instance (2 vcpu + 8gb memory) on Ubuntu 14.04 and runned:

$ sudo apt-get update
$ sudo apt-get install git
$ curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.2/install.sh | bash
$ nvm install 4.1.2
$ nvm alias default 4.1.2
$ npm install Unitech/pm2#development -g
$ pm2 ls

Then running commands:

$ pm2 link <secret> <public> EC2CHECK

Then starting an app in cluster mode:

$ pm2 start pm2/examples/child.js -i 4

PM2 CPU usage vary between 0% and 1.3%

I cannot reproduce that issue... Did you followed the same steps to install the entire kitchen sink @kzjeef @Tomino2112 @kurenov ?

System summary:

ubuntu@ip-172-31-0-156:~/pm2$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
ubuntu@ip-172-31-0-156:~/pm2$ node -v
v4.1.2
ubuntu@ip-172-31-0-156:~/pm2$ pm2 --version
2.0.0-rc3

[Unitech]

working great so far

[Unitech]

Ok I cannot reproduce this issue on that environment,

After 12 hours with 4 clustered app and 2 modules installed I get this result:

30mb PM2 2.0 memory usage in average, that's quite a lap comparing to the 1.1.3 (~80mb)
With the keymetrics agent it heads to 55mb memory usage.
Also CPU usage in idle and under load is reduced by 70%.

[Unitech]

Add 2 apps flooding the stdout/stderr:

[Unitech]

@frontierpsycho @elijahchancey @6a68 @ajschlosser @qspencer @ratsbomb @kurenov @Fantus @kerberjg @kzjeef
I'm closing this issue, please feel free to reopen if you run into this issue.

Make sure you installed PM2 v2 before:

$ npm install Unitech/pm2#development -g
$ pm2 update

[Tomino2112]

Asking me to give you access to my AWS box and closing issue just to be able to release? ouch...

Anyways,

https://app.keymetrics.io/#/bucket/577a812fd08ea4b43bce4097/apps-signs

[Tomino2112]

[Unitech]

Thank you @Tomino2112 for help! I just reopened the issue,

I've upgrade your bucket so we can make further inspection and started a stress test on Node.js 6.2.2.

Could we see your ecosystem.json configuration?
How did you installed Node.js? (NVM?)
Which Linux distro are you using?

[Tomino2112]

I was going to ask if there is upgrade in it for me :D

Ecosystem config:

{
  "apps":[{
    "name": "ipp-api-2",
    "script": "app.js",
    "watch": true,
    "ignore_watch": ["./**/*.ts","./**/*.map",".git",".idea","./**/*.log"],
    "watch_options": {
      "usePolling": true,
      "useFsEvents": false,
      "atomic": false,
      "followSymLinks": false
    },
    "env_development": {
      "NODE_ENV": "development",
      "DEBUG": "*"
    },
    "env_production": {
      "NODE_ENV": "production"
    }
  }]
}

• Node installed via NVM
• Linux distro: Ubuntu 14.04 trusty 64 bit

[Unitech]

;)

@soyuka any thoughts about the watch options?
@Tomino2112 could you run pm2 kill and pm2 ls to see if the CPU is back to normal without applications?

[Tomino2112]

The watch options are only because of my vagrant dev that is guest on windows. I did try without watch options before and same issue, but that was on 1.x version of pm2

Will kill all apps now, but can do only for about 30 minutes right now

[Unitech]

Yep! CPU back to normal!

[Tomino2112]

ok done, btw apps were not running before, only probe, and probe is not being killed with pm2 kill

[soyuka]

@Unitech watch options are really bad for CPU here (ie polling). Also I have the feeling that you're watching node_modules, which is again bad for CPU and memory (especially with many dependencies)

[Tomino2112]

I can try and start the app without watch options, but as I mentioned above the app was not running (was stopped) when you were watching probe stats.

[Tomino2112]

Also after I installed pm2 v2 and with running app cpu was 1%, it went all mayham only after some time it was running, and didnt go back to normal even after stopping the app

[jiangtao]

I have the same problem. PM2 can help me manage my application. But the CPU is always near to 100% with cluster mode.
OS: centos 65
Node: 0.10.25

[Tomino2112]

now app is running without watch options

[soyuka]

chokidar with polling will do fs.Stat on every watched file almost every 100 ms

[Tomino2112]

you can see some spikes now, I did load test. It goes back to normal so thats great, but as I said, it goes wrong after XXX time its being running

[Unitech]

@Tomino2112 alright definitely comes from the watch on node_modules, that folder must be ignored! ++ "ignore_watch" : ["node_modules", "client/img"]. Ok if you say so about waiting some time before it goes wrong, keep the application like that and we will see how it behaves in the following hours

@Jerret321 do you use watch also? Could you copy paste the ecosystem file or how did you started the app? Which PM2, Node.js version are you using?

[Tomino2112]

You cna already see some odd spikes to 50-100% and now there is minimal usage of the app.

[Unitech]

That's normal, there is a worker inspecting your apps, running every 30 seconds, it does not interfere with your app performance at all

[Tomino2112]

I will leave this running for couple of hours and then switch the watch options back on and see if it makes difference

[jiangtao]

@Unitech Thanks for your help.
version:
NodeJS: 0.10.41
PM2: 1.1.3

I don't use watch option. I start my app with config.json:

[{
      name      : "app",
      script    : "",
      exec_mode: "cluster",
      instances: "max",
      env: {
          "MONGO_URL":"",
          "ROOT_URL":"http://localhost:1001",        
          "PORT":"1001",
      },
      env_production : {
        NODE_ENV: "production"
      }
    }]

[Unitech]

@Jerret321 please upgrade to a newer Node.js version, the 4.x LTS is a great fit. Or use PM2 v2 with the command:

$ npm install Unitech/pm2#development -g
$ pm2 update

@Tomino2112 alright let's see how it behaves

[jiangtao]

@Unitech sorry, I can't update the Node.js version. Because of the meteor framework that was based on Node.js 0.10.41.
I have update the pm2 version. I will be focusing this comment and feedback the condition when the users are in used tomorrow.

[Unitech]

@Tomino2112

So the result after 12 hours of testing, confirms that your high CPU usage come from the watch option. Just ignore the node_modules in the ignore_watch and everything will work as expected.

[Tomino2112]

Im sorry I didnt re-enabled the watch options, will do one. First I will re-eanble as it was before to see if same problem comes back, then I will change the watch ignore and see if it goes away

[Unitech]

@Tomino2112 just add the node_modules into the ignore_watch option or for sure the CPU will go crazy again. Refer to what @soyuka said about that

[Tomino2112]

Updated watch option so far so good, Will keep monitoring during the day. Thanks! btw how do I remove pm2 probe once Im done?

[Unitech]

great! Thanks for your help.
Just do:

$ pm2 uninstall pm2-probe

[Unitech]

Inspection procedure

• Are you using watch? Make sure you add the 'node_modules' in ignore_watch
• Node.js, PM2 versions? Linux distribution?
• How many applications have you started?

---

Deep inspection

In case this is not sufficient to find the CPU issue:

$ npm install Unitech/pm2#dev-v8 -g
$ pm2 update

This special dev-v8 branch has v8-profiler embedded. To start to CPU profile PM2 do:

$ pm2 profile start
# Wait some time, open HTOP look at usage resources
$ pm2 profile stop

A cpu profiling file will be generated.
Once you have this file send it to us or inspect it with the chrome developer tool to see where the CPU is used.

[Tomino2112]

@Unitech Hi, on our ec2 server everything is still ok, but on my local dev machine pm2 started acting up again. Keymetrics bucket: 5797814d751e5326eb0f0099

[Tomino2112]

Keymetrics:

[dmr07]

Having high CPU Usage in single instance Fork mode. Recently we were using a 2 instance cluster, which yielded lower CPU usage (0%-5%), but it maxed out memory (1.6G) every time it ran Phantom and the server would just stall. Fork mode also performed better on stress test than cluster.

System Information

Node: 4.2.1
PM2: 2.0.1
Ubuntu: 14.0.4

PM2 Configuration

{
    "name"                : "production"
  , "script"              : "app.js"
  , "node_args"           : ["--expose-gc"]
  , "instances"           : 1 
  , "exec_mode"           : "fork"
  , "watch"               : false // will run infinite loop due to updating `pid/`
  , "merge_logs"          : true
  , "cwd"                 : "/var/www/somefile/"
  , "env"                 : {
      "NODE_ENV"          : "production"
    }
}

KeyMetrics:

[soyuka]

Gosh, this should not happen ^^.
Is the in-memory pm2 version correct? pm2 list does show that nothing is watched? You can also check this in ~/.pm2/pm2.log. Btw, please give a sample from this log file, maybe something is wrong.

What does "will run infinite loop due to updating pid/" mean?

Can you try without gc?

/edit, oh those are stresstests? I felt bad about the cpu usage but nvm.

but it maxed out memory (1.6G) every time it ran Phantom and the server would just stall.

Is phantomjs taking that much? How did you ran it?

[Shadowman4205]

Hello,
i am running docker container with pm2
i am using keymetrics/pm2-docker-alpine:7 (not the latest one cause of this issue )
I have got CPU usage in idle without using application without doing any request?
i am using cluster mode with 4 instanses,wathcing is disabled.

[smolinari]

It would probably be better to start your own issues.

Scott

[FelixIsaac]

Hey, I am having the same issues with the CPU but the memory seems fine. The WMI Provider Host CPU usage jumps is when I use PM2 when I use NodeJS "node index,js" the Provider Host is around 10% CPU Usage, What can I do to fix this?

[rlarner-quizlet]

I'm running into a similar issue where very infrequently pm2 processes start eating up tons of memory on a few of our CentOs servers. On "healthy" servers pm2 uses about 4.3 Mb - on the problematic servers it uses 5.5 Gb. We're running verison 4.2.1 of pm2 in fork mode and watch is disabled in the ecosystem config via:

    watch: false,

anything else to check?

[kumarrishav]

do you expect to see more cpu utilization in case of i=4 (compare to i=2) under the same load