Pm2: Poor pm2 watch performance on APFS when including node_modules

[fluxsauce]

Our team has been scratching their heads on this one. Happy to provide any additional technical info and facilitate any suggested troubleshooting.

We've been using pm2 in production for over three years and have been transitioning from nodemon to pm2 locally over the past year or so.

We all have MacBook Pro laptops with fairly similar specs, each of them has a fresh install (not upgrade) of macOS Mojave.

Some of them experience massive performance problems when in watch mode and some do not. Upon investigation, the problem manifests itself in incredibly high CPU utilization (150%-250%) and Activity Monitor > PM2 vx.x.x: God Daemon > Statistics > Unix System Calls continually increments, sometimes 100,000 a second.

Some key points:

• The performance problems did not "appear" overnight; workstations either experience them or they don't.
• We share a common pm2 local configuration and have unified our local workstation setup process to keep things consistent
• We are all running the same version of Node.js installed locally via n
• We tried downgrading PM2 to 2.7, no difference
• We tried reformatting GAMMA's machine from APFS to APFS (Case-sensitive), no difference
• We are not running any active file system or virus scanners
• The problem is not present in workstations running macOS 10.13, nor is it present on a system running Ubuntu.
• The problem is present when nothing else is accessing the files (no editors open for example)

Here's a table for comparisons. "Services" refers to services required for our stack, such as redis, elasticsearch, and so forth.

|User|PM2 Watch Problem|Model|Processor|Memory|HD|OS Version|FileVault|pm2d|node|File System|Services|
|---|---|---|---|---|---|---|---|---|---|---|---|
|ALPHA|N|Retina, 13-inch, Early 2015|2.7 GHz Intel Core i5|16 GB 1867 MHz DDR3|APPLE SSD SM0256G|macOS 10.14.3|N|3.3.1|8.15.0|APFS (Case-sensitive)|Docker|
|BETA|N|Retina, 13-inch, Early 2015|2.7 GHz Intel Core i5|16 GB 1867 MHz DDR3|APPLE SSD SM0256G|macOS 10.14.3|Y|3.3.1|8.15.0|APFS (Case-sensitive)|Homebrew|
|GAMMA|Y|Retina, 13-inch, Mid 2014|3.7 GHz Intel Core i5|16 GB 1867 MHz DDR3|APPLE SSD SM0256F|macOS 10.14.2|N|3.3.1|8.15.0|APFS (Case-sensitive)|Docker|
|DELTA|Y|Retina, 13-inch, Early 2015|3.1 GHz Intel Core i7|16 GB 1867 MHz DDR3|APPLE SSD SM0256G|macOS 10.14.3|N|3.2.3|8.15.0|APFS|Docker|
|EPSILON|Y|Retina, 13-inch, Early 2015|3.1 GHz Intel Core i7|16 GB 1867 MHz DDR3|APPLE SSD SM1024G|macOS 10.14.2|N|3.3.1|8.15.0|APFS|Docker|
|ZETA|N|Linux|AMD Ryzen 5 2400G|16 GB 2666 MHz DDR4|Samsung SSD 860 250GB|Ubuntu 18.10|N/A|3.3.1|8.15.0|ext4|Docker

We generate the PM2 configuration startup script so they're really consistent. Here's an example:

{  
   "apps":[  
      {  
         "name":"REDACTED",
         "cwd":"/home/jon/projects/NOPE/SERVICENAME",
         "script":"./app.js",
         "watch":true,
         "ignore_watch":[  
            "[\\/\\\\]\\.",
            ".nyc_output",
            "coverage",
            ".git",
            ".idea",
            "node_modules/.cache",
            "node_modules/.staging",
            "build"
         ],
         "env":{  
            "NODE_ENV":"development"
         },
         "exec_mode":"fork"
      }
   ]
}

pm2 report with no problem

--- PM2 report ----------------------------------------------------------------
Date                 : Tue Mar 05 2019 09:21:43 GMT-0800 (PST)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.3.1
node version         : 8.15.0
node path            : /Users/ALPHA/n/bin/pm2
argv                 : /Users/ALPHA/n/bin/node,/Users/ALPHA/n/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : ALPHA
uid                  : 502
gid                  : 20
uptime               : 31min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.3.1
node version         : 8.15.0
node path            : /Users/ALPHA/n/bin/pm2
argv                 : /Users/ALPHA/n/bin/node,/Users/ALPHA/n/bin/pm2,report
argv0                : node
user                 : ALPHA
uid                  : 502
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
cpus nb              : 4
freemem              : 3177799680
totalmem             : 17179869184
home                 : /Users/ALPHA
===============================================================================

pm2 report with problem

--- PM2 report ----------------------------------------------------------------
Date                 : Sat Mar 02 2019 16:53:58 GMT-0800 (PST)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.3.1
node version         : 8.15.0
node path            : /Users/EPSILON/n/bin/pm2
argv                 : /Users/EPSILON/n/bin/node,/Users/EPSILON/n/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : EPSILON
uid                  : 501
gid                  : 20
uptime               : 1min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.3.1
node version         : 8.15.0
node path            : /Users/EPSILON/n/bin/pm2
argv                 : /Users/EPSILON/n/bin/node,/Users/EPSILON/n/bin/pm2,report
argv0                : node
user                 : EPSILON
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
cpus nb              : 4
freemem              : 1454260224
totalmem             : 17179869184
home                 : /Users/EPSILON

This is the first time in years we've ever had a problem with pm2, and it's a weird one. Thanks for everything so far, and in advance!

[fluxsauce]

Hi there, are there any troubleshooting steps that you'd recommend? Thanks!

[fluxsauce]

@Unitech I've been troubleshooting. Some insight would really be appreciated.

If I specify watch_options usePolling: false, performance is good until I use pm2 kill, at which point CPU utilization spikes and it never exits.

I took one of the macOS 10.14 machines and reformatted it with APFS in 10.13 and the performance problem still happens.

[Unitech]

Hey @fluxsauce

Thanks for this detailed report but I did not managed to reproduce this issue on a macOS 10.13.

For information, the watch & restart system is handled by chokidar module
You could tweak the watch_options:{} in ecosystem file, this option is directly passed to chokidar instance. For example you could reduce the depthoption to limit number of sub directories to watch.

Else if option tweaking does not solve this issue, you can cross post this issue on chokidar repository

[freshlogic]

@fluxsauce thanks for this report. I'm experiencing the same issue. When I install PM2 I'm running into this issue and wondering if it's somehow related: https://github.com/paulmillr/chokidar/issues/828

[freshlogic]

Seems like I was able to solve my issue by uninstalling pm2 and then installing again with sudo npm i -g --unsafe-perm pm2

[fluxsauce]

@freshlogic the unsafe-perm workaround did not work for me. However, I was able to unblock our team by ignoring node_modules. That's suboptimal, but we have to move on.

This is what we're using:

            name: application.name,
            cwd: path.resolve(__dirname, application.repository),
            script: './app.js',
            watch: true,
            watch_options: { // eslint-disable-line camelcase
                usePolling: false,
            },
            ignore_watch: [ // eslint-disable-line camelcase
                './.nyc_output/*',
                './coverage/*',
                './.git/*',
                './.idea/*',
                './node_modules/*',
                './build/*', // for PROJECT Vue build directory
            ],

I was able to reduce, but not completely eliminate some of the performance problems by manually updating chokidar within pm2 and updating fsevents within chokidar.

[Unitech]

Chokidar 3.0 will land soon, fixing many performance issues. Also PM2 4.0 will be released in one month. Stay tuned.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.