Pm2: spawn EACCES when upgrading pm2

Created on 5 Sep 2017 · 14Comments · Source: Unitech/pm2

On our servers, pm2 is installed globally under root, but our processes are running under user "node".
Thus following the steps mentioned here: http://pm2.keymetrics.io/docs/usage/update-pm2/

node> pm2 save
root> npm install pm2 -g
node> pm2 update

pm2 update
Be sure to have the latest version by doing `npm install pm2@latest -g` before doing this procedure.
[PM2] Saving current process list...

>>>> In-memory PM2 is out-of-date, do:
>>>> $ pm2 update
In memory PM2 version: 2.0.16
Local PM2 version: 2.6.1

[PM2] Stopping PM2...
[PM2] Applying action deleteProcessId on app [all](ids: 0,1,2,3,4,5,7)
[PM2] [PORT80APP](0) ✓
[PM2] [WEBHOOK](5) ✓
[PM2] [TLDINFO](7) ✓
[PM2] [CP3DEV](3) ✓
[PM2] [CP3DEV](1) ✓
[PM2] [CP3DEV](2) ✓
[PM2] [CP3DEV](4) ✓
[PM2] All processes have been stopped and deleted
[PM2] PM2 stopped
[PM2] Spawning PM2 daemon with pm2_home=/home/node/.pm2
internal/child_process.js:313
    throw errnoException(err, 'spawn');
    ^

Error: spawn EACCES
    at exports._errnoException (util.js:1022:11)
    at ChildProcess.spawn (internal/child_process.js:313:11)
    at Object.exports.spawn (child_process.js:380:9)
    at Client.launchDaemon (/usr/lib/node_modules/pm2/lib/Client.js:234:40)
    at /usr/lib/node_modules/pm2/lib/API.js:391:19
    at /usr/lib/node_modules/pm2/lib/API.js:636:25
    at /usr/lib/node_modules/pm2/lib/Client.js:512:19
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:421:16
    at iteratorCallback (/usr/lib/node_modules/pm2/node_modules/async/dist/async.js:996:13)
    at /usr/lib/node_modules/pm2/node_modules/async/dist/async.js:906:16
node@fcp3-dev1:/root$ pm2 list
[PM2] Spawning PM2 daemon with pm2_home=/home/node/.pm2
internal/child_process.js:313
    throw errnoException(err, 'spawn');
    ^

Error: spawn EACCES
    at exports._errnoException (util.js:1022:11)
    at ChildProcess.spawn (internal/child_process.js:313:11)
    at Object.exports.spawn (child_process.js:380:9)
    at Client.launchDaemon (/usr/lib/node_modules/pm2/lib/Client.js:234:40)
    at /usr/lib/node_modules/pm2/lib/Client.js:102:10
    at /usr/lib/node_modules/pm2/lib/Client.js:293:14
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Updating the in-memory version kills all processes, and we manually then need to restart all of them to have the app working again. Any idea on how to solve this?

We upgraded from 2.0.16 to 2.6.1.

===============================================================================
--- PM2 REPORT (Tue Sep 05 2017 04:39:53 GMT-0400 (EDT)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.6.1
node version         : 6.10.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/nodejs,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
uptime               : 40min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.6.1
node version         : 6.10.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/nodejs,/usr/bin/pm2,report
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz
cpus nb              : 4
freemem              : 1320284160
totalmem             : 4294967296
home                 : /home/node
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────┬────────────┬──────┬──────────┐
│ App name  │ id │ mode    │ pid   │ status │ restart │ uptime │ cpu │ mem        │ user │ watching │
├───────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────┼────────────┼──────┼──────────┤
│ CP3DEV    │ 1  │ cluster │ 26085 │ online │ 1       │ 39m    │ 0%  │ 313.1 MB   │ node │ disabled │
│ CP3DEV    │ 2  │ cluster │ 26088 │ online │ 1       │ 39m    │ 0%  │ 336.5 MB   │ node │ disabled │
│ CP3DEV    │ 3  │ cluster │ 26179 │ online │ 1       │ 38m    │ 0%  │ 337.9 MB   │ node │ disabled │
│ CP3DEV    │ 4  │ cluster │ 26186 │ online │ 1       │ 38m    │ 0%  │ 336.5 MB   │ node │ disabled │
│ PORT80APP │ 0  │ fork    │ 26141 │ online │ 1       │ 39m    │ 0%  │ 23.4 MB    │ node │ disabled │
│ TLDINFO   │ 6  │ cluster │ 25932 │ online │ 0       │ 39m    │ 0%  │ 45.3 MB    │ node │ disabled │
│ WEBHOOK   │ 5  │ fork    │ 25802 │ online │ 0       │ 40m    │ 0%  │ 23.5 MB    │ node │ disabled │
└───────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/node/.pm2/pm2.log last 20 lines:
PM2        | 2017-09-05 04:00:54: App [CP3DEV] with id [_old_1] and pid [25641], exited with code [0] via signal [SIGINT]
PM2        | 2017-09-05 04:00:54: Stopping app:CP3DEV id:_old_2
PM2        | 2017-09-05 04:00:54: App name:CP3DEV id:_old_2 disconnected
PM2        | 2017-09-05 04:00:54: App [CP3DEV] with id [_old_2] and pid [25668], exited with code [0] via signal [SIGINT]
PM2        | 2017-09-05 04:00:54: pid=25641 msg=process killed
PM2        | 2017-09-05 04:00:54: Starting execution sequence in -cluster mode- for app name:CP3DEV id:3
PM2        | 2017-09-05 04:00:54: pid=25668 msg=process killed
PM2        | 2017-09-05 04:00:54: App name:CP3DEV id:3 online
PM2        | 2017-09-05 04:00:54: Starting execution sequence in -cluster mode- for app name:CP3DEV id:4
PM2        | 2017-09-05 04:00:54: App name:CP3DEV id:4 online
PM2        | 2017-09-05 04:00:56: -softReload- New worker listening
PM2        | 2017-09-05 04:00:56: -softReload- New worker listening
PM2        | 2017-09-05 04:01:04: Stopping app:CP3DEV id:_old_3
PM2        | 2017-09-05 04:01:04: App name:CP3DEV id:_old_3 disconnected
PM2        | 2017-09-05 04:01:04: App [CP3DEV] with id [_old_3] and pid [25694], exited with code [0] via signal [SIGINT]
PM2        | 2017-09-05 04:01:04: Stopping app:CP3DEV id:_old_4
PM2        | 2017-09-05 04:01:04: App name:CP3DEV id:_old_4 disconnected
PM2        | 2017-09-05 04:01:04: App [CP3DEV] with id [_old_4] and pid [25712], exited with code [0] via signal [SIGINT]
PM2        | 2017-09-05 04:01:04: pid=25694 msg=process killed
PM2        | 2017-09-05 04:01:04: pid=25712 msg=process killed

node@fcp3-dev1:~$ ls -altrd ~/.pm2
drwxr-xr-x 5 node node 4096 Sep  5 03:59 /home/node/.pm2

Source

papakai

Most helpful comment

So indeed this can only be avoided by taking a machine out of load balancing, then upgrading and activating it again. Thanks again dude.

papakai on 8 Sep 2017

👍2

All 14 comments

Tried this again on another host.
Recognized a 2nd deamon to be running under root. Killed that one and then tried to follow the upgrade way.

node@fcp3-dev2:/root$ pm2 save
[PM2] Spawning PM2 daemon with pm2_home=/home/node/.pm2
internal/child_process.js:298
    throw errnoException(err, 'spawn');
    ^

Error: spawn EACCES
    at exports._errnoException (util.js:907:11)
    at ChildProcess.spawn (internal/child_process.js:298:11)
    at Object.exports.spawn (child_process.js:362:9)
    at Client.launchDaemon (/usr/lib/node_modules/pm2/lib/Client.js:214:40)
    at /usr/lib/node_modules/pm2/lib/Client.js:101:10
    at /usr/lib/node_modules/pm2/lib/Client.js:273:14
    at nextTickCallbackWith0Args (node.js:420:9)
    at process._tickCallback (node.js:349:13)
node@fcp3-dev2:/root$ pm2 list
[PM2] Spawning PM2 daemon with pm2_home=/home/node/.pm2
internal/child_process.js:298
    throw errnoException(err, 'spawn');
    ^

Error: spawn EACCES
    at exports._errnoException (util.js:907:11)
    at ChildProcess.spawn (internal/child_process.js:298:11)
    at Object.exports.spawn (child_process.js:362:9)
    at Client.launchDaemon (/usr/lib/node_modules/pm2/lib/Client.js:214:40)
    at /usr/lib/node_modules/pm2/lib/Client.js:101:10
    at /usr/lib/node_modules/pm2/lib/Client.js:273:14
    at nextTickCallbackWith0Args (node.js:420:9)
    at process._tickCallback (node.js:349:13)

So this already breaks on pm2 save...

NOTE: the root daemon is not getting triggered with reboot, tested this especially. just in case someone does pm2 list under root, it is getting instantiated.

papakai on 5 Sep 2017

I did a fresh re-setup
root> pm2 startup systemd -u node -hp /home/node and did a reboot - looked good so far.
but still the above issues are reproducable :(

papakai on 5 Sep 2017

PM2 need to have the access over pm2 files binary because it spawn the daemon itself, we advise to install PM2 under the user you want to use, in your case node

vmarchaud on 5 Sep 2017

😕1

PM2 need to have the access over pm2 files binary because it spawn the daemon itself, we advise to install PM2 under the user you want to use, in your case node

@vmarchaud Honestly under user node the step npm install pm2@latest -g wouldn't work as only root user would have the required access rights. What is the best way then?
npm install pm2@latest - so not globally and then adding an alias to .bashrc ?

I hardly suggest this part of the pm2 documentation needs a review especially for the non-root user case.

/UPDATE: I gave the non-global installation a try, but this also doesn't work.

Error: EACCES: permission denied, scandir '/root'
    at Error (native)

Error: EACCES: permission denied, open 'npm-debug.log.1892515987'
    at Error (native)

Any requirements which I am missing?

papakai on 5 Sep 2017

I expect Unitech is suggesting to use nvm for that purpose? or do you have better suggestions?
I found this: https://docs.npmjs.com/getting-started/fixing-npm-permissions
Any preferences?

papakai on 6 Sep 2017

We are using nvm to manage nodejs version which is always under non-privileged user, much easier to maintain, why do you want to install node/npm as root exactly ?

vmarchaud on 6 Sep 2017

Our sys-admin told me one day that using a package manager on top, isn't an option for him. honestly I cannot provide the detailed reasons - but at least I think one of his arguments was that mixing up the os package manager with another package manager on top, might lead to side-effects. I might need to discuss this again. We just use nvm on local development area for now.
I just read that when switching the node version with nvm, it is necessary to care about reinstalling the global packages, that might be one thing to keep in mind then.

papakai on 6 Sep 2017

Indeed, thats the only problem with nvm that we have.
I think there are two options here :

Allow node user to read/exec where PM2 is installed
Run PM2 as root (just run pm2 ls and it will spawn the daemon as root), then add export PM2_HOME=/root/.pm2 on the .bashrc of node and finally allow node to access the daemon RPC server inside /root/pm2.

The later is problematic because all of your processes will run under root by default, you'll need to use --uid and --guid flags of PM2 (not documented right now)

EDIT : changing title so googling the error while make people read this

vmarchaud on 6 Sep 2017

Indeed, thats the only problem with nvm that we have.

May I ask how you solved this on your side?

papakai on 6 Sep 2017

We didn't solve it since we rarely upgrade nodejs version on bare metal, plus we are moving to container with pm2-docker so the problem will not exist anymore

vmarchaud on 6 Sep 2017

Thanks for the insights, very appreciated.

papakai on 6 Sep 2017

@vmarchaud during pm2 update, i see the following output: 'All processes have been stopped and deleted'. Does that mean that updating pm2 always leads to downtime? Isn't it technically possible to solve it without downtime?

papakai on 8 Sep 2017

@papakai Nope, since you are restarting pm2 daemon you cannot avoid downtime

vmarchaud on 8 Sep 2017

So indeed this can only be avoided by taking a machine out of load balancing, then upgrading and activating it again. Thanks again dude.

papakai on 8 Sep 2017

👍2

Was this page helpful?

0 / 5 - 0 ratings