Pm2: Systemctl error: PID file not readable (yet)
The issue
When running startup for unprivileged user, the service fails to start.
Context
System info:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
Linux myuser-demo-1 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
The command run:
pm2 startup -u myuser --hp /home/myuser
Output:
[PM2] Init System found: systemd
Platform ubuntu
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target
[Service]
Type=forking
User=myuser
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/home/myuser/.pm2
PIDFile=/home/myuser/.pm2/pm2.pid
ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill
[Install]
WantedBy=multi-user.target
Target path
/etc/systemd/system/pm2-myuser.service
Command list
[ 'chmod +x /etc/systemd/system/pm2-myuser.service',
'systemctl enable pm2-myuser',
'systemctl start pm2-myuser',
'systemctl daemon-reload',
'systemctl status pm2-myuser' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-myuser.service
[PM2] Making script booting at startup...
>>> Executing chmod +x /etc/systemd/system/pm2-myuser.service
[DONE]
>>> Executing systemctl enable pm2-myuser
[DONE]
>>> Executing systemctl start pm2-myuser
Job for pm2-myuser.service failed because a configured resource limit was exceeded. See "systemctl status pm2-myuser.service" and "journalctl -xe" for details.
[ERROR] Exit code : 1
[PM2][ERROR] systemctl start pm2-myuser failed, see error above.
The error when service tries to start:
● pm2-myuser.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-myuser.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Wed 2017-05-31 10:02:17 UTC; 1min 4s ago
Docs: https://pm2.keymetrics.io/
Process: 1605 ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect (code=exited, status=0/SUCCESS)
May 31 10:02:17 myuser-demo-1 pm2[1605]: Daemonize Application
May 31 10:02:17 myuser-demo-1 pm2[1605]: $ pm2 start <app>
May 31 10:02:17 myuser-demo-1 pm2[1605]: Monitoring/APM solution
May 31 10:02:17 myuser-demo-1 pm2[1605]: https://app.keymetrics.io/
May 31 10:02:17 myuser-demo-1 pm2[1605]: -------------
May 31 10:02:17 myuser-demo-1 pm2[1605]: [PM2] Spawning PM2 daemon with pm2_home=/home/myuser/.pm2
May 31 10:02:17 myuser-demo-1 systemd[1]: pm2-myuser.service: PID file /home/myuser/.pm2/pm2.pid not readable (yet?) after start: No suc
May 31 10:02:17 myuser-demo-1 systemd[1]: Failed to start PM2 process manager.
May 31 10:02:17 myuser-demo-1 systemd[1]: pm2-myuser.service: Unit entered failed state.
May 31 10:02:17 myuser-demo-1 systemd[1]: pm2-myuser.service: Failed with result 'resources'.
How I solved it
I found no solution chmoding whatever I could. I found it could be a bug with systemctl, and found no reliable solution to fix this permission issue… So I changed /etc/systemd/system/pm2-myuser.service to change how it's started:
[Service]
#Type=forking
Type=simple
…
#PIDFile=/run/pm2-ooz.pid
So it's now a foreground process, where no pidfile is required and everything works fine again.
My issue now is: will it work reliably in the future? Has anyone experienced this? If everything is OK, how about modifying your template so it doesn't happen anymore?
naholyr
All 41 comments
Well, anyway, once startup script is installed, whatever the way I start scripts they run as root. I'll just go with process.setuid in my server.js, it will be more reliable…
Suddenly "pm2 start" failed working for "myuser", now nothing seems to work properly, I give up.
naholyr
on 31 May 2017
+1, pm2 version 2.4.6,
can't find the file home/pi/.pm2/pm2.pid
pi@pi:/usr/local/lib/node_modules/pm2/lib/templates $ sudo systemctl status pm2-pi.service
● pm2-pi.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-pi.service; enabled)
Active: failed (Result: resources) since Wed 2017-05-31 18:27:35 CST; 9s ago
Docs: https://pm2.keymetrics.io/
Process: 2648 ExecStart=/usr/local/lib/node_modules/pm2/bin/pm2 resurrect (code=exited, status=0/SUCCESS)
May 31 18:27:34 pi pm2[2648]: [PM2] Spawning PM2 daemon with pm2_home=/home/pi/.pm2
May 31 18:27:35 pi systemd[1]: PID file /home/pi/.pm2/pm2.pid not readable (yet?) after start.
May 31 18:27:35 pi systemd[1]: Failed to start PM2 process manager.
May 31 18:27:35 pi systemd[1]: Unit pm2-pi.service entered failed state.
ashqal
on 31 May 2017
I’m running into the same issues as stated above.
Context
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
Node.js: v8.0.0
NPM: v5.0.1
PM2: 2.4.6
When installing PM2 with NPM (npm install -g [email protected]), everything looks fine.
Afterwards, I create my dedicated application user that will run the app with PM2.
When executing
pm2 startup systemd
everything is fine as well. The problems start when running
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u my-user --hp /home/my-user
So I checked the pm2.log file within the /home/my-user/.pm2/ directory and the contents are like five times this entry:
Error: EACCES: permission denied, open '/usr/lib/node_modules/pm2/node_modules/array-unique/index.js'
at Object.fs.openSync (fs.js:638:18)
at Object.fs.readFileSync (fs.js:540:33)
at Object.Module._extensions..js (module.js:579:20)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/micromatch/lib/utils.js:13:16)
at Module._compile (module.js:569:30)
fs.js:638
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Maybe this information helps to find the issue.
marcuspoehls
on 2 Jun 2017
+1
CentOS-72-64-minimal
[email protected] (everything was fine before the last update)
Target path
/etc/systemd/system/pm2-root.service
Command list
[ 'chmod +x /etc/systemd/system/pm2-root.service',
'systemctl enable pm2-root',
'systemctl start pm2-root',
'systemctl daemon-reload',
'systemctl status pm2-root' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] Making script booting at startup...
>>> Executing chmod +x /etc/systemd/system/pm2-root.service
[DONE]
>>> Executing systemctl enable pm2-root
[DONE]
>>> Executing systemctl start pm2-root
Job for pm2-root.service failed because a configured resource limit was exceeded. See "systemctl status pm2-root.service" and "journalctl -xe" for details.
[ERROR] Exit code : 1
[PM2][ERROR] systemctl start pm2-root failed, see error above.
pm2-root.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-root.service; enabled; vendor preset: disabled)
Active: failed (Result: resources) since Fri 2017-06-02 13:07:29 CEST; 3min 38s ago
Docs: https://pm2.keymetrics.io/
Process: 9373 ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect (code=exited, status=0/SUCCESS)
systemctl status pm2-root.service
PID file /root/.pm2/pm2.pid not readable (yet?) after start.
Failed to start PM2 process manager.
Unit pm2-root.service entered failed state.
pm2-root.service failed.
mpolinowski
on 2 Jun 2017
The last update was a month ago, i think the problem here is node 8 ?
vmarchaud
on 2 Jun 2017
I just updated to 2.4.6 - then the problem came up.
I downgraded to 2.4.5 just now - now everything is back to normal
And I am still running node 7.9
+++ edit: deleted error log.... that was something else +++
mpolinowski
on 2 Jun 2017
PM2 2.4.6 commit here
PM2 2.4.5 commit here
There are strictly no change so it's clearly not coming from the latest version
vmarchaud
on 2 Jun 2017
I am not sure what version I was running before - was there a big change earlier this year? I did not update for a while.
mpolinowski
on 2 Jun 2017
I think we changed the way pm2 startup works by registering a unit file depending of the user who launched the command but no changes since december otherwise
vmarchaud
on 2 Jun 2017
ok thank you.
"pm2 startup" worked again after I downgraded to 2.4.5 and the apps seem to be running now.
++++ Update +++
Tested it over night / all apps restart after sever reboot
I cannot reproduce the error anymore after the downgrade to 2.4.5
mpolinowski
on 2 Jun 2017
@mpolinowski What version of Node are you running?
marcuspoehls
on 3 Jun 2017
@marcuspoehls node v7.9.0
mpolinowski
on 3 Jun 2017
@mpolinowski Alright, thank you. I need to do further checks and see if I can downgrade PM2 and use it with Node 8 and NPM 5. Will report back to this issue. Maybe someone else is able to find a solution :)
marcuspoehls
on 3 Jun 2017
@marcuspoehls would be interesting to know
mpolinowski
on 4 Jun 2017
Can everyone verify that PM2 have full access to all the files contained in ~/.pm2 ?
vmarchaud
on 4 Jun 2017
I updated to 2.4.6 again and the old error message is back:
[root@CentOS-72-64-minimal ~]# pm2 startup
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target
[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill
[Install]
WantedBy=multi-user.target
Target path
/etc/systemd/system/pm2-root.service
Command list
[ 'chmod +x /etc/systemd/system/pm2-root.service',
'systemctl enable pm2-root',
'systemctl start pm2-root',
'systemctl daemon-reload',
'systemctl status pm2-root' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] Making script booting at startup...
>>> Executing chmod +x /etc/systemd/system/pm2-root.service
[DONE]
>>> Executing systemctl enable pm2-root
[DONE]
>>> Executing systemctl start pm2-root
Job for pm2-root.service failed because a configured resource limit was exceeded. See "systemctl status pm2-root.service" and "journalctl -xe" for details.
[ERROR] Exit code : 1
[PM2][ERROR] systemctl start pm2-root failed, see error above.
I set the permissions for .pm2 to rwxrwxrwx recursively - the error stays the same.
Though, the old startup script I set with pm2 v2.4.5 is still active - I was able to pm2 resurrect all processes. So basically, everything is working, but I cannot update the script anymore in v2.4.6.
mpolinowski
on 4 Jun 2017
@marcuspoehls I think your issue come from https://github.com/npm/npm/issues/16896, not the same issue as OP
vmarchaud
on 7 Jun 2017
@vmarchaud yeah, looks like I’m running into a permission issue. Will do some further checks by the end of this week. Thanks for pointing me to this NPM issue :)
marcuspoehls
on 7 Jun 2017
I'm experiencing the same problem:
● pm2-node.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-node.service; enabled; vendor preset: enabled)
Active: failed (Result: resources) since Thu 2017-06-22 03:12:49 UTC; 4s ago
Docs: https://pm2.keymetrics.io/
Process: 15182 ExecStart=/usr/local/lib/node/lib/node_modules/pm2/bin/pm2 resurrect (code=exited, status=0/SUCCESS)
Jun 22 03:12:48 xxx pm2[15182]: Start PM2 at boot
Jun 22 03:12:48 xxx pm2[15182]: $ pm2 startup
Jun 22 03:12:48 xxx pm2[15182]: Increase Performance
Jun 22 03:12:48 xxx pm2[15182]: $ pm2 start api.js -i 4
Jun 22 03:12:48 xxx pm2[15182]: -------------
Jun 22 03:12:48 xxx pm2[15182]: [PM2] Spawning PM2 daemon with pm2_home=/home/node/.pm2
Jun 22 03:12:49 xxx systemd[1]: pm2-node.service: PID file /home/node/.pm2/pm2.pid not readable (yet?) after start: No such file or directory
Jun 22 03:12:49 xxx systemd[1]: Failed to start PM2 process manager.
Jun 22 03:12:49 xxx systemd[1]: pm2-node.service: Unit entered failed state.
Jun 22 03:12:49 xxx systemd[1]: pm2-node.service: Failed with result 'resources'.
Here is ls -la of /home/node:
# ls -la /home/node
total 12
drwxr-xr-x 3 node node 4096 Jun 22 03:12 .
drwxr-xr-x 3 root root 4096 Jun 22 03:12 ..
drwxr-xr-x 5 node node 4096 Jun 22 03:12 .pm2
And /home/node/.pm2:
# ls -la /home/node/.pm2
total 32
drwxr-xr-x 5 node node 4096 Jun 22 03:12 .
drwxr-xr-x 3 node node 4096 Jun 22 03:12 ..
drwxr-xr-x 2 node node 4096 Jun 22 03:12 logs
-rw-r--r-- 1 node node 2 Jun 22 03:12 module_conf.json
drwxr-xr-x 2 node node 4096 Jun 22 03:12 node_modules
drwxr-xr-x 2 node node 4096 Jun 22 03:12 pids
-rw-r--r-- 1 node node 730 Jun 22 03:12 pm2.log
-rw-r--r-- 1 node node 13 Jun 22 03:12 touch
indutny
on 22 Jun 2017
Oh, apparently:
fs.js:651
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: EACCES: permission denied, open '/usr/local/lib/node/lib/node_modules/pm2/node_modules/array-unique/index.js'
at Object.fs.openSync (fs.js:651:18)
at Object.fs.readFileSync (fs.js:553:33)
at Object.Module._extensions..js (module.js:579:20)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/usr/local/lib/node/lib/node_modules/pm2/node_modules/micromatch/lib/utils.js:13:16)
at Module._compile (module.js:569:30)
Yeah, fixed permissions for /usr/local/lib/node and it works now!
indutny
on 22 Jun 2017
Looks like a permission issue more than a PM2 issue, just checkout that you user can use the ~/.pm2 directory and its dependencies.
vmarchaud
on 23 Jun 2017
Not working on Node 8.x (Debian)
Back to Node 7.10.0 and PM2 2.4.5, everything works again
brickpop
on 5 Jul 2017
I just tried to setup startup script on Ubuntu 16.04, with Node 8.1.3, npm 5.0.3 and pm2 2.5, I haven't got any issue.
@ledfusion how have you installed node.js? Via nvm?
What is the service and systemd version?
ubuntu@test-2912:~$ systemd --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN
ubuntu@test-2912:~$ service --version
service ver. 0.91-ubuntu1
Unitech
on 5 Jul 2017
Hi @Unitech,
I'm using the raw binary of NodeJS 8.1.3 on Debian Jessie (untar and copy through n, no NVM in my case)
As for Systemd and Service:
$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR
$ sudo service --version
service ver. 0.91-ubuntu1
The problem is the same as @marcuspoehls when trying to set the startup service
Also, if I try to run pm2 list from the unprivileged user, it will fail silently
I I try to run sudo su -c "pm2 list" my-user it will be verbose about afile permissions error (I am root¿?) and fail as well
I'd love to post the command's output here, but unfortunately I can't afford to switch back to the unconfigured state now that the server is un use :(
brickpop
on 10 Jul 2017
I also had this problem on a centos 7 system using node 8 (installed using yum), npm 5.2.0, and pm2 2.5.0 as linux user "pm2".
I was able to fix it by doing:
chown -R root:root /usr/lib/node_modules/
chmod -R ugo+r /usr/lib/node_modules/
PM2_HOME=/home/pm2/.pm2 pm2 startup -u pm2
bmiller59
on 12 Jul 2017
Only a feedback:
- I had the problem centos / node 7.9.0 when updating to pm2 v2.4.6
downgrading to pm2 v2.4.5 solved to problem
today I upgraded both node to 8.2.1 and pm2 to v2.6.1 and everything seems to be running fine.
mpolinowski
on 24 Jul 2017
I have this issue on Arch Linux with Node.js v9.6.1 and pm2 2.10.1. It is strange that it DOES created the pm2.pid file, but after I run the sudo command provided by pm2 startup, it reported the same error as OP and then the pid file disappeared (though the daemon is still running). There is a way to solve: after failing the sudo command, the systemd unit file is already created, so stop the daemon with pm2 kill, and then run sudo systemctl start pm2-myuser.service. No error reported, but not sure whether this solve the problem finally.
perqin
on 25 Feb 2018
@perqin I'm running into same thing on pm2 2.10.2 using node v9.7.1 also on Arch Linux. Your solve works for me.
dhensen
on 3 Apr 2018
@perqin Thank you
alexandre1985
on 13 Apr 2018
@perqin thank you very much, it worked for me
abumalick
on 28 Apr 2018
Fixed with the latest pm2 2.10.4! No more need to run pm2 kill
$ pm2 unstartup
$ npm install pm2@latest -g
$ pm2 startup
I'm constantly getting this error after boot:
[PM2][ERROR] Permission denied, to give access to current user:
$ sudo chown myuser:myuser /home/myuser/.pm2/rpc.sock /home/myuser/.pm2/pub.sock
Those files are owned by root, so I tried to run chown as suggested and rebooted, but the problem persists. Also tried commenting the line User=myuser in the unit, with no luck.
$ node -v
v8.12.0
$ pm2 -v
3.1.2
System is Ubuntu 16.04
rraallvv
on 5 Oct 2018
I made a fresh install on a ubuntu server and got exacly the same problem:
systemctl status pm2-myuser
● pm2-myuser.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-myuser.service; enabled; vendor preset: enabled)
Active: failed (Result: protocol) since Wed 2018-10-03 11:31:42 CEST; 2 days ago
Docs: https://pm2.keymetrics.io/
I tried to run pm2 unstartup and pm2 startup but it did not work.
after running the two commands given by @perqin it worked again
$ pm2 --version
3.1.3
$ node --version
v8.12.0
@abumalick do you mean the two commands below?
$ sudo pm2 kill
$ sudo systemctl start pm2-myuser.service
With those it seems to work, but when I reboot I get the same permission error again.
I'm trying to run pm2 startup systemd as root to install the service for the root user, but can't get the node app to run as root yet.
rraallvv
on 5 Oct 2018
Sorry I didn't try with root, with my user it works after reboot.
I think it is not advised to run node processes as root through, why don't you use a user ?
abumalick
on 5 Oct 2018
I'm using a user for the node app and the pm2 service, I'm just trying to see if running the app as root instead would get rid of the problem.
rraallvv
on 5 Oct 2018
did you run pm2 startup with sudo ?
abumalick
on 5 Oct 2018
No, when I run pm2 kill as myuser I get the error I posted:
[PM2][ERROR] Permission denied, to give access to current user:
$ sudo chown myuser:myuser /home/myuser/.pm2/rpc.sock /home/myuser/.pm2/pub.sock
That's why I run it with sudo now
rraallvv
on 5 Oct 2018
Finally I got rid of the issue downgrading to pm2 version 2.10.4
rraallvv
on 5 Oct 2018
Had the same issue. Upgrading to [email protected] solved it.
karlpokus
on 4 May 2020
Related issues
psparago
·
3Comments
rangercyh
·
4Comments
ghost
·
3Comments
waygee
·
4Comments
chaos-git
·
3Comments
Most helpful comment
I have this issue on Arch Linux with Node.js v9.6.1 and pm2 2.10.1. It is strange that it DOES created the pm2.pid file, but after I run the sudo command provided by
pm2 startup, it reported the same error as OP and then the pid file disappeared (though the daemon is still running). There is a way to solve: after failing the sudo command, the systemd unit file is already created, so stop the daemon withpm2 kill, and then runsudo systemctl start pm2-myuser.service. No error reported, but not sure whether this solve the problem finally.