My script is in an errored status and none of the log files shown by pm2 desc contain anything (they don't exist). How can I debug what caused the 15 restarts in a row?
02:16:28 [0s] stockbase:~/stockbase$ pm2 desc 4
Describing process with pid 4 - name stockbase-frontend-prod
┌───────────────────┬────────────────────────────────────────────────────────────────────┐
│ status │ errored │
│ name │ stockbase-frontend-prod │
│ id │ 4 │
│ path │ /home/stockbase/stockbase-prod/frontend/stockbase-frontend-prod.js │
│ exec cwd │ /home/stockbase/stockbase-prod │
│ error log path │ /home/stockbase/stockbase-prod/stockbase-frontend-prod.err │
│ out log path │ /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log │
│ pid path │ /home/stockbase/.pm2/pids/stockbase-frontend-prod.pid │
│ mode │ cluster_mode │
│ watch & reload │ ✘ │
│ interpreter │ node │
│ restarts │ 15 │
│ unstable restarts │ 0 │
│ uptime │ 0 │
│ created at │ 1970-01-01T00:00:00.000Z │
└───────────────────┴────────────────────────────────────────────────────────────────────┘
02:16:45 [0s] stockbase:~/stockbase$ cat /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log
cat: /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log: No such file or directory
02:17:03 [0s] stockbase:~/stockbase$ cat /home/stockbase/stockbase-prod/stockbase-frontend-prod.err
cat: /home/stockbase/stockbase-prod/stockbase-frontend-prod.err: No such file or directory
Also, ~/pm2/logs is empty.
dandv
All 18 comments
- Are you sure that pm2 can write in
/home/stockbase/stockbase-prod/and in/home/stockbase/.pm2/pids/? - Are you catching unhandled exceptions in your application?
Make sure that the cwd has the wanted impact on your application.
You could try pm2 start myapp.js --no-daemon to see if it gets started as intended.
soyuka
on 2 Jul 2014
It seems to be a permissions issue, though I'm trying to figure out exactly which one. The exact same deployment script works if I'm logged in as dandv, and launch it from /home/dandv/stockbase, but fails as above if I'm logged in as stockbase and launch it from /home/dandv/stockbase.
- Yes, the user
stockbasecan create files in both those directories, though there are no.pidfiles when I run the deployment script asstockbase(there's one when I run it asdandv, even after I stop the process). - Probably, but you never know.
pm2 start frontend/stockbase-frontend-prod.js --no-daemon shows
>> pm2 is already daemonized ! You should kill it before launching it in no-daemon mode
fs.js:432
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: ENOENT, no such file or directory '/home/stockbase/.pm2/pm2.pid'
The permissions of /home/stockbase/.pm2/ are 775.
I ran touch pm2.pid in that directory, then pm2 start frontend/stockbase-frontend-prod.js --no-daemon -f again. Got, among other things, PM2 dameon PID =
0 with /home/stockbase/.pm2/pm2.log file not found, and again, 15 restarts.
After touch /home/stockbase/.pm2/pm2.log and running pm2 start a 3rd time, I got process launched... PM2 dameon PID = [blank, no PID], and pm2 was hanging (no output). pm2 l in another shell showed again 15 restarts. Hit ^C.
/home/stockbase/.pm2/pm2.log and /home/stockbase/.pm2/pm2.pid are 0-byte.
Looks like pm2 doesn't quite support more than one user running commands? I've been running pm2 as dandv for months, created boot scripts, then yesterday we wanted to switch to a dedicated user (stockbase) to manage pm2.
dandv
on 2 Jul 2014
What user has started pm2 the first time?
For example
ps -ef | grep pm2
Could you try specifying logs (out and err) and pid paths through a json file?
soyuka
on 2 Jul 2014
That must be it - pm2 was first started by dandv and now trying to control it via stockbase results in the errors above. Is it a bug perhaps? Otherwise I could try using a JSON file for the logs and paths, but it seems like command line parameters should work as well. A similar issue is why I stopped using forever.
dandv
on 2 Jul 2014
Maybe linked to https://github.com/Unitech/pm2/pull/329#issuecomment-47815701 (multi-user suggestions) ?
The link leads to a 404 error.
soyuka
on 2 Jul 2014
Fixed the link, it was missing http://.
dandv
on 2 Jul 2014
Ok thanks, could you try changing the user that is running pm2 to see if it works?
You could also try using --run-as-user stockbase on a process but it should be executed from the dandv account (i.e the user that runs pm2):
dandv@x:~/ pm2 start stockbase --run-as-user stockbase
It should resolve such things but this feature is not really working according to #329.
soyuka
on 2 Jul 2014
We 'll try to fix those user-related things, closing feel free to reopen if it's not fixed.
soyuka
on 5 Jul 2014
Now I'm getting the following error:
Error: EPERM, Operation not permitted
at /usr/lib/node_modules/pm2/lib/ProcessContainer.js:177:15
at WriteStream.<anonymous> (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:128:9)
at WriteStream.emit (events.js:95:17)
at WriteStream.<anonymous> (fs.js:1667:10)
I started the pm2 daemon as the same user as the one that runs pm2 start. The script is relatively complicated; any idea what information I need to provide to debug this?
dandv
on 15 Jul 2014
What error is this? You only gave a piece of stack trace.
soyuka
on 15 Jul 2014
Sorry, updated the trace above.
dandv
on 15 Jul 2014
Did you use --run-as-user or --run-as-group? Please don't use these atm.
soyuka
on 16 Jul 2014
I was using --run-as-user as indicated earlier. In the meantime I solved the error, while still using --run-as-user, though I'm not sure exactly how due to numerous config changes. One possibility is that I restarted the pm2 daemon as the same user as the one in the .json file.
dandv
on 16 Jul 2014
Multi-user configuration is not fully functionnal atm, we are looking for a solution to improve this but it's not an easy task (https://github.com/Unitech/pm2/pull/329#issuecomment-47815701).
soyuka
on 16 Jul 2014
I ran into the same issue and @soyuka's comment about checking the first user who started pm2 helped me. It was my personal user account and not the specific account I had created for the app ( ghost blog )
So I stopped pm2, did su - ghost and started pm2 again by just running
pm2 start index.js --name ghost
And it worked. This might be the way to do it until --run-as-user is fixed
SuryaSankar
on 15 Aug 2014
This one just bothered me yesterday, I was trying to restart in cluster mode with an option of "-i" to start multiple apps for the same node-worker but I was unable to do it. Then I looked into the folder logs/ created inside "$userHome/.pm2/" by giving a command "ls -al $userHome/.pm2/logs/*" I found that there were files which were owned by root and when I deleted them and then restarted the Apps, they were just turning ON as it never happened before I meant to say the ISSUE of ERRORED. Hope this helps somebody.
aamiryu
on 1 Oct 2014
In my case the problem was trivial. I looked at $ tailf ~/.pm2/pm2.log to see:
vagrant@lol-slack-bot-local:~/.pm2$ tailf ~/.pm2/pm2.log
2016-10-19 14:42:53: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:53: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:54: App [lol-slack-bot-TEST] with id [0] and pid [14834], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:54: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:54: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:55: App [lol-slack-bot-TEST] with id [0] and pid [14860], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:55: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:55: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:56: App [lol-slack-bot-TEST] with id [0] and pid [14886], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:56: Script /mnt/host/mount_dir/test.js had too many unstable restarts (16). Stopped. "errored"
The part exited with code [0] via signal [SIGINT] mans that execution of my script has finished and therefore it needs to be restarted.
I did not have any server/while loops inside my script - so it run from top to bottom and was exiting as soon as it run the last line of code. As expected... This was causing the 15 restarts.
_I had logs - so it is not exactly the same issue - but it was the first result I found when looking for a solution - so I'm leaving it here in case anyone else runs into the same issue._
jkulak
on 19 Oct 2016
I started to have same problems with pm2 2.3.0, when trying to run cluster app with -i 2 option. The app logs didn't show anything, because the problem probably was related to pm2 itself:
$ pm2 logs --lines 200
[TAILING] Tailing last 200 lines for [all] processes (change the value with --lines option)
/home/ubuntu/.pm2/pm2.log last 200 lines:
...
PM2 | path.js:1144
PM2 | cwd = process.cwd();
PM2 | ^
PM2 |
PM2 | Error: ENOENT: no such file or directory, uv_cwd
PM2 | at Error (native)
PM2 | at Object.resolve (path.js:1144:25)
PM2 | at Function.Module._resolveLookupPaths (module.js:361:17)
PM2 | at Function.Module._resolveFilename (module.js:431:31)
PM2 | at Function.Module._load (module.js:388:25)
PM2 | at Module.require (module.js:468:17)
PM2 | at require (internal/module.js:20:19)
PM2 | at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:12:15)
PM2 | at Module._compile (module.js:541:32)
PM2 | at Object.Module._extensions..js (module.js:550:10)
PM2 | 2017-02-20 14:16:08: App name:app id:0 disconnected
PM2 | 2017-02-20 14:16:08: App [app] with id [0] and pid [28520], exited with code [1] via signal [SIGINT]
PM2 | 2017-02-20 14:16:08: Script /.../app.js had too many unstable restarts (16). Stopped. "errored"
PM2 | 2017-02-20 14:16:08: App name:app id:1 disconnected
PM2 | 2017-02-20 14:16:08: App [app] with id [1] and pid [28538], exited with code [1] via signal [SIGINT]
PM2 | 2017-02-20 14:16:08: Script /.../app.js had too many unstable restarts (16). Stopped. "errored"
Running it in non-cluster mode worked. Also worked in cluster mode, but with --no-daemon.
However after I upgraded pm2 to 2.4.0 it started to work for me.
ktrzeciaknubisa
on 20 Feb 2017
Related issues
alexpts
·
3Comments
liujb
·
3Comments
lefam
·
3Comments
chaos-git
·
3Comments
webchaz
·
3Comments
Most helpful comment
/home/stockbase/stockbase-prod/and in/home/stockbase/.pm2/pids/?Make sure that the
cwdhas the wanted impact on your application.You could try
pm2 start myapp.js --no-daemonto see if it gets started as intended.