Pm2: PM2 save doesn't seem to be working

[memphisraynz]

I am running centos7 and have just installed PM2.
Added a service "pm2 start app.js --name lets-chat" which starts the program and its working fine.
But after running pm2 startup centos and the pm2 save, upon reboot there is nothing list when I run pm2 list.

I'm not sure I am doing something wrong or if there is a bug.

Thanks

[Unitech]

Can you try to do pm2 resurrect to see if the process are spawned back?
Once the server is restarted, can you see the process PM2 in memory (via top/htop)?

[memphisraynz]

While pm2 is working and I run pm2 resurrect it spawns another process
After reboot pm2 is in top, right at the bottom ( 2061 root 20 0 661384 21796 5160 S 0.0 2.2 0:00.14 PM2 v0.12.6: Go)
Running pm2 list and pm2 resurrect after reboot brings up an emtpy table.

Thank you for your help

[soyuka]

Try this:

pm2 kill #start on a fresh basis
pm2 start myapp.js 
pm2 save #dump
pm2 kill #kill again to test
pm2 resurrect 
pm2 list #is the process myapp in the list? 

If it's not in the list we got an issue. Rebooting seems not to be the issue here.

[memphisraynz]

Doing the above steps results in having the app.js handled by pm2 website
is running no problems

On Wed Feb 25 2015 at 19:01:55 Antoine Bluchet [email protected]
wrote:

Try this:

pm2 kill #start on a fresh basis
pm2 start myapp.js
pm2 save #dump
pm2 kill #kill again to test
pm2 resurrect
pm2 list #is the process myapp in the list?

If it's not in the list we got an issue. Rebooting seems not to be the
issue here.

—
Reply to this email directly or view it on GitHub
https://github.com/Unitech/PM2/issues/1035#issuecomment-75926601.

[memphisraynz]

Any other ideas on why PM2 is having troubles resurrecting the processes after reboot?

Thanks

[webuniverseio]

Same issue on debian 7. When I manually use save and resurrect it is working just fine. When I'm rebooting (obviously I executed pm2 startup before that) I'm getting empty list.

[frankized]

same issue on beaglebone black debian.
As a temporary workaround, I set the dump file as immutable after "pm2 save":
$ chattr +i /path/to/my/dump.pm2

[EdHubbell]

This happens to me, but only when the server runs out of memory. I get an out of memory error in my pm2 logs like so (which I know is my fault, not pm2's):

PM2: 2015-04-20 21:04:48: ERROR CAUGHT BY DOMAIN:
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)
PM2: child_process.js:1162
PM2:     throw errnoException(err, 'spawn');
PM2:           ^
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)

Then on restart, pm2 list show no processes and pm2 restart does nothing. This happens even though from what I can tell, the dump.pm2 file in ~/.pm2 doesn't get changed in any way. I saved a copy of a file that was fine and a file that was no good (after crash), and they were binary same.

Need to dig into this some more - I can't seem to change my dump.pm2 file to be read only. Probably because I'm a debian idiot. As it stands, if I write some bad node.js code, rebooting the server won't restart what had been running. This is keeping me from being able to migrate servers - Any tips would be helpful. I've been running pm2 for over a year on another server just fine. Maybe wasn't wheezy tho..

[EdHubbell]

I changed my dump.pm2 file to be immutable (essentially read only even for root) as suggested by @frankized, and that seems to have helped. Actually got a restart after an out of memory error.

PM2: 2015-04-20 22:22:09: ERROR CAUGHT BY DOMAIN:
PM2: Error: spawn ENOMEM
PM2:     at exports._errnoException (util.js:746:11)
PM2:     at ChildProcess.spawn (child_process.js:1162:11)
PM2:     at exports.spawn (child_process.js:995:9)
PM2:     at Object.exports.execFile (child_process.js:689:15)
PM2:     at exports.exec (child_process.js:649:18)
PM2:     at getUrl (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/git.js:17:5)
PM2:     at fn (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
PM2:     at Immediate._onImmediate (/usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
PM2:     at processImmediate [as _immediateCallback] (timers.js:358:17)
PM2: Be sure to have the latest version by doing `npm install pm2@latest -g` before doing this procedure.
PM2: Error: EACCES, permission denied '/root/.pm2/dump.pm2'
PM2:     at Error (native)
PM2:     at Object.fs.openSync (fs.js:500:18)
PM2:     at Object.fs.writeFileSync (fs.js:1099:15)
PM2:     at fin (/usr/local/lib/node_modules/pm2/lib/CLI.js:646:12)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:655:28)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at ex (/usr/local/lib/node_modules/pm2/lib/CLI.js:661:14)
PM2:     at /usr/local/lib/node_modules/pm2/lib/CLI.js:662:7
PM2: [PM2] Stopping PM2...
PM2: 2015-04-20 22:22:16: Stopping app:mainOauth id:0
PM2: 2015-04-20 22:22:16: App name:mainOauth id:0 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9708 killed
PM2: [PM2] deleteProcessId process id 0
PM2: 2015-04-20 22:22:16: Stopping app:geotweet id:1
PM2: 2015-04-20 22:22:16: App name:geotweet id:1 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9711 killed
PM2: [PM2] deleteProcessId process id 1
PM2: 2015-04-20 22:22:16: Stopping app:crontruxie id:2
PM2: 2015-04-20 22:22:16: App name:crontruxie id:2 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9715 killed
PM2: [PM2] deleteProcessId process id 2
PM2: 2015-04-20 22:22:16: Stopping app:cronCalendar id:3
PM2: 2015-04-20 22:22:16: App name:cronCalendar id:3 exited with code SIGTERM
PM2: 2015-04-20 22:22:16: [HandleExit] PM2 is being killed, stopping restart procedure...
PM2: 2015-04-20 22:22:16: Process with pid 9722 killed
PM2: [PM2] deleteProcessId process id 3
PM2: [PM2] All processes have been stopped and deleted
PM2: 2015-04-20 22:22:16: PM2 is being killed via kill method
PM2: 2015-04-20 22:22:16: RPC socket closed
PM2: 2015-04-20 22:22:16: PUB socket closed
PM2: [PM2] PM2 stopped
PM2: 2015-04-20 22:22:17: [PM2][WORKER] Started with refreshing interval: 30000
PM2: 2015-04-20 22:22:17: [[[[ PM2/God daemon launched ]]]]
PM2: 2015-04-20 22:22:17: BUS system [READY] on port /root/.pm2/pub.sock
PM2: 2015-04-20 22:22:17: RPC interface [READY] on port /root/.pm2/rpc.sock
PM2: 2015-04-20 22:22:17: Starting execution sequence in -fork mode- for app name:mainOauth id:0
PM2: 2015-04-20 22:22:17: App name:mainOauth id:0 online
PM2: Process /srv/truxie/mainOauth.coffee launched
PM2: 2015-04-20 22:22:17: Starting execution sequence in -fork mode- for app name:geotweet id:1
PM2: 2015-04-20 22:22:18: App name:geotweet id:1 online
PM2: Process /srv/geotweet/geotweet.coffee launched
PM2: 2015-04-20 22:22:18: Starting execution sequence in -fork mode- for app name:crontruxie id:2
PM2: 2015-04-20 22:22:18: App name:crontruxie id:2 online
PM2: Process /srv/crontruxie/crontruxie.coffee launched
PM2: 2015-04-20 22:22:18: Starting execution sequence in -fork mode- for app name:cronCalendar id:3
PM2: 2015-04-20 22:22:18: App name:cronCalendar id:3 online
PM2: Process /srv/crontruxie/cronCalendar.coffee launched
PM2: >>>>>>>>>> PM2 updated
PM2: ┌──────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
PM2: │ App name     │ id │ mode │ pid   │ status │ restart │ uptime │ memory      │ watching │
PM2: ├──────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
PM2: │ mainOauth    │ 0  │ fork │ 10043 │ online │ 0       │ 1s     │ 16.914 MB   │ disabled │
PM2: │ geotweet     │ 1  │ fork │ 10046 │ online │ 0       │ 1s     │ 13.301 MB   │ disabled │
PM2: │ crontruxie   │ 2  │ fork │ 10051 │ online │ 0       │ 1s     │ 12.383 MB   │ disabled │
PM2: │ cronCalendar │ 3  │ fork │ 10058 │ online │ 0       │ 1s     │ 8.000 MB    │ disabled │
PM2: └──────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘
PM2:  Use `pm2 show <id|name>` to get more details about an app

I need a reliable way to create out of memory issues so I can test and make sure this is going to settle things.

[jshkurti]

It should work better with PM2 v0.12.13

[buunguyen]

@jshkurti I still have this issue running the latest version of PM2 (0.12.15) on Ubuntu 15.

After starting a process and running pm2 dump, the dump file is created with contents as expected. Testing with pm2 resurrect also works. However, after reboot, the process is not resurrected because the dump file is empty (cat ~/.pm2/dump.pm2 prints []).

Please reopen this issue and have a look if you could. Thank you.

PM2 0.12.15
Node 0.12.3
Ubuntu 15.04

[jshkurti]

@buunguyen Are you using Keymetrics ?
Does ps aux | grep Keymetrics show something about 'Keymetrics.io Agent' ?

[buunguyen]

No, I haven't installed Keymetrics on this particular host yet but I intend to. If I install the agent now, do you think it would fix the problem?

[crashbell]

@buunguyen I got the same issue and updating to 0.12.16 seems working now.

[buunguyen]

@crashbell thanks. Currently I'm working around by making dump file immutable. A bit annoying but not too painful. Will try the new version later.

[ravishivt]

I'm using v0.14.5 and still have the issue. On every reboot, the .pm2/dump.pm2 gets cleared.

[arisAlexis]

doesn't work with vanilla instructions on ubuntu 14.04 either (npm install pm2) and startup steps and save.

[sfakir]

Same problem on RedHat 7.0 :( the dump file is empty after reboot

[tangjie0710]

I also got the same problem on ubuntu 14.04 and PM2 0.14.7
I found a solution in https://github.com/Unitech/PM2/issues/1055.
rbudiharso gave the solution. You have to add PM2_HOME to the start up script.
I hope this will give help.

[crashbell]

My server restarted last night and then all processes were started automatically. The dump file wasn't empty and Keymetrics was installed. I don't remember if I did any configuration. I'll check around and let you know soon.

Still using 0.12.16

[mojojoseph]

Likewise, I have the same issue - a reboot wipes out ~/.pm2/dump.pm2. I am on a BeagleBone Black running Ubuntu with pm2 0.14.7.

@tangjie0710 My start-up script does have PM2_HOME set:
export PATH=/usr/bin:$PATH
export PM2_HOME="/root/.pm2"

[glskiba]

I've has similar problems, and checked other online threads.
try making sure all files in ~/user/.pm2 are owned by the user, not root.
This worked for me: sudo chown ~/.pm2/*

[jshkurti]

Thanks @glskiba !

[EdHubbell]

I love it when I google a problem I'm having and I get back to a GitHub issue thread that I don't even remember commenting on.

Lost a process after a reboot earlier this week. Couldn't write to the dump.pm2 file because I had set it to immutable 6 months ago. I don't remember doing that specifically, but there it is in my comment.

For the record, updating PM2 to the latest version seems to have solved my reboot problem from 6 months ago - Now all my processes restart. Debian server working just fine. Apologies for using this thread as my dev log. But maybe this helps someone else.

[artursudnik]

I did some narrowing of the same problem that I have now.

Ubuntu 15.04
node v0.12.7
pm2 0.14.7

configured pm2 with:

sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u artur"

edited pm2-init.sh file:

NAME=pm2
PM2=/usr/lib/node_modules/pm2/bin/pm2
USER=artur

export PATH=/usr/bin:$PATH
export PM2_HOME="/home/artur/.pm2"

after reboot

dump file is being emptied

pm2.log file contains then (this is only what has been written to the log after executing reboot):

2015-09-10 17:14:10: [PM2][WORKER] Started with refreshing interval: 30000
2015-09-10 17:14:10: [[[[ PM2/God daemon launched ]]]]
2015-09-10 17:14:10: BUS system [READY] on port /home/artur/.pm2/pub.sock
2015-09-10 17:14:10: RPC interface [READY] on port /home/artur/.pm2/rpc.sock
2015-09-10 17:14:11: PM2 is being killed via kill method
2015-09-10 17:14:11: RPC socket closed
2015-09-10 17:14:11: PUB socket closed
2015-09-10 17:14:41: [PM2][WORKER] Started with refreshing interval: 30000
2015-09-10 17:14:41: [[[[ PM2/God daemon launched ]]]]
2015-09-10 17:14:41: BUS system [READY] on port /home/artur/.pm2/pub.sock
2015-09-10 17:14:41: RPC interface [READY] on port /home/artur/.pm2/rpc.sock

[akmoulai]

Debian GNU/Linux 8 (jessie) / systemd / io.js 2.5.0 / pm2 0.14.7

Same issue: after a reboot, the dump.pm2 contains an empty array (i.e. []) instead of what it contained after pm2 save. pm2.log contains about the same as @artursudnik 's one (no issue reported).

I had to resort to @frankized 's solution (thank you!) which is an easy workaround for my small project.

[artursudnik]

I just made a test on Ubuntu 14.04.3 LTS and there pm2 seems to work correctly.

I guess problem is somehow related to shutdown sequence od operating system. Maybe it is killed then starts and pm2 dump is being made before applications manage to resurrect?

[OriginalEXE]

I am also experiencing this issue.

Ubuntu 14.04.3 LTS, dump.pm2 being emptied reboot (leaves only [] in the file).

[sydcanem]

Had the same issue in Centos7 with pm2 v0.14.7 running as normal user.

After reboot dump.pm2 is empty.

[qraynaud]

Same problem here. I have made my file immutable in the meantime but it is not a good solution for the long run.

[lklepner]

I had a similar problem on my end running on Ubuntu 14.04.

When I ran -
sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u ds"

The init.d script has -
export PM2_HOME="/root/.pm2"

Whereas it should be -
export PM2_HOME="/home/ds/.pm2"

[ashleydavis]

It seems I can't have my pm2 processes survive restart. This used to work. I still use the same deployment script. But 'pm2 save' doesn't persist across VM restart. Using Ubuntu 14.04.

[iongion]

@lklepner same issue - same wrong $PM2_HOME, this ticket should be re-opened

[artursudnik]

wrong $PM2_HOME value is not the only reason of not restarting processes

[iongion]

When installing with an user:
iongion: pm2 startup ubuntu

[PM2] You have to run this command as root. Execute the following command:
sudo su -c "env PATH=$PATH:/usr/local/bin pm2 startup ubuntu -u iongion"

I do that and the pm2-init.sh header becomes:

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=root
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/home/iongion/.pm2"

When installing from a user with sudo:
sudo pm2 install ubuntu

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=iongion
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/root/.pm2"

Aren't they both wrong ? Shouldn't the config be:

NAME=pm2
PM2=/usr/local/lib/node_modules/pm2/bin/pm2
USER=iongion
export PATH=/usr/local/bin:$PATH
export PM2_HOME="/home/iongion/.pm2"

In all cases, after each restart the dump file is empty, it gets populated by save but becomes empty after each restart

[tomlankhorst]

I do experience the same problem. My situation:

• Ubuntu 15.04, PM2 0.15.7, Node v4.2.0
• Dedicated user for PM2 process (which owns the .pm2 directory and its contents)

Using an init.d script generated using the su env ... command as proposed by pm2 startup. However, because the dedicated user is not in the sudoers I executed this command as another user and changed the init.d script to the right user and home folder.
Setting the immutable bit solves this (temporarily) for me too.

[choxnox]

I have the same issue on the following:

• Ubuntu 15.04
• Node 0.12.7
• PM2 0.15.8

What also worked for me is setting the immutable attribute which is an ugly hack because I'll most probably forget I did it when I want to modify dump.pm2 in the future (adding/removing applications in PM2).

[Holger-Will]

Same issue here

• Ubuntu 15.04
• Node 5.0.0
• PM2 0.15.10

dump.pm2 is empty after reboot. I'm also using the immutable hack now...

[ccravens]

PING Any help with this?? My dump.pm2 also clears after reboot and none of my processes start, which kind of defeats the whole purpose of using PM2 as a process manager in the first place! I think marking the dump file as immutable on the filesystem is a terrible workaround... Why doesn't PM2 just do what it claims to do instead?

PM2 v0.15.10
CentOS 7
node v4.2.1

[tomlankhorst]

Please reopen the issue.

[ccravens]

+1 for re-opening this issue

[lklepner]

+1 for reopen

[dimd13]

Debian 8.2 /home/user/.pm2/dump.pm2 is empty after reboot

[mlota]

Same problem for me I'm using pm2 v0.15.10 on Raspbian Jessie. The $PM2_HOME looks correct but it loses the data after every reboot

[bhaveshgohel]

@mlota : Same here. :(
Node.js : v4.2.3
Pm2 version : v0.15.10
Hardware : Raspberry Pi 1 B+
OS : DietPi Jessie

Workaround if anyone needed:
Make your dump.pm2 file IMMUTABLE (Unchangeable).

Run below commands to make dump.pm2 IMMUTABLE:

sudo pm2 save
sudo chattr +i  /your/path/to/dump.pm2

The problem is that for any change you have to,

sudo chattr -i  /your/path/to/dump.pm2
sudo pm2 save
sudo chattr +i  /your/path/to/dump.pm2

Hoping this issue in pm2 solves soon. :)

[artursudnik]

Why it takes so long to address this very serious issue?

[webuniverseio]

@artursudnik are you offering your help with these words?

[artursudnik]

No, I am just curious about a reason. Is this matter of design? If I only were familiar with the source code I could try to debug it. What I has done is performing some tests on number of linux versions but cannot find any correlations. Seems to be random for me.

[mlota]

I've also tried all the workarounds posted but in the end just had to ditch PM2 and use some startup scripts instead. It's a real shame as I have it working on another system flawlessly which has an identical setup. Like @artursudnik, I find the issue is totally random

[vamship]

I'm having the same issue on a minimal jessie on the raspberry pi.

It appears that PM2 is dumping all running processes at shutdown, but this is somehow taking place after all the PM2 processes have been killed, which means that the dump file now has an empty array, and nothing starts on bootup.

My workaround was to change the /etc/init.d/pm2-init.sh, and comment out the pm2 dump statement under the stop() function. I can then add scripts to startup by manually calling pm2 dump after launching the processes.

It would be cool if the pm2 startup script took an option that would automatically suppress auto save of processes at shutdown, instead of me having to go in there and tweak the shell script.

[artursudnik]

But it is really strange that it does not happen always and apparently there is no correlation to OS version.

[Holger-Will]

In my production machines which run ubuntu 14.04 LTS it all runs fine. On the devel boxes which are on ubuntu 15.04 or 15.10 however i can See this bug... has to be connected OS Version, as i update pm2 on all machines regularly.

[OriginalEXE]

has to be connected OS Version

I doubt it is since many of us has problems on different OS versions. I for example have this issue on 14.04

[yoava]

Same bug for me on raspbian (jessie/8).
Worked fine 6 months ago (don't know which version).
Node v4.2.1

[Holger-Will]

this might be connected to upstart, or ubuntus transition to systemd. I'm currently running a fresh install of arch with systemd, and pm2 works as expected. As soon as i have access to an ubuntu which shows the problem, i can check.

[Holger-Will]

oh, for everyone experiencing this error, please try

pm2 startup systemd

[rishabhmhjn]

@Holger-Will I tried the systemd and it also didn't work!

2 issues with systemd approach:

• /usr/bin/env: node if we use the auto generated startup pm2.service file
• When fixed that by adding source /path/to/nvm.sh, it still didn't work

PM2 v0.15.10
CentOS 7
node v5.3.0

PS: going with the immutable option for now as it seems to be solving the issue temporarily

[Holger-Will]

well ok :-( thanks for testing anyways...

[wacky6]

Same problem on Raspberry Pi 2, Debian 7, node v4.2.2, PM2 0.14.7

@vamship's solution seems better to me.
Maybe PM2 should stop dumping process list in pm2-init.sh stop, and require user to manually commit(save) process list(similar to update-rc.d command)?

I am also curious about how PM2 dumps process list, if the process is killed but not restarted by PM2 during system shutdown.

[rudders]

Whilst I appreciate that keymetrics is an excellent commercialisation strategy for and open source project like PM2 the fact that this bug continues to not be resolved makes a mockery of their claims for production worthy use of this tool. A starting price of $30/month puts it at business level pricing but this bug flys in the face of anyone wanting to consider this a worthy platform for production use. #IMHO

[artursudnik]

+10

[ronaldb]

Commenting out the "super $PM2 dump" in /etc/init.d/pm2-init.sh seems to work. You just have to remember to do a "pm2 dump" manually after making changes.

A crude work around, but easier to do than making pm2.dump read-only and re-enabling writes when you make a change.

[artursudnik]

Ronalddb, seems it is most convenient workaround. Thanks! I've just used it.

[kenhowardpdx]

@ronaldb thanks for your workaround!

[h4t0n]

@ronaldb It works for me. Thanks.

pm2 v1.0.0
node v4.2.4
Centos 7.2

[benforshey]

@vamship Your workaround helped—thanks!

pm2 v1.0.0
node v5.4.1
Ubuntu 15.10

[Braynid]

@ronaldb - that works, thanks!

pm2 1.0.0
node v5.0.0
CentOS 7.2.1511

[pwang2]

Take me a while to get this workaround. is it an issue under working?

[artursudnik]

This issue is real mystery for me.

[OriginalEXE]

What I'm surprised about the most is the lack of interest from the authors on this issue. It's obviously widespread and a very important part of the software doesn't seem to be working, so why they still have not solved this (or at least let us know it can't be solved), I have no idea. I mean, I am grateful for their contribution and all, but this should be taken more seriously IMO.

[Unitech]

The issue about node paths / user permissions have been fixed since the PM2 0.15.x.

As I understand by reading this thread is that the automatic process dump (pm2 dump on stop) is the main issue.

I will disable this behavior and now users who wants to keep their current processes will have to expressly type pm2 save

[artursudnik]

Great decision!

[OriginalEXE]

Thank you @Unitech !

[Unitech]

Alright just tried on different machines, it works like a charm!

I've updated the documentation if anyone would have some time to verify that it's grammatically correct

[Unitech]

I wonder if this issue is linked with #1321

[Unitech]

1891

[Tomino2112]

I am not sure why, but for me pm2 is totally ignoring user configs.

what I need to do is : sudo -u www-data pm2 start ecosystem.json

But I get error: EACCES: permission denied, open '/home/tomas/.pm2/pm2.log'

How is it that even if I run command as a different user, it still goes into the original user's home?

I tried the startup script changing user to www-data but after restart no apps are running.

Also when I run pm2 start ecosystem -u www-data pm2 starts the app but still as a user tomas

[roccomuso]

Same error for me in Raspbian on Raspberry pi 2.

Tried to pm2 save and restart, but the processes are gone.

This is a crucial feature.

[kylecordes]

I am also surprised to see the trouble with this, so far in to the life of a generally very slick and polished tools - it seems to be right at the heart of pm2's purpose.

[artursudnik]

The workaround has been already implemented: https://github.com/Unitech/pm2/commit/f2714c7385681c7592cf4c5629a6a75efc7db4d9

[willkramer]

@kylecordes I agree and I've given up on the pm2 save functionality. The workarounds don't seem to work for me reliably on the Amazon Linux AMIs.

My solution was to create a bash script that runs the "pm2 start server.js" command and then create another bash script as a service in /etc/init.d/myserver. I can then start, stop, and restart pm2 with the command "service myserver start"

My pm2 start/stop/restart scripts look something like (this is my avoneu_start.sh):

#!/bin/bash
cd /avoneu/api
pm2 start server.js

I then create a file in /etc/init.d/ called avoneu. My service bash script looks like:

#!/bin/bash
# chkconfig: - 85 15
# description: The Avoneu Node Server
HOME_FOLDER="/home/ec2-user"

start(){
        echo "Starting node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_start.sh"
}
stop(){
        echo "Stopping node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_stop.sh"
}
restart(){
    echo "Restarting node server"

        eval "runuser - ec2-user -c $HOME_FOLDER/avoneu_restart.sh"
}
case "$1" in
  start)
        start
    ;;
  stop)
        stop
    ;;
  restart)
    restart
        ;;
  *)
        echo "Usage: avoneu {start|stop|restart}"
        exit 1
esac

exit 0

Also, I can easily ensure the the service auto starts on reboots by running the command

chkconfig avoneu on

This solution has been 100% reliable for the last 8 months and I was able to make it part of my server formation templates. I don't fault pm2 for having auto start issues; auto start on each Linux OS can be very different and it is hard to account for every minor difference.

[kylecordes]

I am using Amazon Linux. I found that it worked for me if I did the chattr thing to forcibly prevent the file from being damaged. The version I'm using already had the "save" command in the script commented out, that was not enough to prevent the breakage. So apparently it is not just that one command causing the breakage, there is something beyond that going on.

[zettam]

So what is the final state with this?
Do we have to manually "save" the state now?

If this is the case, there is still a bug, which is: when you don't save anything and just power off the machine (no reboot or shutdown command, direct power off) and restart the final state is there (even though pm2 save command was not run before!)

So some things apparently get saved. I just do not understand why it just fails to work on machine reboots.

[minwe]

Not working +1：

Ubuntu 14.04.3 LTS
node -v v4.2.1
pm2 -v 1.1.3

I'm using Microsoft Azure, install pm2 with azureuser.

UPDATE：

I think it caused by fish shell. It works well after changing default shell to bash.

So, just check your default shell, does it work with export order? Fish shell doesn't.

[mjkoster]

I had to make the file immutable also, on Ubuntu 15.10. The dump file is being cleared when I do "pm2 kill". Apparently there is some problem...

[cmundi]

Problem still exists. Behavior suggests a race condition.
Debian 8.5
pm2 version 1.1.3

Sometimes apps are restarted after reboot, sometimes not. When the dump file is wiped out, at least there is a reason for the failure to restart apps. But it also sometimes happens that the dump file is still intact after reboot, but apps are not restarted. Smells like race condition(s) to me, but I have no proof. These are multiple, successive reboots. There's no other change in system configuration between reboots. This is disappointing, but fortunately there are lots of other ways to run jobs at startup on a linux box if pm2 ends up not being reliable on my boxes.

Making the dump file immutable may solve the problem for some systems. But immutability does _not_ solve the problem for my minimal Debian 8.5 boxes. I made the dump file immutable, so at least it "can't" be rewritten. (And I have cached a copy and verify that it is not changing between reboots.) Then I do a series of reboots and checked to see if my app restarted:

reboot 1 ... app running
reboot 2 ... app NOT running
reboot 3 ... app running
reboot 4 ... app running
reboot 5 ... app running
reboot 6 ... app NOT running
reboot 7 ...app running

So even after the file is made immutable, the behavior is inconsistent. I think it's fair to call that a surprising result. I suspect a race condition. As others have reported, the logs reflect the problem but offer no clues to the cause.

Not that it should matter but this experiment was performed on a VM. All pm2 commands run as root.

[boatrite]

Just to throw the solution I found into this massive issue...
I was following the instructions at https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps
and ran sudo env PATH=$PATH:/opt/node-v4.4.6-linux-x64/bin/ pm2 startup ubuntu -u deploy.
This is as the ubuntu user of an Amazon AWS instance, Ubuntu 14.04.

However, this is what was generated at /etc/init.d/pm2-init.sh (relevant bit only):

export PM2_HOME="/home/ubuntu/.pm2"

So the problem is that pm2 startup didn't set PM2_HOME correct, even with specifying the user. After changing PM2_HOME to be correct, everything worked perfectly.

[cmundi]

@boatrite Thanks for that link and example. That addresses one of the several behaviors which make RCA challenging for this constellation of related behaviors:

1. User not being set correctly, as you describe.
2. The dump file being overwritten, sometimes.
3. My report that even when user is correct (including root) and dump file is made immutable, sometimes pm2 starts on boot and sometimes it does not. It seems random.

I like the goals of pm2 and wish I had time to dive into the code. Right now I need to go with the reliability and transparency of init/systemd. Even though our clusters are small and well instrumented, we'll keep an eye on pm2.

Thanks!

[qraynaud]

@cmundi : your problem seems to be a completely separate bug then.

Maybe you should file it separately then? My rationale is that since the issue described here is properly fixed by the changes suggested by @boatrite, we need 2 patches : one or this issue and one for yours.

[github-jeff]

Having read this commentary. I have tried everything which was plausible. The dump file was not getting overwritten (wiped out) on reboot, but the dump file structure was changed from human readable to one single line. By making it read only this did save the human readable structure. The pm2-init.sh file already had the super dump line already commented out. On reboot pm2 does not run as a service, but it does run demonized when I try pm2 list. In any case the app does not survive reboot on ubuntu 14.04.

Any other tips and tricks I should try?

[Unitech]

New startup system that will surely fix a bunch of problems mentioned on this issue:

https://github.com/Unitech/pm2/issues/2559

[dikkedimi]

I'm currently having this issue on a raspberry pi 3, with raspbian (latest build) and pm2 v2.4.2

[vmarchaud]

@dikkedimi Check this issue : https://github.com/Unitech/pm2/issues/2747

[Ext3R]

"pm2 resurrect" works for me after rebooting - but not automatically.
-> added the code: "@reboot pm2 resurrect" to "crontab -e" -> now everything works just fine.
Don´t know what fails here. It saves the logs perfectly but cant start the apps automatically.

[lukepighetti]

Is there any way to save and resurrect the running state of each?

Lets say:

fooApp  online
barApp  stopped
bazApp  online

Can I set it up so that fooApp/bazApp are online, and barApp is stopped on reboot?

Currently they start up all online.

[wallet77]

Hi @lukepighetti

Currently is not possible to save and resurrect the exact state.
PM2 as a process manager always tries to restart your apps.

But this is part of a reflection about pm2 save/resurrect, see this issue : https://github.com/Unitech/pm2/issues/3759

[RicardoZap]

Una disculpa por comentar en español en un foro en Inglés pero encontré la solución al problema.
Ami me funcionó ejecutando el siguiente método:

1.- pm2 start "ejemplo.js"
2.- pm2 save
3.- pm2 startup
4.- sudo env PATH=$PATH:/usr/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup systemd -u pi --hp /home/pi

Después de ejecutar lo anterior hacer un "sudo reboot" y con eso el problema queda resuelto. Saludos.