Pm2: API restart() not working within PM2 started process

[ColdFireIce]

Hi,

I have some weird behavior with the programmatic restart() function:

In my test I have a global PM2 installation (npm install pm2 -g)

And a Test-Project with 2 files and a local PM2 install (npm install pm2 --save):
foo.js:

setInterval(function () {console.log(new Date().toISOString());}, 2000);

test.js (not showing the pm2.disconnect() on SIGTERM or SIGINT):

var pm2 = require('pm2');
pm2.connect();

setTimeout(function () {
    pm2.restart('foo', function (err, apps) {
        console.log("RESULT", err, apps);
    });
}, 5500);

I then start the foo process with: pm2 start foo.js (works as expected)
Starting the test script with: pm2 start test.js also works,
but as soon as the restart code is triggered some weird things happen:

1. RESULT null [ { name: 'test', pm_id: '6', status: 'online', restart_time: '0', pm2_env: { name: 'test', pm_id: '6', status: 'online', restart_time: '0', env: [Object] } } ] this is the console output. It seems that it is not restarting "foo", but itself
2. pm2 list shows foo as restarted once and stopped, as well as "watching" now enabled

┌──────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid  │ status │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ foo      │ 0  │ fork │ 2887 │ online │ 0       │ 21s    │ 19.863 MB   │ disabled │
│ test     │ 1  │ fork │ 2908 │ online │ 0       │ 5s     │ 29.168 MB   │ disabled │
└──────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────────────┴──────────┘

┌──────────┬────┬──────┬──────┬─────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid  │ status  │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼──────┼─────────┼─────────┼────────┼─────────────┼──────────┤
│ foo      │ 0  │ fork │ 0    │ stopped │ 1       │ 0      │ 0 B         │  enabled │
│ test     │ 1  │ fork │ 2922 │ online  │ 0       │ 3s     │ 29.242 MB   │  enabled │
└──────────┴────┴──────┴──────┴─────────┴─────────┴────────┴─────────────┴──────────┘

1. every 5 seconds a new instance of test.js is started, but not in PM2, it only shows in "ps aux | grep node"

If I start foo.js with pm2 and run test.js with node (node test.js) everything works as expected.

I had a simillar issue with version conflicts between the global and the local pm2 version for the API start() function but they are gone after i updated both versions to 1.1.1.
stop(), describe() and delete() seem to work fine.

Greetings

[btsantos]

I have the same issue, executing command restart ...

The process send de Stop correctly, but then send a start to the process that execute/call the action

Running node standalone, process name YYYYY:
PM2 Stopping app:XXXXX id:12
PM2 App [XXXXX] with id [12] and pid [14039], exited with code [0] via signal [SIGINT]
PM2 Starting execution sequence in -fork mode- for app name:XXXXX id:12
PM2 App name:XXXXX id:12 online

Running node with pm2, process name YYYYY:
PM2 Stopping app:XXXXX id:12
PM2 App [XXXXX] with id [12] and pid [14039], exited with code [0] via signal [SIGINT]
PM2 Starting execution sequence in -fork mode- for app name:YYYYY id:10
------ERROR'S-------

[FredyC]

I just encountered similar issue. Essentially all have been already said, restarted process is stopped correctly and then same as already running one is started instead.

First I tried a workaround to actually restart through spawning child process pm2 restart server, but that has failed with this:

/usr/lib/node_modules/pm2/node_modules/amp-message/index.js:126
  return new Buffer('j:' + JSON.stringify(arg));
                                ^
TypeError: Converting circular structure to JSON
    at Object.stringify (native)
    at pack (/usr/lib/node_modules/pm2/node_modules/amp-message/index.js:126:33)
    at encode (/usr/lib/node_modules/pm2/node_modules/amp-message/index.js:101:14)
    at Message.toBuffer (/usr/lib/node_modules/pm2/node_modules/amp-message/index.js:68:10)
    at ReqSocket.Socket.pack (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/sock.js:92:14)
    at ReqSocket.send (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/req.js:96:21)
    at Client.call (/usr/lib/node_modules/pm2/node_modules/pm2-axon-rpc/lib/client.js:34:13)
    at Object.executeRemote (/usr/lib/node_modules/pm2/lib/Satan.js:563:23)
    at /usr/lib/node_modules/pm2/lib/CLI.js:1047:13
    at /usr/lib/node_modules/pm2/node_modules/async/lib/async.js:181:20
  at ChildProcess.exithandler (child_process.js:202:12)
  at emitTwo (events.js:106:13)
  at ChildProcess.emit (events.js:191:7)
  at maybeClose (internal/child_process.js:850:16)
  at Process.ChildProcess._handle.onexit (internal/child_process.js:215:5)

Next attempt was to stop process through API as that seems to be working. However since it stays in PM2 memory then trying to start a new process with same name has actually another weird outcome as it replaces currently running process. So I had to delete whole process before starting new one. It's far from ideal, but at least it works.

It's really strange behavior I would not expect from this. It almost looks like that API calls doesn't see already running processes, yet when I try list or describe, everything seems in order.

[ageorgios]

Same issue as FredyC.
both node v5.10.0 and v6.1.0

[reinaldorauch]

+1 on node v6.1.0

[jojow]

Same issue here when running the pm2 restart command with

• node v4.4.4 (running on official Docker node image)
• pm2 v1.1.3

# pm2 restart my-app                                                                                                                                             
/usr/local/lib/node_modules/pm2/node_modules/amp-message/index.js:126                                                                                                            
  return new Buffer('j:' + JSON.stringify(arg));                                                                                                                                 
                                ^                                                                                                                                                

TypeError: Converting circular structure to JSON                                                                                                                                 
    at Object.stringify (native)                                                                                                                                                 
    at pack (/usr/local/lib/node_modules/pm2/node_modules/amp-message/index.js:126:33)                                                                                           
    at encode (/usr/local/lib/node_modules/pm2/node_modules/amp-message/index.js:101:14)                                                                                         
    at Message.toBuffer (/usr/local/lib/node_modules/pm2/node_modules/amp-message/index.js:68:10)                                                                                
    at ReqSocket.Socket.pack (/usr/local/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/sock.js:92:14)                                                                   
    at ReqSocket.send (/usr/local/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/req.js:96:21)                                                                           
    at Client.call (/usr/local/lib/node_modules/pm2/node_modules/pm2-axon-rpc/lib/client.js:34:13)                                                                               
    at Object.executeRemote (/usr/local/lib/node_modules/pm2/lib/Satan.js:563:23)                                                                                                
    at /usr/local/lib/node_modules/pm2/lib/CLI.js:1047:13                                                                                                                        
    at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:181:20

[ageorgios]

Could anyone determine the cause?

[jojow]

It's kind of strange. When I try the pm2 restart command inside the Docker node 4.4.4 container (https://hub.docker.com/_/node), it fails as shown previously.

However pm2 restart works inside the alpine-node 4.4.4 container (https://hub.docker.com/r/mhart/alpine-node), but only with its default shell.
When I try to run pm2 restart inside bash in that container, it fails with the same error.

[kongnet]

node v6.1.0
pm2 v1.1.3
same issue in my CI system when finish testing and restart process.

let result = yield co_exec(`pm2 restart core100`);

[Tarrask]

I guess that I have the same issue.

pm2.restart(process_name);                --> stop "child" process, but start "controller" one
child_process.exec("pm2 restart child");  --> JSON circular

What I've found is that with an empty env, the child_process works:

child_process.exec("pm2 restart child", {env: {PM2_HOME: process.env.PM2_HOME}});  --> works nice

Digging a bit more, it look like process.env.name and process.env.pm_id force pm2 to start the wrong process, process.env.PM2_JSON_PROCESSING cause the JSON circular exception and process.env.PORT cause the child process to starts on the host port (on my setup at least).

For me that works well:

var env = _.clone(process.env);
delete env.name;
delete env.cwd;
delete env.pm_exec_path;
delete env.pm_cwd;
delete env.pm_out_log_path;
delete env.pm_err_log_path;
delete env.pm_pid_path;
delete env.PM2_JSON_PROCESSING;
delete env.PORT;
delete env.pm_id;
exec(`pm2 restart ${childName}`, {env}, (err, stdout, stderr) => { ... });

[smallpath]

node v5.6.0
pm2 v1.1.3

same issue in my webhook service.Have spent 5 hours to figure out it's a bug only with pm2

[geronime]

We also run into this issue, it took some time to figure this out. Is there any estimation as to when this will get fixed?

[aurium]

node v6.2.2
pm2 v1.1.3

Same issue. My app is receiving a SIGINT each second. It was working nice on my old server with an old pm2 version.

[kipgraham]

node v4.4.0, v4.4.7
pm2 v1.1.3

pm2.restart(process_name) restarts the first process in my pm2 list, not the one I intend. Same happens when I pass the pm_id instead of the name. Both my processes are then set to "watching".

I can get it to work If I pm2 delete all apps except the one I intend to restart, but that's not desirable.

[Unitech]

Could you please try with pm2 v2:

$ npm install Unitech/pm2#development -g
$ pm2 update

[Unitech]

alright restart call is now fixed

fixed via safeExtend when restart command called: https://github.com/Unitech/pm2/blob/167d8243207fa0acc5231a9babd53d50df92062d/lib/CLI.js#L921

tests: https://github.com/Unitech/pm2/blob/development/test/programmatic/inside.mocha.js#L5
fixture scripts: https://github.com/Unitech/pm2/tree/development/test/fixtures/inside

Available in development branch:

$ npm install Unitech/pm2#development -g
$ pm2 update

[kipgraham]

Thanks for the quick update! After upgrading to the development branch, I'm getting the following error whenever I run any pm2 command (even pm2 version) unless I run it with sudo. This is occurring both on my local OS X and my remote Ubuntu server. I ran the install script using sudo (it gave permission errors when I ran without).

fs.js:549
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: EACCES: permission denied, open '/usr/local/lib/node_modules/pm2/bin/.cache-require-paths.json'
    at Error (native)
    at Object.fs.openSync (fs.js:549:18)
    at Object.fs.writeFileSync (fs.js:1156:15)
    at process.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/cache-require-paths/index.js:47:6)
    at process.g (events.js:260:16)
    at emitOne (events.js:77:13)
    at process.emit (events.js:169:7)
    at process.exit (node.js:750:17)
    at tryToExit (/usr/local/lib/node_modules/pm2/lib/CLI.js:204:19)
    at /usr/local/lib/node_modules/pm2/lib/CLI.js:223:7

[Unitech]

@kipgraham alright I see the issue, I will move cached paths for require into .pm2 configuration folder

https://github.com/bahmutov/cache-require-paths

[kongnet]

update pm2 2.0.0-rc3 but Problem still exists.
I cant complete restart when run shell script.

[kongnet]

!/bin/bash

echo "shell begin"
git pull
pm2 restart zc_core13010

not restart complete

git hook

yield co_exec(sh ${__dirname}/../deploy_zc_core_restart.sh);

old pm2 version is ok

[Unitech]

@kipgraham yes for now it supports call via the PM2 API, I have to check for another way to discover if the command is runned inside PM2
https://github.com/Unitech/pm2/blob/development/lib/CLI.js#L920

[colegrigsby]

@Unitech The dev branch fix seems to fix it for pm2 calls with the process names, but there are still some issues for a pullAndRestart/pullAndReload call from a module.

ReferenceError: EXEC_TIMEOUT is not defined at exec (/usr/lib/node_modules/pm2/lib/CLI/Version.js:247:16)

[Unitech]

@kongnet fixed
https://github.com/Unitech/pm2/commit/55d910b144e9988d884773e01cc2f5f11a1fb5d2

[Unitech]

@colegrigsby ok a test suit was missing for the versioning feature so I've added one and fixed the EXEC_TIMEOUT variable

Test suit: https://github.com/Unitech/pm2/blob/722845b521ef85827ed41d43e13ff5b77c6fec60/test/bash/versioning-cmd.sh

closing, please try fixes and tell me if everything is fine,
thanks for your feedback, much appreciated!

[kipgraham]

@Unitech everything works now, thanks again for the quick turnaround!

[colegrigsby]

@Unitech Everything works perfectly when I am running a module using pm2 install ., but if I publish the module using pm2 publish and install using pm2 install MODULE_NAME then the pm2.reload call does not behave correctly. It stops the application and starts the module a second time.

Any ideas why this may be happening? I am still on the pm2 development branch.

[rigor789]

I am still seeing this issue on the v2 version.

I added some logging for process:event and here is the outcome of running restart programatically:

FirstProcess-0 Event start SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event online SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event exit SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event exit SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event stop SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event restart SecondProcess /path/to/project/src/SecondProcess.js
FirstProcess-0 Event online SecondProcess /path/to/project/FirstProcess.js

The format is console.log('Event', e.event, e.process.name, e.process.pm_exec_path)

As you can see the SecondProcess starts runing the FirstProcess.js script.

I will keep digging to find the cause.

Update:
It all works as expected if the FirstProcess.js is started vie node FirstProcess.js without pm2.

Update2:
Deployed to production, had to ditch running the master process with pm2, since at the first restart / stop it would turn itself into the managed process, so I would end up with 2 processes with the same name and everything would break. I am currently running the master process with forever, and this way the pm2 programmatic api works fine and as expected. Just updated the servers pm2 from 0.14.* where everything worked fine. 1.1.3 does not work, nor does the latest development version (2.0.4).

[rigor789]

Can we reopen this issue? (see my update2 above)

[vmarchaud]

the issue should be fixed in version 2.0.12, can you confirm this @rigor789 ?

[rigor789]

I just updated to 2.0.15 and I seem to be having the same issue, any ideas where to look / how can I help pinpoint this issue?

[vmarchaud]

@rigor789 Should be really useful if you provide us some files to reproduce the issue easily !

[rigor789]

Here is a small project that shows the issue. https://github.com/rigor789/PM2-bug

running pm2 start entry.js && pm2 logs you can see the output. First it starts the other.js file, and after 10 seconds it tries to restart the process, and everything goes wrong. Both processes become Other and in this case just run 2 instances, but in case you need to bind to a port it fails. (that's the case on my production app)

[kipgraham]

after a few months without any issues this bug is now back for me - the wrong process is called on restart. Tried upgrading pm2 to 2.0.19, node to 6.9.1 and still no success. Having the issue on MacOS and Ubuntu 16.04

[beniboy87]

Yep, I'm having the same issue right now, I can only solve the problem if I delete the processes and start them again. This happens only with pm2, when I run the scripts with just node, everything works as expected.

[wallet77]

Hi @rigor789

I'm not able to reproduce even with a clone of your example.
Tested on :

• ubuntu 18.04
• node 4/6/8
• pm2 3.0.3

[rigor789]

@wallet77 I haven't been using it recently, but glad to see it fixed in 3.x!

The workaround I used is still up and running in production 👍