Pm2: TCP port isn't released on reload when app is running in cluster mode

[vinogradov-m]

What's going wrong?

When I try to reload the application which listens on some TCP port the process fails with error Error: listen EADDRINUSE

How could we reproduce this issue?

Generate the ecosystem file

mkdir cluster-mode-test
cd ./cluster-mode-test
pm2 init

add exec_mode: 'cluster' to the app definition.

Here is the sample app:

const http = require('http');

const PORT = 3010;

const server = http.createServer((req, res) => {
  res.write('Hello World!');
  res.end();
});
server.listen(PORT);

Then start and reload the app:

pm2 start ecosystem.config.js 
pm2 reload ecosystem.config.js 

Supporting information

When I define exec_mode as 'cluster_mode' or don't use ecosystem file everything works fine. It seems the issue is related to using alias 'cluster' for exec_mode.

I found that with pm2 v3.1.3 everything works flawlessly.
As I understood, somewhere on reload app.exec_mode is compared with string 'cluster_mode', not just 'cluster'.
Earlier there was a function checkExecMode() which sets exec_mode to 'cluster_mode' if it was just 'cluster'. But it was removed several days ago.

===============================================================================
--- PM2 REPORT (Mon Oct 08 2018 12:48:44 GMT+0300 (Moscow Standard Time)) -----
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.2.2
node version         : 10.10.0
node path            : /Users/stealth/.nvm/versions/node/v10.10.0/bin/pm2
argv                 : /Users/stealth/.nvm/versions/node/v10.10.0/bin/node,/Users/stealth/.nvm/versions/node/v10.10.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : stealth
uid                  : 501
gid                  : 20
uptime               : 2min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.2.2
node version         : 10.10.0
node path            : /Users/stealth/.nvm/versions/node/v10.10.0/bin/pm2
argv                 : /Users/stealth/.nvm/versions/node/v10.10.0/bin/node,/Users/stealth/.nvm/versions/node/v10.10.0/bin/pm2,report
argv0                : node
user                 : stealth
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
cpus nb              : 8
freemem              : 7536812032
totalmem             : 17179869184
home                 : /Users/stealth
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────────────┬────┬─────────┬─────────┬───────┬─────────┬─────────┬────────┬─────┬───────────┬─────────┬──────────┐
│ App name          │ id │ version │ mode    │ pid   │ status  │ restart │ uptime │ cpu │ mem       │ user    │ watching │
├───────────────────┼────┼─────────┼─────────┼───────┼─────────┼─────────┼────────┼─────┼───────────┼─────────┼──────────┤
│ Cluster mode test │ 0  │ 1.0.0   │ cluster │ 12181 │ online  │ 0       │ 2m     │ 0%  │ 33.7 MB   │ stealth │ disabled │
│ Cluster mode test │ 1  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
│ Cluster mode test │ 2  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
│ Cluster mode test │ 3  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
└───────────────────┴────┴─────────┴─────────┴───────┴─────────┴─────────┴────────┴─────┴───────────┴─────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
┌───────────────────┬────┬─────────┬─────────┬───────┬─────────┬─────────┬────────┬─────┬───────────┬─────────┬──────────┐
│ App name          │ id │ version │ mode    │ pid   │ status  │ restart │ uptime │ cpu │ mem       │ user    │ watching │
├───────────────────┼────┼─────────┼─────────┼───────┼─────────┼─────────┼────────┼─────┼───────────┼─────────┼──────────┤
│ Cluster mode test │ 0  │ 1.0.0   │ cluster │ 12181 │ online  │ 0       │ 2m     │ 0%  │ 33.7 MB   │ stealth │ disabled │
│ Cluster mode test │ 1  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
│ Cluster mode test │ 2  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
│ Cluster mode test │ 3  │ 1.0.0   │ cluster │ 0     │ errored │ 15      │ 0      │ 0%  │ 0 B       │ stealth │ disabled │
└───────────────────┴────┴─────────┴─────────┴───────┴─────────┴─────────┴────────┴─────┴───────────┴─────────┴──────────┘

[jybleau]

Yes, it wrongfully tries to restart in fork mode when using exec_mode : "cluster"
But it's the correct way to set it according to the doc: https://pm2.io/doc/en/runtime/reference/ecosystem-file/

On first pm2 start ecosystem.json5, everything loads as expected:
_From the .pm2/pm2.log file:_

starting in -cluster mode-

Then on pm2 reload ecosystem.json5or restart or stop/start :

starting in -fork mode-

Also, when using instances : "max" or instances : "n" the exec_mode is not required, it is implicitly set to cluster correctly and start/restart as expected.

[fozboz]

I had a similar issue when using the alias fork instead of fork_mode. On first run, it looks fine. But on subsequent restarts, it says it's starting in fork mode, but pm2 list says otherwise. This only happens if I restart using the json file as the argument. Using pm2 restart <id> or pm2 restart <app_name> works fine.

{
  "apps" : {
    "name":       "my-service",
    "version":    "1.5.2",
    "script":     "my-service/app.js",
    "watch":      false,
    "args":       "myconfig=myconfig",
    "exec_mode":  "fork"
  }
}

pm2 start my-service.json
PM2 > App [my-service:54] starting in -fork mode-

pm2 list
my-service │ 55 │ 1.5.2 │ fork │ 14262 │ online │ 0 │ 2m │ 0% │ 50.0 MB │ env │ disabled

pm2 restart my-service.json
PM2 > App [my-service:54] starting in -fork mode-

pm2 list
my-service │ 56 │ 1.5.2 │ cluster │ 14941 │ online │ 1 │ 0s │ 0% │ 17.0 MB │ env │ disabled

Using "exec_mode": "fork_mode" resolves the issue.

[stgogm]

I've had a similar issue when declaring exec_mode to cluster in my ecosystem file and it caused the app to throw EADDRINUSE after a reload, restart or starting the app again without deleting the previous instances.

It seems that the problem is that when exec_mode is set to cluster PM2 doesn't delete the previous instances on reload or restart.

I can confirm that setting exec_mode to cluster_mode solves that issue.

[blacksunset]

@stgogm thx

[nezed]

I've meet this bug on versions 3.2.0 - 3.2.2, now in 3.2.3 it was fixed

My ecosystem.config.js:

module.exports = {
  apps: [{
    name: 'events_broker',
    script: './app.js',
    exec_mode: 'cluster',
    instances: 4,

    wait_ready: true,
    listen_timeout: 3000,
    min_uptime: 5000,

    kill_timeout: 11000,
    restart_delay: 500,

    max_memory_restart: '800M',

    // some logs configs
  }],
}

And after pm2 reload ecosystem.config.js app looks like:

I've noticed that graceful startup are also broken

[Unitech]

With PM2 updated to 3.2.3, run pm2 delete all and start your application again, everything should work as expected. If not please let me know

[nezed]

@Unitech just

npm i -g pm2
pm2 kill
pm2 start ecosystem.config.js

helped for me

[Unitech]

fixed long time ago! Make sure to install latest PM2 version:

$ npm install pm2@latest -g
$ pm2 update