Pm2: Cluster mode and EADDRINUSE

Created on 31 Mar 2019  ·  10Comments  ·  Source: Unitech/pm2

This might be a duplicate issue however all linked issues are not able to resolve this issue I am having right now with the particular build version of https://github.com/vanila-io/spectrum.

What's going wrong?

  • Express API >> Program exits with EADDRINUSE, status is still online.
  • Bull Queue >> Program does not restart on error, status is still online.
  • Removing exec_mode does not work.
  • Changing cluster to cluster_mode does not work.

How could we reproduce this issue?

Unfortunately this happens on a production version, not on a development version. I have zero clues about reproduction right now.

Supporting information

--- PM2 report ----------------------------------------------------------------
Date                 : Sun Mar 31 2019 12:58:52 GMT+0000 (UTC)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.2.9
node version         : 9.11.2
node path            : /root/.nvm/versions/node/v9.11.2/bin/pm2
argv                 : /root/.nvm/versions/node/v9.11.2/bin/node,/root/.nvm/versions/node/v9.11.2/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 18min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.2.9
node version         : 9.11.2
node path            : /root/.nvm/versions/node/v9.11.2/bin/pm2
argv                 : /root/.nvm/versions/node/v9.11.2/bin/node,/root/.nvm/versions/node/v9.11.2/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : AMD EPYC 7501 32-Core Processor
cpus nb              : 2
freemem              : 991858688
totalmem             : 4143312896
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬─────────┬─────────┬──────┬────────┬─────────┬────────┬──────┬────────────┬──────┬──────────┐
│ App name │ id │ version │ mode    │ pid  │ status │ restart │ uptime │ cpu  │ mem        │ user │ watching │
├──────────┼────┼─────────┼─────────┼──────┼────────┼─────────┼────────┼──────┼────────────┼──────┼──────────┤
│ api      │ 0  │ 0.34.0  │ cluster │ 7642 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ api      │ 3  │ 0.34.0  │ cluster │ 7655 │ online │ 0       │ 18m    │ 0.1% │ 109.0 MB   │ root │ disabled │
│ athena   │ 2  │ 0.34.0  │ cluster │ 7654 │ online │ 0       │ 18m    │ 0%   │ 109.2 MB   │ root │ disabled │
│ athena   │ 1  │ 0.34.0  │ cluster │ 7647 │ online │ 0       │ 18m    │ 0%   │ 108.5 MB   │ root │ disabled │
│ chronos  │ 4  │ 0.34.0  │ cluster │ 7671 │ online │ 0       │ 18m    │ 0%   │ 108.4 MB   │ root │ disabled │
│ chronos  │ 6  │ 0.34.0  │ cluster │ 7695 │ online │ 0       │ 18m    │ 0.1% │ 108.9 MB   │ root │ disabled │
│ hermes   │ 13 │ 0.34.0  │ cluster │ 7800 │ online │ 0       │ 18m    │ 0%   │ 107.9 MB   │ root │ disabled │
│ hermes   │ 12 │ 0.34.0  │ cluster │ 7785 │ online │ 0       │ 18m    │ 0.1% │ 108.9 MB   │ root │ disabled │
│ hyperion │ 5  │ 0.34.0  │ cluster │ 7676 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ hyperion │ 7  │ 0.34.0  │ cluster │ 7713 │ online │ 0       │ 18m    │ 0.1% │ 109.0 MB   │ root │ disabled │
│ mercury  │ 8  │ 0.34.0  │ cluster │ 7727 │ online │ 0       │ 18m    │ 0.1% │ 108.2 MB   │ root │ disabled │
│ mercury  │ 10 │ 0.34.0  │ cluster │ 7755 │ online │ 0       │ 18m    │ 0.1% │ 109.2 MB   │ root │ disabled │
│ vulcan   │ 9  │ 0.34.0  │ cluster │ 7741 │ online │ 0       │ 18m    │ 0%   │ 109.2 MB   │ root │ disabled │
│ vulcan   │ 11 │ 0.34.0  │ cluster │ 7769 │ online │ 0       │ 18m    │ 0.1% │ 109.2 MB   │ root │ disabled │
└──────────┴────┴─────────┴─────────┴──────┴────────┴─────────┴────────┴──────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        |     at Function.Module.runMain (internal/modules/cjs/loader.js:697:11)
PM2        |     at startup (internal/bootstrap/node.js:201:19)
PM2        |     at bootstrapNodeJSCore (internal/bootstrap/node.js:516:3)
PM2        |   errno: 'EADDRINUSE',
PM2        |   code: 'EADDRINUSE',
PM2        |   syscall: 'listen',
PM2        |   address: '::',
PM2        |   port: 3006 }
PM2        | Error: listen EADDRINUSE :::3006
PM2        |     at Server.setupListenHandle [as _listen2] (net.js:1346:14)
PM2        |     at listenInCluster (net.js:1387:12)
PM2        |     at Server.listen (net.js:1475:7)
PM2        |     at Function.listen (/root/spectrum/node_modules/express/lib/application.js:618:24)
PM2        |     at module.exports._reactLoadable2.default.preloadAll.then (/root/spectrum/build-hyperion/webpack:/hyperion/index.js:208:7)
PM2        |     at <anonymous>
PM2        |     at process._tickCallback (internal/process/next_tick.js:182:7)
PM2        |     at Function.Module.runMain (internal/modules/cjs/loader.js:697:11)
PM2        |     at startup (internal/bootstrap/node.js:201:19)
PM2        |     at bootstrapNodeJSCore (internal/bootstrap/node.js:516:3)
PM2        | 2019-03-31T12:41:54: PM2 log: pm2 has been killed by signal, dumping process list before exit...

Here is the process.json file, I tested with cluster and cluster_mode stated on https://github.com/Unitech/pm2/issues/3946#issuecomment-434338770 .

{
  "apps": [
    {
      "name": "api",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:api"
    },
    {
      "name": "athena",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:athena"
    },
    {
      "name": "chronos",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:chronos"
    },
    {
      "name": "hyperion",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start"
    },
    {
      "name": "mercury",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:mercury"
    },
    {
      "name": "vulcan",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:vulcan"
    },
    {
      "name": "hermes",
      "script": "yarn",
      "exec_mode":"cluster_mode",
      "instances":2,
      "args": "start:hermes"
    }
  ]
}

In sum:

  • Removing exec_mode does not work.
  • Changing cluster to cluster_mode does not work.
  • Program exits with EADDRINUSE, status is still online.
  • Program does not restart on error, status is still online.

All related issues are either open or does not resolve this issue,

stale
Source
picture entrptaher
👍4

All 10 comments

Tahy, thanks for posting this issue here and hope we can solve it once for all. If anyone can help, give any points, would mean a lot to us.

shtefcs picture shtefcs on 31 Mar 2019

To add more to above post, it's really weird the error goes to /root/.pm2/pm2.log instead of the error log. Maybe the program checks for port and manually rejects with EADDRINUSE.

entrptaher picture entrptaher on 7 Apr 2019

I tried

  • updating pm2 version latest 3.5.0.
  • downgrading pm2 version to 3.0.0, 2.5.0.

The process exits but does not restart either. Really fraustrating experience.

Also:

  • pm2 scale api 2 or any other scaling simply fails.

image

Without cross-env (and yarn/npm script together)

However, today I tried a different script as well, previously the start script had cross-env and it was failing (as mentioned above). But when I tried without cross-env, the program ran properly.

I used this code to run,

 NODE_ENV=production pm2 start build-api/main.js --name=api --instances=2

We need cross-env though to maintain env across multiple processes properly. But it's a good lead.

image

image

I am not sure how it will behave without cross-env, only can say after experiments.

entrptaher picture entrptaher on 10 Apr 2019

I was also having this issue and I was able to fix it by running pm2 delete on the apps that were having the EADDRINUSE issue and recreating them with pm2 start. I am not sure why this would fix it, but my only guess is that something saved in the pm2 env/config was causing an issue. For context, this issue started happening after I upgraded some packages on my ubuntu machine.

I confirmed that this permanently fixed the issue by running pm2 save and rebooting. The clustered app started and ran as expected without errors.

bam-tbf picture bam-tbf on 17 Apr 2019
👀1 🎉1

I have the same issue for very long time on all my projects.

my pm2.config.js:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */

  apps: [
    {
      name: 'media-app',
      script: 'npm',
      args: "run serve",
      exec_mode: 'cluster_mode',
      exec_interpreter: 'node',
      wait_ready: true,
      listen_timeout: 10000,
      instances: 2,
      max_memory_restart: '200M',
      max_restarts: 2, // if restarts this many times in min_uptime time, app is considered errored and will not start
      min_uptime: 5000,
      watch: false,
      log_date_format: 'YYYY-MM-DD HH:mm Z',
      env: {
        NODE_ENV: 'production',
      },
    },
  ],
};

in git hooks/post-receive I have this:

pm2 startOrReload pm2.config.js --update-env

It fails consistently after each reload.

What I do to fix it after each reaload:

pm2 delete media-app
pm2 startOrReload pm2.config.js --update-env

This issue deservers a fix after years of not functioning properly.

zsolt-dev picture zsolt-dev on 2 Jul 2019

I was getting this same error when I tried to cluster my app that was already running with PM2. I deleted the running instance and started it again using the cluster param and no more errors!

munroe7 picture munroe7 on 21 Aug 2019

I am having the same problem. I even tried uninstalling and reinstalling pm2, deleting all the old files .pm2 folder but i still get EADDRINUSE error in cluster mode.
Here is my config file

{
"apps": [{
"script": "npm",
"args": "start",
"name": "backend",
"instances": "max",
"exec_mode": "cluster_mode",
"env": {
"NODE_ENV": "production"
},
"merge-logs": true,
"error_file": "/var/log/pm2/backend-err.log",
"out_file": "/var/log/pm2/backend-out.log"
}]
}

matharuajay picture matharuajay on 26 Sep 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] picture stale[bot] on 24 May 2020

Is this fixed in newer versions? Experienced it on a production website after restarting pm2 with an ecosystem file and experienced a bunch of downtime

wdifruscio picture wdifruscio on 27 May 2020
👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] picture stale[bot] on 26 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  ·  3Comments
shaunwarman picture shaunwarman  ·  3Comments
rangercyh picture rangercyh  ·  4Comments
psparago picture psparago  ·  3Comments
phra picture phra  ·  3Comments