Nodemon: Ctrl + C ends up with continual EADDRINUSE errors after nodemon tries to restart expressJS

[dschinkel]

I do not know if this is a bug but suspect it is?

I'm using node v14.4.0 so latest. nodemon ^2.0.4

Don't know what the behavior should be but I figured if I Ctrl + C, nodemon and my expressJS server should stop and no longer hog whatever port expressJS is running on

Error: bind EADDRINUSE null:3000
    at listenOnMasterHandle (net.js:1380:18)
    at rr (internal/cluster/child.js:132:12)
    at Worker.<anonymous> (internal/cluster/child.js:99:7)
    at process.onInternalMessage (internal/cluster/utils.js:47:8)
    at process.emit (events.js:327:22)
    at emit (internal/child_process.js:906:12)
    at processTicksAndRejections (internal/process/task_queues.js:85:21)
Emitted 'error' event on Server instance at:
    at listenOnMasterHandle (net.js:1381:21)
    at rr (internal/cluster/child.js:132:12)
    [... lines matching original stack trace ...]
    at processTicksAndRejections (internal/process/task_queues.js:85:21) {
  errno: -48,
  code: 'EADDRINUSE',
  syscall: 'bind',
  address: null,
  port: 3000
}

The only way I can manually kill it and temporarily move on is to use pkill node.lsof -i :3000 -t | xargs kill` doesn't work.

This happens after I Ctrl + C and then try to run yarn dev again

My Scripts:

"start": "PORT=3000 nodemon --trace-warnings dist/server/server.js",
"build": "NODE_ENV=production webpack -p --env=prod --watch && yarn compile-server && yarn start",
"dev": "NODE_ENV=development yarn lint && yarn copyData && yarn compile-server && yarn start & webpack-dev-server",
"compile-server": "tsc -b ./src/server",
"copyData": "mkdir -p dist/shared/data && cp src/shared/data/companies.json dist/shared/data && cp src/shared/data/countries.json dist/shared/data",

here it is running the first time I fire it up after say I reboot my machine and there are no processes running yet:

If I then Ctrl + C then run yarn dev OR if nodemon or something errors and nodemone trys to restart itself, I get:

[remy]

Can you replicate with pared down code? What's the minimal case?

[MonarchChakri]

I guess this issue is similar to this (https://github.com/remy/nodemon/issues/1734).
@remy
I can work on this once you are done merging my PR.

[BibekStha]

I can confirm the error too.

Most of the times, when the development server restarts after I save a file, I get the error.

node version: 14.6.0
nodemon version: 2.0.4

I have been using ss -tnlp | grep node to find the process id and then kill it using kill -9 <pid>

I searched a lot about this. Found issue #1734 but it was closed pointing that the issue was solved. Apparently, it's not.

[remy]

The result is similar the source issue is not.

Provide pared down code (as per every issue template) and it can be investigated (though apparently only by me, I must have some super powers or something…)

[jylauril]

I'm suspecting that this might not actually a problem with nodemon, but maybe your server implementation.

To me, without knowing more about the implementation, it sounds like you might not handling process shutdowns gracefully by actually closing the http server connections before the process exits. So when nodemon brings the process back up, the old connections haven't timed out yet and the port is still bound to the old http server.

There are several ways to get around this by either using some existing libraries (ExpressJs suggests for example this library: https://github.com/godaddy/terminus) for this type of process handling, or by implementing your own version of it.

At the simplest you could at least do this:

const app = express();
// ..
// instrument your server here
// ..

// app.listen returns a NodeJS http.Server instance
const httpServer = app.listen(myPort);

// this is a standard signal sent when process is asked to shut down
// nodemon also sends this signal unless otherwise configured
process.on('SIGTERM', => httpServer.close());
// this signal is sent when you ctrl+c
process.on('SIGINT', => httpServer.close());

I've seen cases where people just forcefully process.exit() and think that it also magically shuts down all connections, but sadly that is not the case. For truly graceful shutdown one should always close any possible connections, which includes express' http server, mongo, redis, and/or other active connections. Also long running timers can also prevent the process from gracefully exiting, but generally those shouldn't affect any connections staying open.

[remy]

Folks reporting on this issue, as @jylauril says:

I've seen cases where people just forcefully process.exit() and think that it also magically shuts down all connections, but sadly that is not the case. For truly graceful shutdown one should always close any possible connections, which includes express' http server, mongo, redis, and/or other active connections. Also long running timers can also prevent the process from gracefully exiting, but generally those shouldn't affect any connections staying open.

Can those reporting add this logic to your code and report back.

If indeed it does not leave the express process still running, I can add this added to the FAQ (because goodness knows how many times this similar issue is reported…).

If I don't hear back - I'll assume the issue is magically fixed. And I'm not really a believer of magic…

[BibekStha]

I tried adding the code provided by @jylauril. It didn't work for me.
But from what he said (also in #1734), I found a stackoverflow post which solved the problem temporarily.

[jylauril]

@BibekStha just to sanity-check: do you have the signal modified in your nodemon config by any chance?
If yes, then that's the signal you have to listen to with the process.on code. And if you don't have it changed, you could try adding that config option as "signal": "SIGTERM" to tell it to alway send the SIGTERM when restarting and then the code I provided should definitely catch it.

And if that doesn't work, you could modify the code that I provided to console log instead of httpServer.close() to verify that you are actually getting those signals properly.

[BibekStha]

I have not modified my nodemon config.

Currently, I combined your code with a couple other I found in the previously mentioned SO post and it seems to be working fine.

    const httpServer = app.listen(port, host);

    [`exit`, `SIGINT`, `SIGUSR1`, `SIGUSR2`, `uncaughtException`, `SIGTERM`].forEach((event) => {
      process.on(event, () => {
        console.log('Process event type: ', event);
        httpServer.close()
        process.exit();
      })
    })

Any feedback on this code block?

[jylauril]

@BibekStha Yeah, just one thing: the httpServer.close is actually asynchronous so you shouldn't call process.exit immediately after it. Otherwise you're bound to have the exact same problem of exiting the process before the connection is closed 😄

Technically if all your connections and servers are closed, your process should exit on its own, but sometimes it can be a bit cumbersome to manage everything. In this case the original problem was the web server not being able to re-bind itself after the restart so if the process doesn't exit on its own and you don't want to deal with other connections then it should be fine to process.exit.

Just edit the code to do this instead:

httpServer.close(() => process.exit())

---

In my node projects I usually make a little class called ServerState that extends from EventEmitter and it has a shutdown method that just emits shutdown event on itself. The constructor to that class just binds the signal listeners that then calls the shutdown method on itself. Then every time I register a new server or connection, I just do:
in my express code:

const httpServer = app.listen(port);
serverState.once('shutdown', () => httpServer.close())

in my mongoose initialization code:

await mongoose.connect(options);
serverState.once('shutdown', () => mongoose.connection.close())

This way once all the connections are closed, the process exits on its own and I don't have to worry about anything leaving hanging.

[BibekStha]

That helps. Thanks a lot @jylauril!!
I was looking for a solution to this problem for a long time. I now have the solution and also understand a little more about nodemon and nodejs.

@remy As you were going to, it would be great to add the code in FAQ, as many people are facing this issue.
Here is the modified code as per @jylauril, for your reference.

    // app.listen returns a NodeJS http.Server instance
    const httpServer = app.listen(port, host);

    // clean exit the server and node process when one of these events occur
    [`exit`, `SIGINT`, `SIGUSR1`, `SIGUSR2`, `uncaughtException`, `SIGTERM`].forEach((event) => {
      process.on(event, () => {
        httpServer.close(() => process.exit())
      })
    })

Please modify the code block with your expertise, thanks.

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3