Nodemon: Nodemon Fails to Wait for Process Exit when using Docker + Inspector

[darkobits]

Created per @remy's request in this issue.

It seems that when restarting a process, Nodemon isn't waiting for the old process to completely exit before starting a new one, but:

• _only_ when in a Docker container and
• _only_ when the inspector/debug mode is active.

This issue can be intermittent in applications without a shutdown handler. But, having an application that implements a listener for Nodemon's SIGUSR2 signal that takes even a second or so to complete can be enough to cause this issue to happen reliably.

The presence of --inspect _does_ cause the new process to immediately crash because its trying to open a debug server on the same port that is still in use by the previous process' debug server, but this merely a side-effect of this issue.

If running Nodemon in a Docker container without --inspect, Nodemon correctly waits for the old process to exit, but when --inspect is used inside the container, Nodemon _does not_ wait for the process to exit, leading to an immediate crash.

This can be reproduced with Nodemon 1.18.7 and the following:

$ node -v
v10.13.0

$ docker -v
Docker version 18.09.0, build 4d60db4

$ docker-compose -v
docker-compose version 1.23.1, build b02f1306

Here is a repo that replicates this issue: https://github.com/darkobits/nodemon-restart-issue.

[ImTheDeveloper]

Just started to see this issue when I upgraded to latest versions of nodemon too.

Ubuntu 18
nodemon 1.18.6 global
node 10.7.0
docker 18.06.1-ce build e68fc7a

Running this with vscode debugging. Worked perfectly fine previously until I did some package upgrades after the event-stream vulnerability.

NODE_ENV=development nodemon --legacy-watch --delay 2000ms --inspect-brk=0.0.0.0:9222 --nolazy index.js",

[daniele-orlando]

Same issue, but only on Linux. No problem on Mac with Docker For Mac.
On my Linux machine the port is used by docker-proxy, an executable running as root user.

[remy]

I think this _might_ be solved with the next release going up at the moment - assuming it passes, it'll land in [email protected].

@darkobits can you test this once it's up?

[denic]

@remy Works for me. Thanks!

[ImTheDeveloper]

Great thank you for this fix!

[andrewhughson]

I'm still seeing the issue with 1.18.8

[daniele-orlando]

v1.18.8

Same here, less frequent but still there.

[darkobits]

I've updated the demo repo referenced in the original post to nodemon 1.18.9, and this issue is still occurring.

[darkobits]

@remy, thoughts regarding leaving this closed vs. re-opening?

[justinpage]

I'm seeing intermittent results. Sometimes when I save a file change it restarts correctly.
Other times it errors out the following message:

crumb_1     | Starting inspector on 0.0.0.0:9229 failed: address already in use
crumb_1     | [nodemon] app crashed - waiting for file changes before starting...
crumb_1     | [nodemon] restarting due to changes...
crumb_1     | [nodemon] starting `node --inspect=0.0.0.0:9229 ./index.js`

What I'm forced to do, as shown above, is save again which triggers nodemon to restart.
The second time around works correctly.

• version: 1.18.9
• command ./node_modules/.bin/nodemon --inspect=0.0.0.0:9229 ./index.js

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[justinpage]

Still applicable.

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[daniele-orlando]

Still persists

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[anodynos]

I still get this error with [email protected] & latest node@10

[d-wagner]

I also have this problem with [email protected] and 1.8.10

[aaronup]

We are also having this exact issue with [email protected]

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[abdohossam5]

Still persists

[yjwong]

We're also having this issue with [email protected].

Reading the nodemon code, it seems to be the case that nodemon does wait for the child process to exit. The new process is only started in the exit handler for the child process (see https://github.com/remy/nodemon/blob/master/lib/monitor/run.js#L202 - restart calls run again).

When trying to exec('ps -ef', (err, stdout) => console.log(stdout)) after the spawn call here, it appears that the old child process is either in the Z (zombie)/X (dead) state when this happens

So far, I've hit a dead end trying to debug this problem, but hopefully it's of help to someone.

[nomcopter]

This happens to me completely outside of docker. Repro repo here: https://github.com/nomcopter/nodemon-no-wait-repro

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[nomcopter]

Issue persists

[alemarcha]

It's not working here either

[nomcopter]

Repro repo that fails outside of docker on OS X @remy https://github.com/nomcopter/nodemon-no-wait-repro

[stale[bot]]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[nomcopter]

Issue persists

[remy]

@nomcopter I've (finally) added a config to get stalebot to back off completely. I'll check out your repo on replication today (sorry for the lack of attention!)

[remy]

I'm unable to replicate with OS X - see screenshot.

@nomcopter I'm using

• [email protected]
• nodemon@latest
• OS X @ 10.13.6 (High Sierra)

[ChristianIvicevic]

I am unable to replicate this issue as well. Using node 10.15.3, 12.1.0 with latest nodemon and OS X 10.14.4. Will try later on Windows because I have the same issue in a different setting on Windows.

My own application only listens to the signals on Windows when Ctrl+C-ing, but does not recognize the signals when nodemon restarts. On my Mac it's actually the other way round: restarts are recognized followed by proper cleanup but Ctrl+C just kills everything outright without any cleanup.

[ChristianIvicevic]

Quick update: instead of process.once I had to use process.on in my own app to track a SIGINT via Ctrl+C on macOS which is rather confusing. Will report on the Windows behaviour for the repro repo later this evening.

[ChristianIvicevic]

Alright, I just tried it on Windows and the repro repo does not work as intended. Nodemon just kills the process without any cleanup at all and I think this issue isn't actually related to Nodemon but Node and Windows in general

[mdobroggg]

I'm having the same issue. I think it has to do with using an async call in process.on to exit. It seems like nodemon is acting on the original kill signal rather than the one re-thrown from process.on

[nomcopter]

@remy welp. Will try again locally when I have a chance, thanks for taking a look.

[mdobroggg]

I think adding a small delay between kill and restart would be enough to fix this issue. If I cause nodemon to restart once I will get the error, but if I cause a restart after that failure, it will succeed

[marcelkottmann]

Hello, today I worked on this and I think I have understood what is happening here.

First of all it seems, that this is only a Linux related issue (from the comments I see docker and Ubuntu). I had this problem since several weeks with Ubuntu 18.

The problem is, that even if you think nodemon starts only one child process, nodemon actually starts two processes: one sh-process AND one node-process (run.js:97). The child.on('exit')-callback in run.js only gets events for the sh-Process (because it's the parent of the node process).
The node documentation explicitly says, that on Linux platform terminating the shell command does not necessarily kill all the shell child commands:

On Linux, child processes of child processes will not be terminated when attempting to kill their parent. This is likely to happen when running a new process in a shell or with the use of the shell option of ChildProcess.

(see https://nodejs.org/api/child_process.html#child_process_subprocess_kill_signal)
Although here the native kill command is executed in line run.js:368, it seems it behaves the same way.

Explicitly waiting for all other kids to be terminated before restarting the application solves the problem.

See my PR #1579

[daniele-orlando]

@pepe79 Thank you very much for having worked on the issue and the extensive explanation. Very interesting and I've learned something new about Node.

[pavles6]

Still persits, running on docker:

[nodemon] Internal watch failed: ENOSPC: System limit for number of file watchers reached, watch '/sys/kernel/slab/:A-0000040/cgroup/pde_opener(667:console-setup.service)/cpu_partial'

[aymather]

I've been having this issue for a while and my workaround might provide some information.

Right before I run npm run dev I run the following command: lsof -ti tcp:5000 | xargs kill which basically kills whatever processes are on the port.

However, even when I manually kill the port, I sometimes still get the same :::5000 address in use error. In that case I just kill the port again and it will usually work on the second try.

I'm just confused why even when I manually kill the port, I still get the error, and makes me think that maybe nodemon is the one blocking the port?

I'm running node v8.11.3 on MacOS 10.14.3.

[Remzi1993]

I'm unable to replicate with OS X - see screenshot.

@nomcopter I'm using

• [email protected]
• nodemon@latest
• OS X @ 10.13.6 (High Sierra)

Because the issue is on Linux! I hope that delaying it could fix this problem. It seems like @pepe79 has fixed this issue, see PR #1579

@remy I hope you could take a look at his repo/commit.

[towerofnix]

now that #1632 (which contains 0e6ba3ce6d3ad19f1348e547d15136b3c789732b) has been merged & nodemon v2 has been released, should this issue be closed?

[remy]

👍

[marcelkottmann]

Hello @remy , sorry to say that, but this issue is back again since version 2.0.2. It was fixed in 2.0.0 and 2.0.1. I just had the chance to upgrade to version 2.0.2 and this issue is back again. I looked at 47dfb8be3ff06184793a55d32db4b6171fa2993f and I guess the new line
child.kill(signal);
should only be called after all sub-processes have already been terminated. Moving this line behind the waitForSubProcesses-function call, should fix this.
Otherwise the exit signal of child will restart the process tooo early (again).