Nodemon: Child process not receiving SIGTERM

[stoically]

• nodemon -v: 1.13.2
• node -v: 8.7.0 (nvm: 0.33.5)
• Operating system/terminal environment: Ubuntu 16.04.3 / 4.4.0-104-generic / bash
• Command you ran: nodemon test.js

Hi,

thanks for your work on nodemon!

Expected behaviour

The child process should receive SIGTERM and/or trigger the process.on exit event. Code that relies on these events is not triggered otherwise - e.g. killing child processes that it spawned itself.

Actual behaviour

Does not send SIGTERM or triggers process on exit event

Steps to reproduce

Create file test.js

process.on('SIGTERM', () => {
  console.log('SIGTERM');
});
process.on('exit', () => {
  console.log('exit');
});

function loop () {
  console.log('looping');
  setTimeout(loop, 1000);
}
loop();

Run it nodemon test.js
Make a change in the file

---

node: v8.7.0
nodemon: 1.13.2
command: /home/user/.nvm/versions/node/v8.7.0/bin/node /home/user/.nvm/versions/node/v8.7.0/bin/nodemon test.js --dump
cwd: /home/user/Code/test
OS: linux x64
--------------
{ run: false,
  system: { cwd: '/home/user/Code/test' },
  required: false,   
  dirs: [ '/home/user/Code/test' ],
  timeout: 1000,
  options:
   { dump: true,
     ignore:
      [ '.git',
        '.nyc_output',
        '.sass-cache',
        'bower_components',
        'coverage',
        'node_modules',
        re: /\.git|\.nyc_output|\.sass\-cache|bower_components|coverage|node_modules/ ],
     watch: [ '*.*', re: /.*\..*/ ],
     ignoreRoot:
      [ '.git',
        '.nyc_output',
        '.sass-cache',
        'bower_components',
        'coverage',
        'node_modules' ],
     restartable: 'rs',
     colours: true,
     execMap: { py: 'python', rb: 'ruby' },
     stdin: true,
     runOnChangeOnly: false,
     verbose: false,
     signal: 'SIGUSR2',
     stdout: true,
     watchOptions: {},
     execOptions:
      { script: 'test.js',
        exec: 'node',
        args: [],
        scriptPosition: 0,
        nodeArgs: undefined,
        execArgs: [],
        ext: 'js,json',
        env: {} },
     monitor:
      [ '*.*',
        '!.git',
        '!.nyc_output',
        '!.sass-cache',
        '!bower_components',
        '!coverage', 
        '!node_modules' ] },
  load: [Function],  
  reset: [Function: reset],
  lastStarted: 0,
  loaded: [],
  watchInterval: null,
  signal: 'SIGUSR2', 
  command:
   { raw: { executable: 'node', args: [ 'test.js' ] },
     string: 'node test.js' } }

---

One workaround for cleaning up child processes of the child that get not killed is using the nodemon.json (see full doc with nodemon --help config)

{
  "events": {
    "restart": "pkill -f partial/path/to/child/of/child"
  }
}

[stoically]

After a bit of digging I found that it's not possible to trigger the process on exit event using a signal.

Also, it's possible to configure nodemon to send the SIGTERM signal by putting

{
  "signal": "SIGTERM"
}

into the nodemon.json - although that won't help with code that relies on the process on exit event, obviously.

I would suggest to send SIGTERM by default too though. Unless I miss something that would make that a problem?

[remy]

I ran test.js using node, and ctrl+c to stop it, and there's also no exit notification:

Nodemon uses a signal that's least used to kill the child process, but you if you want to use SIGTERM you can (either through config or the CLI). I don't think this is a bug, and I think you've got a solution for what you're after. 👍

[stoically]

Just to clarify what's actually the problem: under some circumstances child processes stay alive after the "file change restart" triggers. The child processes die when doing a ctrl+c tho. I couldn't reproduce with a simple nodejs child - in this case it works just fine, but in case you want to reproduce it yourself:

Disclaimer: granax downloads and starts Tor

npm install granax

test.js:

const granax = require('granax');
const tor = granax();
tor.process.stdout.once('data', (data) => {
  console.log(data.toString());
})

const { exec } = require('child_process');
exec('ps aux|grep granax', (error, stdout, stderr) => {
  console.log(stdout);
});

function loop () {
  console.log('looping');
  setTimeout(loop, 5000);
}
loop();

Now if you make a changes in the test.js, you'll see the tor processes stacking up. If you hit ctrl+c it kills all processes.

[remy]

Can you retest with the latest nodemon release - there's been a slew of recent fixes I've released, and one in particular is for linux environments.

[stoically]

I've tested with the latest 1.13.3

[remy]

Okay, so I can see what's going on here.

• Nodemon spawns a process (node) which is restarted when code changes.
• This particular example code spawns a new process under the node process (tor.real)
• When a file change is detected, nodemon sends a SIGUSR2 to the full process group (individually)
• USR2 is normally good enough to kill a process, but tor.real will ignore the signal, so it remains running

Here's the tor.real process ignoring the SIGUSR2 signal (in all likelihood, it's using the signal for something else):

Solution: run with nodemon --signal SIGTERM (or SIGINT seems also work for tor.real).

The reason nodemon uses SIGUSR2 is so that the nodemon logic _knows_ that the restart was triggered by nodemon. Honestly, I can't remember exactly what it's SIGUSR2 instead of SIGTERM - the project is well over 7 years old so I'm a little hazy over the original decision.

I'll add this to the docs either way.

[stoically]

Thanks for looking into it. Can confirm that it works with node --signal SIGTERM, cool!

Just for my understanding: is there a reason you don't send SIGTERM per default to the childs? After a little research it seems that USR2 is defined as "user defined signal" - so it could mean everything depending on the process (e.g. unicorn reexecutes the binary). SIGTERM/INT on the other hand is defined as quickly shutting down the process - wouldn't it make sense to make that the default?

[remy]

Yeah, like I said, there was a reason, but it was such a long time ago the thought through justification escapes me.

I'd like to look at this more, but I don't want to dig in too deep this weekend (should be with my kids). I have a gut feeling that a SIGTERM could be worth swapping to, but I've not thought through the full impact.

[stoically]

Alright, yeah, somehow I missed the part explaining the why, sorry. Thanks for your time so far and a nice weekend to you and your kids! :)