Mocha: Mocha exits with 0 exit code with failing tests

[chrisleck]

Node: 7.2.1
Mocha: 3.2.0
MacOSX: Sierra (10.12.3)
Cmdline: mocha my_test.js

Seeing mocha exit with a 0 exit code after failing a test in an after each hook. I traced it down to this code: https://github.com/mochajs/mocha/blob/9ae6a85b6ba50ea415affc88456d469d6915a157/bin/_mocha#L425

The function is called with the correct exit code, but then stream.write('', done); doesn't call the callback before the process terminates.

[chrisleck]

I added some tracing statement to the code as follows and see the following output:

function exit (code) {
  // flush output for Node.js Windows pipe bug
  // https://github.com/joyent/node/issues/6247 is just one bug example
  // https://github.com/visionmedia/mocha/issues/333 has a good discussion
  function done () {
    draining--;
    console.log(`Draining down to ${draining}`);
    if (draining <= 0) {
      process.exit(Math.min(code, 255));
    }
  }

  process.on('exit', function(realExitCode) {
    console.log(`Process is exiting with ${realExitCode} but should exit with ${code}.`);
  });

  var draining = 0;
  var streams = [process.stdout, process.stderr];

  streams.forEach(function (stream) {
    // submit empty write request and wait for completion
    draining += 1;
    console.log(`Draining up to ${draining}`);
    stream.write('', done);
  });

  console.log('Starting extra call to done().');
  done();
  console.log('Extra call to done() finished.');
}

Results in:

Draining up to 1
Draining up to 2
Starting extra call to done().
Draining down to 1
Extra call to done() finished.
Process is exiting with 0 but should exit with 1.

[chrisleck]

Proposal: eagerly set the process.exitCode property and eliminate the extraneous call to done. Will send a PR.

[chrisleck]

FWIW, I'm also able to reproduce this in a Linux container running on CoreOS with the same Node and Mocha versions and same test.

[percyhanna]

Running into this issue on our CI running on Travis. Tests fail, but Mocha exits with 0, so the build does not fail.

[dasilvacontin]

The Eagerly set exitCode fix has been released in v3.4.2! 🎉

Sorry it took long!

[olastor]

I'm using mocha 5.0.1 and it does not exit with an error code if tests fail... I had to use a hacky workaround for this to make it work in travis.

yarn test | tee temp.txt && ! grep -q -oP "\d+\sfailing" temp.txt && rm temp.txt

(yarn test executes mocha)

[thescientist13]

@olastor
I am seeing this behavior as well, also using 5.0.1

[sclarson]

I'm still seeing this happen in mocha 3.5.3 and 5.1.1

[davesag]

I'm seeing this behaviour in mocha 5.2 but only with my integration tests, which use mocha-prepare-promise to set up and tear down connections to the server and database.

I'm refactoring my integration tests now to eliminate the need for this package.

[ORESoftware]

my guess is that the exit codes are somehow tied to the reporter that is being used. try switching out the reporter and see if that changes the exit codes

[sclarson]

I saw this on node 8.9 and it resolved itself going to node 8.10

[nwesterman]

This has been a continuous problem for us. Updating something (like node 8.9->8.10) will seem to fix it and then it spontaneously appears again. I'm seeing this happen with 3.5.3 and 6.1.4 on numerous Node versions 8.10, 8.11, 8.12, 8.16, 10.0, 10.16 (haven't found a working combination yet). Observed on both linux and windows OS. In addition to our gitlab-ci pipeline not failing, I can visually see the output not completing in the console log. Tests lines at the end don't print nor do the error details. Running just the failing suite does cause the exit code to print, but running the entire suite does not. Possibly a timing or issue with high test counts?

[..abreviated output...]
      _findOrSave
        √ does't call save if a result is returned
    parseAndSaveJson
      8) handles user
       √ Parses hex
Done in 5.67s.
$ echo $?
0

[plroebuck]

@nwesterman, this issue is closed. If you're having problems, open a new issue with an MVCE. Since you said output was abbreviated, can't tell whether you only SNIP'd from top, or between "Parses hex" and "Done in 5.67s".