Mocha: moral of the story: quote your globs on the command-line

[sverraest]

I'm using Mocha.js as part of a CI pipeline.
Recently the failing tests don't actually cause the npm test step to fail the build because it looks like Mocha.js doesn't exit in an failure state.

ERROR: {"error":{"message":"Cannot read property 'where' of undefined","stack":"TypeError: Cannot read property 'where' of undefined\n at Object.undeleted
info: Done.
npm info posttest [email protected]
npm info ok

This is quite annoying as it allows commits with failing tests to still be build in our CI pipeline.

[gurdiga]

Hi @sverraest! 🙂

I’m trying to reproduce this and I seem not to be able to get there. 😳

Here is my setup:

    ~/tmp/mocha ɀ  mocha --version
2.5.3
    ~/tmp/mocha ɀ  cat test.js 
it('runs tests', function() {
  assert(false); // this will cause an error because there is no `assert` in the scope
});
    ~/tmp/mocha ɀ  grep "test" package.json 
    "test": "mocha test.js"
    ~/tmp/mocha ɀ  npm test

> [email protected] test /Users/vladgurdiga/tmp/mocha
> mocha test.js



  1) runs tests

  0 passing (15ms)
  1 failing

  1)  runs tests:
     ReferenceError: assert is not defined
      at Context.<anonymous> (test.js:2:3)



npm ERR! Test failed.  See above for more details.
    ~/tmp/mocha ɀ  echo $?
1
    ~/tmp/mocha ɀ  

So it looks like a runtime error causes Mocha to fail. 🤔

I’m wondering if you could share a bit of your test code, just to get a bit more context around this issue and maybe be able to reproduce and find a solution. 🤓

[boneskull]

@sverraest Can you provide more context for the failure?

[unkillbob]

I've also been observing this recently, originally I thought the issue was with protractor or gulp-protractor but I notice the issue only seems to be when the bail: true option is configured for mocha.

[Munter]

@unkillbob can you provide a failing test and a command line that triggers this?

[unkillbob]

@Munter after making that observation I tried to do exactly that but couldn't reproduce it. Either its intermittent or it could be something else in the stack that's to blame - if I ever do manage to reproduce it reliably will be sure to post it here.

[ibratoev]

I hit the same issue and spent some time to extract a reproducible case. It is a bit more complex so I put it in a quick repo: https://github.com/ibratoev/mocha-repro .
@boneskull, does this confirms the issue?

[boneskull]

Well, the problem is a little more complicated than this, but I'm not sure why. running npm test vs executing mocha directly results in different behavior.

[boneskull]

My previous comment (which I deleted) went down the wrong path. After more tinkering, this has to do with the shell.

Basically, tests/**/*.js means something different in say, bash (by default)--than in glob, e.g.:

$ ls tests/**/*.js
tests/tests/bar.js

vs

$ node -e "console.log(require('glob').sync('tests/**/*.js'))"
[ 'tests/tests/bar.js', 'tests/tests/tests2/foo.js' ]

So, yeah. There's nothing Mocha can really do about this. You have to change your shell's behavior, or provide a string value for the glob. In Bash 4.x, you can:

$ shopt -s globstar
$ ls tests/**/*.js
tests/tests/bar.js      tests/tests/tests2/foo.js

I don't know what the equivalent is in Bash 3.x or zsh (though I'd like to know!). Note that MacOS X ships with 3.x; use something like Homebrew to upgrade.

Otherwise, this will work (and is the strategy Mocha uses in its Makefile for cross-env compatibility):

$ mocha 'tests/**/*.js'


  Test
    1) Fake test


  0 passing (8ms)
  1 failing

[ibratoev]

Thanks for the insight @boneskull. My scenario really has nothing to do with mocha but more with how npm works. Some related links:

• https://github.com/npm/npm/issues/10481
• https://medium.com/@jakubsynowiec/you-should-always-quote-your-globs-in-npm-scripts-621887a2a784#.txfhphxqo