Mocha: scoped bail and multiple test files

[Xotic750]

What I would like to do is execute mocha sometimes with the bail option on and sometimes off, something like this

mocha --check-leaks -u bdd -t 140000 -b -R tap tests/*.js

and

mocha --check-leaks -u bdd -t 140000 -R tap tests/*.js

The test directory has numerous test files. There is one file, let's call it array.js that I do not want to bail on when mocha is run with -b, and so I went and set the option in-file like so

    describe('Native array', function () {
        this.bail(false);

Now with -b, even if one of the tests in this file (array.js) fail then it continues to run all the tests in this file (array.js). That's great and that's what I want. However, if a test did fail in this file (array.js) then I was expecting the rest of the files to execute, but they don't.

So I guess I'm looking for advice on how to achieve this, and if it is not currently possible then this is an option that I would like to have available.

Many thanks.

[dasilvacontin]

While you don't execute mocha two times? One with bail and the files you want to have bail, and one without bail and the rest of the tests.

[dasilvacontin]

If you could explain the use-case, that would be great, because it looks like a pretty particular request.

[boneskull]

Unless this is widely clamored for, you can execute Mocha 2x using some sort of task automator, npm script, or Makefile

[Xotic750]

I don't want to execute 600,000+ tests twice across multiple environments (50+). In node environments (and some others) I do want basic environment tests to bail but in browsers I do not, but I still want to show the results of the tests and it is more than 1 file (I just made the example simple), e.g. Array, String, Boolean, Arguments etc. Sure I can spend time and effort using a number of other tools and utilities and eventually get something like what I have described, but the best place and way (IMO) would be for a configuration in mocha like I questioned about.

So the answer is: "There is currently no way of configuring this in mocha"

And the feature request is: "Would not be impossible to implement (difficulty?) but denied unless more people ask for it"

[dasilvacontin]

Can't you add a comment/tag in top of the files and grep/filter them for the first run (bail run), and invert grep for the second run?

[boneskull]

@Xotic750 Hmm, I may have misread the request; sorry about that. Now I'm not sure if this is a bug or not.

[boneskull]

@dasilvacontin There are numerous workarounds; I don't think that's in dispute

[boneskull]

Can anyone confirm the intended behavior of --bail and whether or not @Xotic750 's _other_ test files should be running?

[dasilvacontin]

The logic is that a suite/test with bail doesn't run if there has been any failure before. I think that it even counts hooks, not sure right now.

[dasilvacontin]

From the docs:

-b, --bail, bail after first test failure
Only interested in the first exception? use --bail !

[Xotic750]

In a similar manner

-t, --timeout set test-case timeout in milliseconds [2000]

but

Suite-level timeouts may be applied to entire test "suites", or disabled via this.timeout(0). This will be inherited by all nested suites and test-cases that do not override the value

[dasilvacontin]

Current code: if you set bail to false, bail's logic will be disregarded during those suites/tests. When it's active again, logic turns on, and since the failures count is not zero, runner will abort.

[dasilvacontin]

-b, --bail, bail after first test failure can be ambiguous as a description because it doesn't mention when/where is the test failure produced.

[dasilvacontin]

If we don't discuss/change behavior, I'd vote for improving the definition, at least.

[Xotic750]

Ok, so the answer still stands: "There is currently no way of configuring this in mocha"

But, what sort of effort would be involved (I have no familiarity with the mocha code) in having some option to have the behaviour described?

[dasilvacontin]

Either changing the code's logic regarding bail, or using another counter for failures that gets reset when the bail flag is changed or smth.

Besides that, it would be a breaking change, and there may be people that disagree with the new behavior.

[Xotic750]

If the behaviour was agreed as a wanted feature, which would be the least breaking change? My guess would be to add a new global option, something like -B (not currently used) and then use another counter for use with this.bail(false) when -B was specified. That way the original behaviour would be preserved while offering an extended feature? I still have no idea as to how much code change would be involved?

[FabianTe]

I would really like to see this feature! This currently blocks me from using the bail option, since I have a similar project to the one of @Xotic750

Right would have to run each describe seperately (by having them separated into one describe per file) and then running mocha n times for n files.

I'd love to see this be implemented!

[artyomtrityak]

Yeah, i have thousand of tests and interested at 1st fail _per describe_. Would be awesome to get this done. So as described here https://github.com/mochajs/mocha/issues/2894 when 1st test in describe fails, all describe should fail but other describes should continue.

Workaround is running each test separately but this is a nightmare - thousands of tests...

Another workaround i found:

In setup file:

const FAILED_TESTS = {};
// Skip test if first test from folder failed
beforeEach(function() {
  if (FAILED_TESTS[this.currentTest.file]) {
    this.skip();
  }
});

afterEach(function() {
  if (this.currentTest.state === "failed") {
    FAILED_TESTS[this.currentTest.file] = true;
  }
});