Mocha: Mocha can't run tests twice programmatically

[aDu]

Programmatically, I cannot run a Mocha test twice.

I.e. I can't do mocha.run() twice.

I looked at a similar issue (#995), however, the solutions are not optimal and/or not work.

For example, deleting the require cache is not a solution for me, as I need to use the require cache.

Is there a way to mocha.run() twice?

Thanks in advance!

[aDu]

Mmmm ><. I guess I will stick with my solution of spawning a child process for each mocha run; but I wanted to avoid this.

[KingRial]

I had the same problem.

Reading the similar issue #736, it seems it's all about cleaning the "require.cache" of the previously loaded spec's file

Here my current solution:

launchTests(){
  let _sTestDir = 'tests/specs';
  let _oMocha = new Mocha();
  fs.readdirSync( _sTestDir ).filter( function( oFile ){
    return ( oFile.substr( -3 ) === '.js' );
  }).forEach(function( sFile ){
    let _sPathSpec = path.join( path.resolve(), _sTestDir, sFile );
    // Resetting caches to be able to launch tests multiple times...
   delete require.cache[ _sPathSpec ];
   // Adding test
    _oMocha.addFile( _sPathSpec );
  });
  _oMocha.run();
}

[boneskull]

does the "retry" functionality not work?

[aDu]

No, I don't think so for the context of rerunning an entire test suite again. @KingRial's solution seems like it would work.

My suggestion is to update the mocha code to allow mocha.run() to be executed more than once (or a similar solution).

[josiahruddell]

I ran into the same problem. Why was this issue closed? Making a new instance of Mocha, and adding files should behave deterministically without modifying the require cache. Seems like a bug.

[aDu]

@josiahruddell Good point, I agree.

[jcjolley]

+1

[ORESoftware]

+1

[rawb]

+1

[jamesmcging]

+1

[colinbendell]

I've tested the selective delete of the require.cache with hundreds of parallel reruns. See my comment on #955. (this basically removes the test file from require.cache immediately after it is evaluated). The question is, why do you need a durable require.cache?

[nicojs]

For people interested, we've built the re-run functionality in the Stryker plugin for mocha: https://github.com/stryker-mutator/stryker/blob/930a7c39952e09a7d3f913fd9423bf78f608ec17/packages/mocha-runner/src/MochaTestRunner.ts#L48-L68

The problem we now face is that we _want to rerun mocha, without cleaning require cash or reloading the files_. We want to call it "hot reload". So rerun tests, without changing source code or test code.

Basically, what we want to do is this:

const mocha = new Mocha(...);
mocha.addFile(...);
mocha.run(() => {
  mocha.run() => {
    console.log('ran twice');
  });
});

But right now, the second run always errors with: "Cannot read property 'call' of undefined" (which isn't helpful...)

The problem is that mocha.run has side effects. Namely: it cleans the test functions

here: https://github.com/mochajs/mocha/blob/0dacd1fb0067e40f8567653f828f677022e4fb89/lib/runner.js#L895-L897

and here: https://github.com/mochajs/mocha/blob/0dacd1fb0067e40f8567653f828f677022e4fb89/lib/suite.js#L455-L481

I think mocha.run isn't the place to start listening for the suite end event and clean up after. This should be done in the mocha cli file itself, right?

I'm willing to create the PR if the maintainers agree with this.

We could also add an option for this to keep the API backward compatible.

[plroebuck]

This is just _one_ aspect.. there are several others required to _really_ hope to reuse a Mocha instance.
Have something in mind, but post 6.0 release.

[plroebuck]

Of note, the Mocha-6.0 release API includes Mocha#unloadFiles. It's one step towards eventual mocha reuse...

[nicojs]

Actually... no...

To delete the require cache correctly, not only the test files but also the _source files_ should be cleared. Mocha doesn't keep track of those files. This is what we need to do in Stryker right now (the purgeFiles method takes care of it).

Just to make sure we don't confuse 2 things: we actually want to reuse the same instance of mocha _without clearing any file cache_. Should I create a separate issue for that?

[boneskull]

essentially, you'd need to ensure Mocha's inner state was completely reset. given there are bits of state everywhere, it's kind of a tall order at this point without some significant refactors. granted, having a single source of truth for state would be a great refactor, but...nontrivial.

[nicojs]

you'd need to ensure Mocha's inner state was completely reset. given there are bits of state everywhere, it's kind of a tall order at this point

Could you give me a list of things you can think about at the top of your head, with pointers on how to solve it? We should be able to create some kind of state object and pass that along. We can make it opt-in, so functionality remains the same if you don't want to use it.

[plroebuck]

Could you give me a list of things you can think about at the top of your head, with pointers on how to solve it?

While ongoing efforts are made to payoff some of this technical debt for reusability, there are higher priority issues to address (e.g., correctness, robustness). Would also need to redesign many of the automated tests to now run themselves twice. There can be no "opt-in" -- it would have to be baked-in.

[boneskull]

@nicojs Yes, what @plroebuck said, and that I can't explain Mocha's "state" strategy without saying "look at the code", because there's state everywhere.

[boneskull]

That said, I'm happy to entertain ideas/PRs to solve the "state" question, I just don't think our maintainer team has the resources to embark on such a ...quest.

[nicojs]

I just don't think our maintainer team has the resources to embark on such a ...quest.

Haha

Thanks for the clear responses. If I find a couple of hours in the coming weeks, I might take a look. One might say, I will embark on a quest 🏇

[DiogoMCampos]

I just don't think our maintainer team has the resources to embark on such a ...quest.

Haha

Thanks for the clear responses. If I find a couple of hours in the coming weeks, I might take a look. One might say, I will embark on a quest horse_racing

I don't feel comfortable on undertaking a nontrivial _quest_ without knowing the codebase, but making myself available to help if needed. This would solve a current major issue I have.

[boneskull]

I hate to pull the redux card, but something like redux might be a good solution here.

[plroebuck]

I don't feel comfortable on undertaking a nontrivial _quest_ without knowing the codebase, but making myself available to help if needed.

This seems appropriate somehow.

[wKich]

Hello everyone. I'll try to write tool on top of mocha, to run heavy e2e tests in separate workers for each browser for example. Current solutions (like mocha-parallel-tests) is not prefect for me, it don't allow share selenium webdriver connection between tests. So I want to start pool of workers, that each other initialize webdriver and all test cases, then master thread send message to worker for start specific test case and worker send result back on finish.
Mocha#unloadFiles, seems could help, but unload cache and reinitialize tests for each run it's little bit overwhelmed.

[wKich]

@boneskull, can you clarify, why cleanReferences remove only fn property from each hook instead of clear entire array?

[nicojs]

Ok, I've investigated the issues a bit and I want to propose the following changes. They are designed to be 100% backward compatible and minor while still allowing for the new feature. @boneskull @plroebuck do you agree with these changes? I'd be willing to prepare the PR.

🔀 cleanReferences

I want to add a feature to allow references not to be cleared. Let's call it autoDispose. By default it would be true (non-breaking change). But if you use mocha programmatically you are allowed to set it to false, you would have to call dispose on the mocha instance later if you still want to dispose.

🔢 "bits of state"

For the _bits of state_ issue, I'm pretty sure that the state is situated in Suite, Test and Runnable. I would like to add a reset method in each, which resets the state. It would be called from the Mocha class, whenever an _nth_ test run is started (where n > 1).

This way the way of working within mocha remains mostly the same. No big changes needed in the respective classes, responsibility, etc.

🏡 Housekeeping

I would also want to add 2 new validations:

1. Whenever mocha's run method is called an "object is already disposed" error will be thrown whenever the mocha's instance is already disposed (either by autoDispose, or by a manual dispose).
2. Whenever mocha's run method is called while a run is already in progress a "mocha run is already in progress" error is thrown.

[boneskull]

sounds reasonable to me. you’ll also want to ensure there’s no eventemitter leaks happening.

[nicojs]

The PR is finished. @boneskull would you care to take a look?

Great integration tests you guys have! I broke mocha's watch functionality apparently. Never would have known without those awesome tests!

I choose in the end NOT to call it autoDispose, rather i've called it cleanReferencesAfterRun. My reasoning is that dispose() on the mocha instance should also remove any uncaughtException listeners remaining on the process, but I don't want to remove the listeners directly after a test run, since the previous runner should still handle them (it was one of the integration tests). That's why I instead only clean all references after each run. You can still dispose everything at once using the dispose method on a mocha instance.