Vscode-jest: All tests re-run on every change

[TheAlmightyBob]

Environment

1. node -v: 8.11.1

2. npm -v: 5.6.0

3. Operating system: macOS High Sierra

Steps to Reproduce

Save a change to any specific test.

Expected Behavior

Only the changed file's tests are re-run, leading to quick feedback on test success/failure as a result of the change.

Actual Behavior

All tests in the entire project are re-run, potentially taking a very long time before finding out if the one modified test passes/fails.

This appears to be a regression due to #325 . As one who didn't have problems with the previous behavior, it would be very nice if this could at least be made an optional setting for those who need it.

[orta]

Agree, it should only do the changed tests - happy to take a PR reverting that change 👍

[Jamus123]

I'm on NPM 6.0, Node 8.9.4 and macOS High Sierra and the same thing is happening to me.

[seanpoulter]

Thanks for reporting the issues @TheAlmightyBob and @Jamus123. Can you debug the extension to see what values should be sent to the runner in this call? Calling this.runner.start(true, true) is like calling jest --watch --watchAll.

It should be quick to clone the repo, install the dependencies, open the folder in VS Code, and start debugging with the "Launch Extension" configuration. If you set a breakpoint on that call to start the runner and step into the call you'll end up looking at the Runner from jest-editor-support.

[Jamus123]

@seanpoulter I'm not sure what the values should be, is that something I should be aware of? When I start it up both values passed into this.runner.start are false, false.

[seanpoulter]

Right right, that makes sense when it starts. The default behaviour is to run everything once unless you enable a setting, then subsequent runs are started with --watch or --watchAll (as true, false or false, true).

What is it on subsequent runs? Is it truthy?

[seanpoulter]

Also ... HUGE thanks for debugging this. 🎉

[Jamus123]

Ok, so I Start the debugger and then it calls this.runner.start with false, false as previously mentioned. It then begins to do the Initial full test run which looks to complete and then I hit the breakpoint again where this.runner.start is called with true, true and then it enters watch mode where it looks as though it's running the tests again.

Any changes to a test file after it's entered watch mode will run the entire test suite again and there are no further calls to this.runner.start from what I can tell. I tried mangling a test and then getting it to run correctly and each time, like I was saying, the full test suite would run.

What I realized when I look at it now is that because in the JestProcess.ts file this.runner.start has the same variable(this.watchMode) passed into both parameters there isn't ever a way for this.runner.start to be called with false, true or true, false.

[Jamus123]

Looks as though #325 was kind of a ham-fisted fix.

[TheAlmightyBob]

Yep. Thanks @Jamus123 for investigating.

I'm not sure if there would be any difference between false, true and true, true? "watch" is just a subset of "watch all", no? Or is true, true actually running the tests for modified files _twice_?

I'm not quite sure how widespread the original issue was, or why #325 was the fix for it. I guess it's simply that Jest v23's "watch" mode is broken? (I'm still on v22)

Should #325 simply be reverted and #316 considered a Jest defect? Or would it be worth adding "watch all" as an optional mode to vscode-jest?

[seanpoulter]

The --watch option will use Git or Mercurial to detect the changed files. This is the default mode. Using --watchAll will rerun everything when any file changes. The long game is to:

• add an option in the settings that allows folks who aren't using Git or Mercurial to use watch mode in some capacity
• detect when jest --watch fails because Git or Mercurial have not been configured and start jest --watch with a notification to the user

I'm unfamiliar with the root cause of #325, but it should be relatively straight forward to investigate.