Stryker: Feature request - Add coverage analysis for Jest

[Tobino]

Summary

Is it possible to add coverage analysis, to provide a better experience on mutation testing with jest ?
For now, stryker-jest-runner create mutants on every single part of code, even if statement or branch are not reach.

I would like to add my contribution to stryker, to help bigger application to use stryker, without to add mutation on unreachable statement during initial test run.

Stryker config

module.exports = function(config) {
  config.set({
    testRunner: "jest",
    mutator: "javascript",
    transpilers: [],
    reporter: ["html", "clear-text", "progress"],
    coverageAnalysis: "all",
    mutate: ["src/**/*.js"]
  });
};

Stryker environment

    "jest": "^23.2.0",
    "stryker": "^0.24.1",
    "stryker-api": "^0.17.2",
    "stryker-html-reporter": "^0.14.2",
    "stryker-javascript-mutator": "^0.7.2",
    "stryker-jest-runner": "^0.7.0"

Your Environment

| software | version(s)
| ---------------- | -------
| node | 10.4
| npm | 6.1
| Operating System | OSX

[nicojs]

@Tobino thanks for creating this issue.

We've chatted about this on Gitter, I'll include the explanation I gave there here as a reference for anyone interested.

You've shown interest to help us crack this nut. Thanks a lot in advance! I've created a branch ("965-jest-coverage-analysis") and added one integration test ('Integration test for Strykers Jest runner should be able to retrieve code coverage results'). It uses a reference jest test project located in the testResources: packages/stryker-jest-runner/testResources/exampleInstrumentedProject. This reference project represents the state that a project will be in after Stryker intrumented the code, when coverageAnalysis is "all", at the start of the initial test run (using istanbul). The tests expects there to be a coverage result (which there isn't at the moment). I've also added a line of code that retrieves the __coverage__ object from global scope in the JestTestRunner (packages\stryker-jest-runner\src\JestTestRunner.ts). The __coverage__ variable is missing from global scope because it is cleaned after the tests by Jest.

This is the problem we're currently facing. This should be a good starting place to help you on you're journey.

Happy hunting!

My remark from Gitter (as a reference):

@Tobino setups where the _test runner does the transpiling_ (i.e. jest, or karma with preprocessors) will not work. Istanbul simply does not support anything else but javascript.

If you configure a transpiler in the transpilers array, it can work if the transpiler _produces sourcemaps_ (i.e. only stryker-typescript a.t.m.). Stryker will instrument the _resulting js code_ from the transpiler output and use the source maps to correlate the mutants. To support webpack, we need the stryker-webpack-transpiler to support producing sourcemaps as well (which can be done, simply didn't have time yet).

If you than use a test runner that supports reporting of coverage result (i.e. stryker-karma-runner, stryker-mocha-runner or stryker-jasmine-runner), coverageAnalysis: "all" can be supported. This is the main problem with jest, I'll come back to that later.

If you than also use a test framework that supports _test filtering_ (i.e. stryker-mocha-framework or stryker-jasmine) you can support the coverageAnalysis: "perTest".

This is a complex setup, and for the mid-term we're thinking of a new way to instrumenting code with babel7 (still in beta), as it supports parsing of javascript, typescript and flow _natively_ (pretty awesome). That way we can instrument the _source code_ and _directy_ correlate them to the mutants. No need for source map support by transpilers, and we can support testrunners that transpile code themselfs (i.e. jest and karma with preprocessors).

Even with the new setup, we need to have coverage reporting from the jest runner (be it istanbul coverage, or future stryker instrumented coverage). So if you want to help, you're welcome! We need a way for the stryker-jest-runner to report the global object: __coverage__. The problem there is that jest works really hard to _not_ make you use global variables. If you want to help, please create an issue for it over at stryker-jest-runner and I'll explain further what is needed.

[Tobino]

Hello,

It seems that Jest deep clone __coverage__ on each testSuite, and push this result on coverage field. I guess coverage is the result of each worker from Jest, and I don't know if Stryker is able to use this format directly or a merge of coverage with istanbul-lib-coverage ?

PR: https://github.com/stryker-mutator/stryker/pull/1003

[nicojs]

Thanks for your update, I will take a look next week. Just to be clear, is Jest itself also instrumenting the code?

[Tobino]

I think Jest use istanbulJs to instrument the code, but I am not sure if it's the right answer.

[nicojs]

Yeah, that isn't what we want. We now use istanbul to instrument the code our selves before the initial test run (based on the coverageAnalysis setting). However, this could change in the future our own instrumentation.

We're actually looking for a way to mark a global variable as "don't touch this one" in jest. Do you know if that's possible? I have looked at this problem before, but usually was looking on the global scope. I haven't thought to look at testResults array, so that's still hopeful...

[Tobino]

I think testResult.coverageis untouch by jest, and then coverage report is generate using this field. Maybe there is a chance to improve stryker.

[simondel]

I believe this is implemented with this PR: https://github.com/stryker-mutator/stryker/pull/1235

I'm closing this issue for now. If the problem is still there, let us know!

[Djaler]

@simondel I don't think that this PR is related to the issue. For example, I have a lot of tests for my Vue component. And even with --findRelatedTests Jest will run all of this tests. So, coverage analysis is still required

[PFadel]

I believe this is implemented with this PR: #1235

I'm closing this issue for now. If the problem is still there, let us know!

I agree with @Djaler , this issue is related for Coverage analysis for Jest, this PR is for fixing JestTestRunner. I believe this should be reopened, or if we're wrong the Readme at least should be updated here.