Stryker: Mutation testing successful, yet results to 0%

[rvy-brillio]

Hi team, first of all thanks for creating/maintaining this awesome tool.

Question

I've setup and run the tool in my local and, even though the run is successful, the mutation result is 0%. Almost all of the code is not covered.

Stryker environment

+-- "@stryker-mutator/core": "^2.1.0",
+-- "@stryker-mutator/html-reporter": "^2.1.0",
+-- "@stryker-mutator/jasmine-framework": "^2.1.0",
+-- "@stryker-mutator/jasmine-runner": "^2.1.0",
+-- "@stryker-mutator/javascript-mutator": "^2.1.0"

Test runner:

+-- "jasmine": "^2.8.0"

Other than this, I'm also using jasmine's additional plugins:

+-- "jasmine-console-reporter": "^2.0.1",
+-- "jasmine-node": "^1.14.5",
+-- "jasmine-supertest": "^1.0.0"

Additional context

stryker.conf.js:

module.exports = config => {
  config.set({
    mutate: ['src/**/*.js', '!src/models/**/*.js'],
    mutator: 'javascript',
    packageManager: 'npm',
    reporters: ['html', 'clear-text', 'progress'],
    testFramework: 'jasmine',
    testRunner: 'jasmine',
    coverageAnalysis: 'perTest',
    jasmineConfigFile: 'test/jasmine-unit.json',
    files: ['src/**/*.js', 'test/**']
  });
};

Output of tests:

I'm wondering what I'm doing wrong, since the unit test cases are successfully executed. When I separately run the test cases using npm test, the coverage is very high (>70%).

Please let me know if you need additional details on this.

Thanks.

[kmdrGroch]

it's indeed very strange. Is your repository public so we could try finding an issue?

[rvy-brillio]

Thanks for the quick response @kmdrGroch. The repository is private, but I can share the stryker.log generated from the run. Will that help in any manner?

[kmdrGroch]

It might help. Just give us whatever you can :)

[simondel]

Very strange!

Are you using Git? If so, please remove the files option from the config because we can use Git to copy the required files (your manual setup may miss some things)

As a workaround for now, you could run with coverage analysis "off".

[rvy-brillio]

Sorry for the late replies (I was traveling without network).

@kmdrGroch here's the stryker log file: stryker.log

Hi @simondel, I'm using Git, and I have a file that git cannot pick up because it is ignored in .gitignore. The file that I require to be part of the test run is my test env file. So, I explicitly used files option. (The test env file is used to setup tests' run)

About the workaround for running with coverage analysis "off", the mutation suggests an ETC (I suppose this is estimated time of completion) to be around 25 hours, which is where I stopped it after 8-10 mutations.

[rvy-brillio]

To avoid missing any file (as suggested by simondel), I also tried:

files: ['**', '!node_modules/**']

which also resulted in 0% coverage. (same as above)

[simondel]

25 hours is not that bad in your case :P The initial testrun took 1 minute and 45 seconds. If every run is as slow as that and you don't run in parallel, it'll take almost 10 days to run. However, due to parallelisation and running in a 'hot' environment, you may end up closer to the 25 hour mark.

Coverage analysis will improve this a lot of course (if we figure out why it's broken in your case).

[kmdrGroch]

I don't see any issue on the first glance :/
@rvy-brillio maybe could you prepare some 'lighter' version of your app? (for example, remove files which should not be visible for public / create a simple app with the same configuration you have not)?
I think @simondel would agree it could help. Unless you have another idea Simon?

[rvy-brillio]

@simondel I'll try to do this. I'll get back if the execution completes.

I had the same plan @kmdrGroch. Let me get that app ready and test it accordingly. It might take some time, so please feel free to close this. I can open another issue once the app is ready (and if the issue still persist).

[nicojs]

Yes, a lighter version of the app will be the fastest way to get this up-and running 👍

[kmdrGroch]

Is this issue still present?

[simondel]

Hi! I'm closing this issue for now. If it still persists with Stryker version 4, please open a new issue. Thanks!