Stryker: All tests run, no mutants killed. What am I doing wrong?

[jaimefps]

Question

Stryker seems to have installed correctly and runs fine with "stryker run" command.
However, none of the mutants die, even though we have about 85% coverage on our codebase.
All the mutants surviving seems highly unlikely in a suite of 256 tests, so I'd appreciate a second pair of eyes on my configs to see if I'm missing anything.

Stryker environment

    "@stryker-mutator/core": "^2.0.2",
    "@stryker-mutator/html-reporter": "^2.0.2",
    "@stryker-mutator/jest-runner": "^2.0.2",
    "@stryker-mutator/typescript": "^2.0.2",

    "jest": "^23.6.0",

Additional context

stryker config:

const jestConfig = require('./jest.config.js');

module.exports = function(config) {
  config.set({
    mutate: ['src/**/*.ts?(x)', 'src/**/*.js?(x)', '!src/**/*@(.test|.spec|Spec).ts?(x)'],
    mutator: 'typescript',
    testRunner: 'jest',
    reporters: ['progress', 'clear-text', 'html'],
    coverageAnalysis: 'off',
    jest: {
      projectType: 'custom',
      config: jestConfig,
    },
  });
};

jest config:

module.exports = {
  collectCoverage: true,
  collectCoverageFrom: ['**/*.js*', '**/*.ts*', '!node_modules/**', '!__tests__/**'],
  coverageDirectory: 'reports/coverage',
  coverageReporters: ['json-summary', 'lcov', 'text'],
  coverageThreshold: {
    global: {
      branches: 80,
      functions: 60,
      lines: 85,
      statements: 85,
    },
  },
  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json', 'node'],
  moduleNameMapper: {
    '\\.(css)$': '<rootDir>/node_modules/jest-css-modules',
  },
  reporters: [
    'default',
    [
      'jest-junit',
      {
        suiteName: 'jest tests',
        outputDirectory: './reports',
        outputName: './junit.xml',
        classNameTemplate: '{classname}-{title}',
        titleTemplate: '{classname}-{title}',
        ancestorSeparator: ' › ',
        usePathForSuiteName: 'true',
      },
    ],
  ],
  roots: ['<rootDir>/__tests__'],
  setupTestFrameworkScriptFile: '<rootDir>/__tests__/setup.js',
  testRegex: '__tests__/.*.(spec|test).(js|ts)x?$',
  testResultsProcessor: './node_modules/jest-html-reporter',
  transform: {
    '\\.(css|less)$': 'identity-obj-proxy',
    '^.+\\.jsx?$': 'babel-jest',
    '^.+\\.tsx?$': 'ts-jest',
  },
};

FInal output:

12:33:36 (10605) INFO ConfigReader Using stryker.conf.js in the current working directory.
12:33:36 (10605) WARN InputFileResolver Globbing expression "src/**/*@(.test|.spec|Spec).ts?(x)" did not result in any files.
12:33:36 (10605) INFO InputFileResolver Found 62 of 168 file(s) to be mutated.
12:33:36 (10605) INFO InitialTestExecutor Starting initial test run. This may take a while.
12:34:23 (10605) INFO InitialTestExecutor Initial test run succeeded. Ran 256 tests in 47 seconds (net 2791 ms, overhead 43191 ms).
12:34:24 (10605) INFO MutatorFacade 2294 Mutant(s) generated
12:34:24 (10605) INFO SandboxPool Creating 8 test runners (based on CPU count)
Mutation testing  [==================================================] 100% (ETC n/a) 2294/2294 tested (2294 survived)

Is it just not finding my tests?

12:33:36 (10605) WARN InputFileResolver Globbing expression "src/**/*@(.test|.spec|Spec).ts?(x)" did not result in any files.

I thought the point of that code was to NOT mutate tests. We keep tests in a __tests__ directory outside of src/. Is that a problem for Stryker?

[kmdrGroch]

it shouldn't be problem for stryker to work on /src and __tests__. Although it keeps bugging me that

12:33:36 (10605) WARN InputFileResolver Globbing expression "src/**/*@(.test|.spec|Spec).ts?(x)" did not result in any files.

Is your project open source so we could work with it or private?

[jaimefps]

It's private. If I have time later on, I'll try to create a barebones version of my code, and if the problem persists, I'll create a public repo to share. Thanks for responding.

[kmdrGroch]

thanks, @jaimefps, sorry for bothering and making you wait so much 🙏

[jaimefps]

@kmdrGroch No worries, this wasn't an urgent request.
Mutation testing still seems to be a somewhat new practice to the JS community.
I wan't to do this more as a surprise for my team when I get the chance.
I'm very excited of what mutation testing means for the quality of unit tests.

[kmdrGroch]

@jaimefps I am not good at mutation testing myself. My adventure with it began like 2 years ago, when I was in work and one person told he is doing mutation tests for his PHP code. I was like hmm, I don't even know how to write unit tests xD. After that i learnt how to do it and I was like super impressed by both tests and mutation tests. (mutation did not only improved my tests but also I could remove tons of unnecessary lines, chunks from my code!). So good luck in working on this, and also thanks for helping us improve this tool!

[simondel]

Hi @jaimefps. If you're still having this issue, could you let us know? For now I'll assume that your issue has been resolved.

[tivac]

I'm seeing this issue, and I do have a public repo I can link:

https://github.com/tivac/aws-sig/compare/stryker

If I manually apply any of the mutations I see plenty of jest tests failing, so I know that part should be working correctly. Is there something weird about this setup preventing it from working?