Stryker: Implement mutation switching 👽🔀

[nicojs]

Intro

_Mutation switching_ is a technique that can be used to improve the performance of mutation testing. It is used in Stryker.NET as well as Stryker4s. We believe this has a lot of benefits for JavaScript/TypeScript as well.

In short, with mutation switching all mutants will be placed into the source code at once. Only one mutant can be active at the same time. Here is a small example:

/* 1*/ // original
/* 2*/ function add(a, b) {
/* 3*/   return a + b;
/* 4*/ }
/* 5*/ 
/* 6*/ // Might be instrumented to:
/* 7*/ function add(a, b) {
/* 8*/   if (global.activeMutant === 0) {
/* 9*/      // block mutation, empty function
/*10*/   } else {
/*11*/     return global.activeMutant === 1? a - b: a+b;
/*12*/   }
/*13*/ }

Potential performance improvements

Although JavaScript is not a compiled language, there is still huge performance improvements to be had.

• Less I/O. This can be a big performance overhead, since not only the mutated file has to be cleared from node's require cache, but the entire object graph that is dependent on that file, which in practice means clearing the entire cache (100's of files for running Stryker on Stryker). For karma it means a reload between test runs with sometimes MB's of files to reload.
• One-time transpiling. A lot of projects depend on some kind of transpiler/bundler, also to run the tests. Think of TypeScript, Webpack, Vue CLI, Angular CLI, Jest-ts, ts-node, etc. They will all have to transpile only once, instead of once per mutant.
• Hot reload. Test runners won't have to clear their test suite between tests (there is a caveat here regarding static code, see hurdles).
• Test coverage analysis. Since we're instrumenting the code, it's pretty much free to add mutation coverage instrumentation in the code. For example, line 11 could be mutated to: return global.activeMutant === 1 ? a - b : (global__coverMutant__(1), a + b);. The __coverMutant__ here is an example of how we can count coverage analysis. Mutants that aren't covered, don't need to be tested. If we add the context of which test is running at that point in time, we can only run the tests that cover a particular mutant. We currently have a very complex coverage analysis in place, which we can totally get rid of once we have this new test coverage analysis system in place. It will work for a lot more use cases, no matter which transpiler or bundler or even minified you use. However, we will need to change all test runners to report this new mutation coverage statistic. _There is also complication here for static code, see hurdles_.

Mutation switch instrumenter PoC.

I've created a Proof of Concept of mutation switching. It works for a couple of our mutations and I'm confident we can add them all. It has a lot of benefits:

• It supports both JS and TS with one AST (thanks to babeljs).
• It supports JS, TS, Vue and HTML file formats (thanks to angular-html-parser).
• Support mutant coverage instrumentation.

Hurdles

There are some hurdles to overcome.

• We need to make sure that mutated code can be transpiled by the supported transpilers out there. Webpack, Babel, TypeScript, and friends will have to produce JavaScript to run. This is not trivial since we are definitely introducing type errors into the source code.
• We will still need to type check each mutant. A.t.m. this is the job of the transpiler. We will need to change that, as the transpiler will have to produce code no matter what. I would suggest adding a new plugin type for that "TypeChecker". We might want to not make it a public API at first, to save time and maintenance.
• Test runners will have to support at least running the tests. Reporting the mutation coverage and hot reload are a nice to have, although Stryker might actually slow down without it (due to larger file sizes).
• There's also the concept of _static_ code. Code that is executed once when imported and then never again. If we mutate in that part, we would need to detect it and turn off hot reload for those scenarios. Detecting static code can be done by reporting mutant coverage. Mutants that were covered before a test runs can be considered "static".
• Our current way of test filtering in Mocha will no longer work in combination with hot reload.

Rethinking Transpilers

We currently have a Transpiler API. It is responsible for transpiling the code for the initial test run, as well as transpiling each mutant. Stryker takes care to load the transpiler and keep it alive, so the transpiler can use in-memory --watch functionality if it can.

Examples of transpilers:

• Webpack transpiler
• Babeltranspiler
• TypeScript transpiler

Specific transpiler plugins are _no longer needed_ when using mutation switching. We only need to transpiler once (with all mutants in the code), so there is no need to painstakingly transpile each mutant. The amount of maintenance is simply not worth the effort to keep this transpiler API, if only to run it once.

Instead, we should allow a user to configure a "build command" (name pending). It can be tsc -b for example, or webpack --configFile webpack.conf.js. This build command is then executed inside the sandbox before the initial test run starts. This sandbox is reused for the mutant test runs.

When copying files to the sandbox, we might need to change them slightly. Take this tsconfig file for example:

{
  "extends": "../../tsconfig.settings.json",
  "references": [ 
    { "path": "../api/tsconfig.src.json" },
    { "path": "../util/tsconfig.src.json" }
  ]
}

Simply picking this file up and dropping it in the sandbox won't work. Assuming the sandbox is copied 2 directories deep, we need to preprocess it to be:

{
  "extends": "../../../../tsconfig.settings.json",
  "references": [ 
    { "path": "../../../api/tsconfig.src.json" },
    { "path": "../../../util/tsconfig.src.json" }
  ]
}

The goal of a transpiler

Migration scenario

I think a small step-by-step migration is out of the question, as approximately half of our code will have to be rewritten, or completely scrapped. Our public API will also need to be changed in a couple of places too.

If we want to keep feature parity with the current features, we will at least need support for transpiling using babel, webpack, typescript, jest, ts-node, karma-webpack, mocha-webpack, vue cli, angular CLI and react scripts and test runner support for karma, jasmine, mocha, and jest (I think we can drop WCT).

A high-level overview of the new way of working of a mutation test run:

1. Read config
2. Instrument the code
   
   
   
   1. Parse
   
   
   2. Generate mutants
   
   
   3. Place mutants
   
   
   4. Print the code in memory
   
   
3. Transpile the code (once)
4. Fill one sandbox, run tests once for the control run (initial test run)
5. Check mutants using the configured "check" plugins
6. Test the remaining mutants in a sandbox

Investigation progress

There's a lot of stuff we need to figure out. We'll have to investigate how all features are possible. _Note:_ there is a lot of stuff "nice to have" here, so we shouldn't block implementation.

• [x] [Source code instrumentation](https://github.com/nicojs/mutation-switch-instrumenter#mutation-switch-instrumenter)
  
  
  
  • [x] Source code mutation
  
  
  • [x] Mutation coverage instrumentation
  
  
  • [x] Support for vue, ts and JS
  
  
• [x] TypeCheckers
  
  
  
  • [x] TypeScript. _we should be able to converts parts of the current TypeScript transpiler to a type checker_
  
  
• [x] Transpilers _prefix every ts/js file with // @ts-nocheck_ <ul> <li>[x] TypeScript _we can use <a href="https://github.com/microsoft/TypeScript/issues/29651#issuecomment-616113744">this work around</a> to transpile using typescript_</li> <li>[x] Babel _doesn't perform type checking_</li> <li>[x] Angular CLI. _as long as we don't mutate decorators, we should be able to prefix every ts/js file with <code>md5-3372ddd0141866b8ad22b7f53bde2dce</code> (works from TS3.7)_</li> <li>[x] Web pack: Turn off type checking for typescript related plugins. _prefix every ts/js file with// @ts-nocheck_


• [x] VueCLI _ Turn off type checking for typescript related plugins_

[x] Test runners

• [x] Command: How do we activate a specific mutant when using the command test runner? _will be best effort using env variable_


• [x] Mocha
  
  
  
  
  
  • [x] Report mutation coverage _just reporting the global object_
  
  
  
  • 🤷‍♂️ [nice to have] Hot reload (will be implemented in https://github.com/mochajs/mocha/pull/4234)
  
  
  
  • 🤷‍♂️ [nice to have as long as we don't have hot reload] Support test filtering (investigated in https://github.com/mochajs/mocha/issues/4183), although maybe we can use grep
  
  
  


• [x] Jest
  
  
  
  
  
  • [x] Turn off type checking for typescript code _can not be done by simply removing ts-jest, however, we can use the typescript workaround_
  
  
  
  • 🤷‍♂️ [nice to have, we don't support coverage a.t.m.] Report mutation coverage (we can use workaround proposed in https://github.com/facebook/jest/issues/7421)
  
  
  
  • 🤷‍♂️ [nice to have, we don't support this a.t.m.] Run individual tests
  
  
  
  • 🤷‍♂️ [nice to have] Hot reload
  
  
  


• [x] Karma
  
  
  
  
  
  • [x] Turn off type checking for typescript code in karma plugins _we can use the typescript workaround_
  
  
  
  • [x] Report mutation coverage _can be done with custom middleware and framework? See https://github.com/stryker-mutator/stryker/issues/1514#issuecomment-625971578_
  
  
  
  • [x] Run individual tests (can be done with the current setup, but won't support hot reloading)
  
  
  
  • 🤷‍♂️ [nice to have] Hot reload (probably needs custom framework plugin and middleware plugin)
  
  
  


• [x] Jasmine node
  
  
  
  
  
  • [x] Report mutation coverage _just reporting the global object_
  
  
  
  • [x] Run individual tests (current setup will work)
  
  
  
  • 🤷‍♂️ [nice to have] Hot reload
  
  
  

Implementation plan

Implementation should be done in steps. To make it easier on ourselves, we should allow us to break parts of Stryker in the meantime. I suggest therefore to work in a feature branch. We'll be implementing mutation switching on that branch.

In the meantime, our master branch will keep getting security patches, but we want to keep the functional changes to a minimum.

Question: Do we want to keep Dependabot during this time? I think it is better to point Dependabot to the feature branch instead of master.

The plan:

Preparation

• [x] Add e2e tests for missing use cases. Be sure to add code that will result in type errors when mutation switching is implemented.
  
  
  
  • [x] Karma with karma-webpack and typescript (we can copy a small sample from mutation-testing-elements) (#2183)
  
  
  • [x] Jest with ts (#2177)
  
  
  • [x] React with jest and tsx (#2184 )
  
  
  • [x] Vue-cli with mocha
  
  
  • [x] Vue-cli with jest?
  
  
  • [x] mocha using ts-node
  
  
  • 🤷‍♂️ A setup with Aurelia ?
  
  
• [x] Remove html-reporter package (not used anymore)
• [x] Remove integration tests for api package (chore to maintain, without any actual benefit)
• [x] Setup github workflow to publish to @next for long-lived feature branches.
• [x] Create a long-lived feature branch epic/mutation-switching
• [x] Set Dependabot to update on that branch instead of master

Implementation

• [x] I think we should add the new source code instrumentation in a new package: @stryker-mutator/instrumenter (name pending). That way we keep it out of the Stryker core and it will all be a bit more manageable. We might choose to allow a new "instrumenter" plugin later, for now, I would suggest to let Stryker have a full-fledged dependency on it!
  
  
  
  • [x] Add instrumenter package
    
    
    
    
    
    • [x] Add Parsers
    
    
    
    • [x] Add MutantPlacers
    
    
    
    • [x] Add Mutators
    
    
    
    • [x] Add Ast instrumenters
    
    
    
    • [x] Add Printers
    
    
    
  
  
  • [x] Use instrumenter from @stryker-mutator/core
  
  
• [x] Remove old coverage analysis code and the dependency on instanbul
• [x] The "Mutator" api will disappear. That means:
  
  
  
  • [x] We should remove the mutator from directory from @stryker-mutator/api
  
  
  • [x] We should remove the @stryker-mutator/javascript-mutator
  
  
  • [x] We should remove the @stryker-mutator/vue-mutator
  
  
  • [x] We should remove the mutator from @stryker-mutator/typescript. Since we will create a new transpiler, we can remove this entire package.
  
  
• [x] Add a new transpiler called in @stryker-mutator/typescript-transpiler that uses this work around. It should ignore all type errors and work for both project references and plain projects. _no longer needed, since we're scrapping the entire transpiler api_
• [x] Add a new "checker" plugin type (name pending).
  
  
  
  • [x] Add new api. Proposal:
    
    
    
    
    
    • init(files: []): Promise<void>
      
      
      It should be fed the primary files and to check for errors in the initial test run (no errors)
    
    
    
    • check(mutant: Mutant): Promise<MutantStatus>
      
      
      It should be fed one mutant at a time that is than type checked, or filtered (see #1980 )
    
    
    
  
  
  • [x] Implement @stryker-mutator/typescript-checker (name pending)
  
  
• [x] Update the "test-runner" API
  
  
  
  • [x] The test runner will be responsible to activate a specific mutant. We should also add a flag to say weather coverage should be reported, so it is not reported when testing mutants (only after initial test run).
  
  
  • [x] Implement in @stryker-mutator/karma-runner
  
  
  • [x] Implement in @stryker-mutator/jasmine-runner
  
  
  • [x] Implement in @stryker-mutator/mocha-runner
  
  
  • [x] Implement in @stryker-mutator/jest-runner
  
  
  • [x] Implement in @stryker-mutator/wct-runner
  
  
  • [x] Implement in command runner
  
  
• [x] Re-imagine transpilers
  
  
  
  • [x] Remove `@stryker-mutator/webpack-transpiler
  
  
  • [x] Remove @stryker-mutator/babel-transpiler
  
  
  • [x] Add preprocessors to core (private api), add a typescript preprocessor that adds // @ts-nocheck and updates tsconfig.json files.
  
  
  • [x] Add a build command option to core
  
  
• [x] Karma runner: report mutation coverage
• [x] Karma runner: support activating a mutant
• [x] Mocha runner: report mutation coverage. Please remove the OptionsEditor.
• [x] Mocha runner: support activating a mutant
• [x] Jasmine runner: report mutation coverage.
• [x] Jasmine runner: support activating a mutant
• [x] Core
  
  
  
  • [x] Implement new dry-run process (initial test run)
  
  
  • [x] Implement new mutation run process
  
  
  • [x] Implement preprocessors
  
  
  • [x] tsconfig files
  
  
  • [x] Add // @ts-nocheck at the top of each js/ts file.
  
  

I'll be keeping this post up to date with our current progress.

[nicojs]

This would require a major change in how mutation is done. I've discussed this face to face with @simondel and the easiest thing will be to build it next to the current mutation implementation and let people opt-in with an --expermimentalMutationSwitching command line flag (or something like it). That way we can actually release it before we remove the old implementation while keeping it all backward compatible.

We'll be starting development on this next week probably. We'll update this issue along the way.

[nicojs]

I've been working on a PoC for a couple of weeks now and it seems to be working with some success! https://github.com/nicojs/mutation-switch-instrumenter#mutation-switch-instrumenter

I'll be using that PoC to see if we can run tests on Angular/Vue/React projects with Karma/Mocha/Jest, etc.

[kmdrGroch]

Sounds great @nicojs! If you want, you can use my BigMath library to check the performance of this solution :)

[nicojs]

Not so fast there, cowboy 🤠. I've updated the original issue text to reflect the current progress in research and thoughts. As you can see, still some work to be done before we can start implementing.

[nicojs]

I've started to work on an implementation plan (see original issue text)

[vitalets]

Hi @nicojs !

I'm impressed with your Mutation switching new approach. Looking forward for it!

By the way I was thinking about another way of applying mutations in JS/TS - using debugger API. On the first glance the flow can be following:

1. set the breakpoint on the required line of source code
2. run tests
3. when debugger paused on the breakpoint - capture variable values and mutate it
4. resume execution and check tests result

Seems like such approach has it's own pros and cons.
Pros:

• no instrumentation step: easier integration with frameworks

Cons:

• less mutations: we can only change variable values and function return value

What do you think about it?

[nicojs]

That's interesting. It does seem seriously limited. I'm also not sure if it is truly easier to implement since you'll need to inspect the test runner process, calculate the breakpoint position back to the original source code, and than understand the code enough to know which value you want to debug.

If you want to talk more about it, let's do that on slack: https://app.slack.com/client/TTHUR6NNP

[nicojs]

As for the karma mutation coverage reporting. This seems pretty trivial with Karma's plugin system. A small PoC:

// stryker-karma-plugin.js
module.exports = {
  'framework:stryker': ['factory', strykerFramework],
  'middleware:stryker': ['value', strykerMiddleware],
  'middleware:body-parser': ['value', require('body-parser').json()]
}

function strykerFramework (config) {
  config.files.unshift({
    pattern: require.resolve('./adapter.js'), 
    included: true, 
    watched: false,
    served: true
  });
}

strykerFramework.inject = ['config'];

function strykerMiddleware(request, _response, next) {
  if (request.method === 'POST' && request.url === '/stryker-info') {
    console.log('Received from stryker:')
    console.log(request.body);
  } else {
    next();
  }
}

// adapter.js
const originalComplete = window.__karma__.complete.bind(window.__karma__);
window.__karma__.complete = (...args) => {
  fetch('/stryker-info', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      test: true
    })
  }).then(() => {
    originalComplete(...args);
  });
}

Activate with:

// karma.conf.js

module.exports = function(config){
  config.set({
    files: ['src/*.js', 'test/*.js'],
    frameworks: [
      'jasmine',
      'stryker' // Activate the "stryker" framework
    ],
    plugins: [
      'karma-*',
      require.resolve('./stryker-karma-plugin') // Add plugin
    ],
    singleRun: true,
    browsers: [
      'Chrome'
    ],
    middleware: [
      'body-parser', // body-parser used to parse json
      'stryker' // Activate the "stryker" middleware
    ]
  })
}

[nicojs]

Yow! We've started work on this in epic/mutation-switching.

Now first, let me get a coffee ☕

[nicojs]

About the transpilers plugin. I was wondering if we still need the complexity we now have. Couldn't we just copy files to the sandbox and run the transpiler there? We could allow people to configure an arbitrary command to be run before we're running files.

The files that are copied to the sandbox should still be transformed somehow. For example, we should add // @ts-nocheck to each typescript and javascript file. We should also transform tsconfig.json files to make relative paths work. For example, "extends": "../../tsconfig.settings.json" becomes "extends": "../../../../tsconfig.settings.json". Maybe we can use this transform plugin and add a command transpiler, which runs a simple command once.

@simondel @hugo-vrijswijk thoughts?

[simondel]

If TypeScript projects can be supported as easily as they can now, I'm all for it :)

[nicojs]

This would also be the case for the webpack and babel transpiler. All 3 can be replaced by a simple command from the command line.

[nicojs]

I've updated the issue with our current idea for transpilers. The basic idea is to remove them in favor of letting the user configure a buildCommand (name pending)

[kmdrGroch]

I was thinking, if just before merging mutation switching to master could we make performance tests, we could use them later on blog post or somewhere (like if you are going to make a talk, you could mention it or sth)

[Tummerhore]

This sounds very promising. Can you already make a rough estimation for when this feature will be implemented?
I'm currently doing a case study on the practical usage of mutation testing with an existing (relatively big) TS project and try to find out if and how we can apply mutation testing in a reasonable and efficient way. Sadly my initial test with Stryker took nearly 2 days (!) to complete (running on a laptop with i7-6600U) so it would be very interesting to see how mutation switching (eventually along with coverage analysis for jest) may impact the total execution time in this case and if that would make it more feasible.
I would also be willing to help and contribute to Stryker (e.g. this feature) if you need help and I've got enough time.

[brodybits]

it would be very interesting to see how mutation switching (eventually along with coverage analysis for jest) may impact the total execution time in this case

You should be able to install Stryker from GitHub and give it a try (maybe easier said than done).

[nicojs]

For anyone that wants to help. We're using the 4.0 milestone for this.

https://github.com/stryker-mutator/stryker/milestone/9

[jimmyandrade]

Wow, looking forward for this ❤️

[nicojs]

If anyone wants to give it a try, we've released a beta 🤗
https://stryker-mutator.io/blog/2020-07-13/announcing-stryker-4-beta-mutation-switching

I'm hoping to release v4 in a few weeks 🤐

[nicojs]

@Tummerhore

Sadly my initial test with Stryker took nearly 2 days (!) to complete (running on a laptop with i7-6600U) so it would be very interesting to see how mutation switching (eventually along with coverage analysis for jest) may impact the total execution time in this case and if that would make it more feasible

Wow! I think you are my personal hero 🦸‍♂️.

Mutation switching _by itself_ will not have a significant impact on Jest projects unfortunately. You might have some performance improvement because Jest would be able to cache build results, but definitely not an order of magnitude. However, it does open up the way for coverage analysis in jest. Hot reload should theoretically also be possible with Jest, but not with their current public api alone. We'll be focussing on jest next, after we've released v4.

[kmdrGroch]

I think I'll have to check it on my BigMath project soon :D

[brodybits]

I tried this with Prettier on an 8 vCPU cloud instance, see https://github.com/brodybits/prettier/pull/42 but encountered some issues:

• memory issues with a couple large source files
• timeout error message on initial test run, even with --timeoutMS 5000000 command line option, then went quiet

I think https://github.com/brodybits/prettier/pull/42 should be enough to reproduce my issues, but please let me know if there is anything else I can do to help isolate them.

[Tummerhore]

@brodybits As far as I know --timoutMS is not used for the initial test run but for the actual mutation testing. Currently (or at least in the old stable version) the maximum time for the inital test run is hard coded as 5 minutes.
Also see my issue: https://github.com/stryker-mutator/stryker/issues/2302

[kmdrGroch]

@brodybits there could be some memory leak 🤔

[brodybits]

I tried with React (https://github.com/facebook/react). I was able to get through the initial test run with the following configuration in stryker.conf.js on a local build of Stryker 4.0.0-beta from GitHub after some trial-and-error:

/**
 * @type {import('@stryker-mutator/api/core').StrykerOptions}
 */
module.exports = {
  mutator: 'javascript',
  packageManager: 'yarn',
  reporters: ['html', 'clear-text', 'progress'],
  testRunner: 'command',
  transpilers: [],
  coverageAnalysis: 'off',
  buildCommand: 'yarn',
  symlinkNodeModules: false,
  sandbox: {
    fileHeaders: {},
    stripComments: false
  },
};

(React test does seem to be extremely sensitive to the sandbox. The sandbox config does not seem to be supported by 4.0.0-beta.2 from npm.)

---

When I tried adding this mutate list:

  mutate: [
    'packages/react/**/*.js',
    '!packages/react/node_modules/**/*.js',
    '!packages/react/__tests__/**/*.js',
  ],

then the initial test run failed. I got a huge number of errors from npx stryker run --fileLogLevel trace --logLevel debug starting with this:

FAIL packages/react-dom/src/events/plugins/__tests__/EnterLeaveEventPlugin-test.js

  ● EnterLeaveEventPlugin › should set onMouseLeave relatedTarget properly in iframe

    /home/brodybits/react/.stryker-tmp/sandbox6232602/packages/react/src/ReactBaseClasses.js: Unsupported type ConditionalExpression

      20 |       return evalToString(ast.left) + evalToString(ast.right);
      21 |     default:
    > 22 |       throw new Error('Unsupported type ' + ast.type);
         |             ^
      23 |   }
      24 | }
      25 | 

      at evalToString (scripts/shared/evalToString.js:22:13)
      at PluginPass.CallExpression (scripts/error-codes/transform-error-messages.js:43:35)
      at newFn (node_modules/@babel/traverse/lib/visitors.js:179:21)
      at NodePath._call (node_modules/@babel/traverse/lib/path/context.js:55:20)
      at NodePath.call (node_modules/@babel/traverse/lib/path/context.js:42:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:90:31)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitSingle (node_modules/@babel/traverse/lib/context.js:84:19)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:140:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitMultiple (node_modules/@babel/traverse/lib/context.js:79:17)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:138:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitMultiple (node_modules/@babel/traverse/lib/context.js:79:17)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:138:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitMultiple (node_modules/@babel/traverse/lib/context.js:79:17)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:138:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitMultiple (node_modules/@babel/traverse/lib/context.js:79:17)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:138:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitSingle (node_modules/@babel/traverse/lib/context.js:84:19)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:140:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitSingle (node_modules/@babel/traverse/lib/context.js:84:19)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:140:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitSingle (node_modules/@babel/traverse/lib/context.js:84:19)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:140:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitMultiple (node_modules/@babel/traverse/lib/context.js:79:17)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:138:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at NodePath.visit (node_modules/@babel/traverse/lib/path/context.js:97:18)
      at TraversalContext.visitQueue (node_modules/@babel/traverse/lib/context.js:112:16)
      at TraversalContext.visitSingle (node_modules/@babel/traverse/lib/context.js:84:19)
      at TraversalContext.visit (node_modules/@babel/traverse/lib/context.js:140:19)
      at Function.traverse.node (node_modules/@babel/traverse/lib/index.js:84:17)
      at traverse (node_modules/@babel/traverse/lib/index.js:66:12)
      at transformFile (node_modules/@babel/core/lib/transformation/index.js:107:29)
          at transformFile.next (<anonymous>)
      at run (node_modules/@babel/core/lib/transformation/index.js:35:12)
          at run.next (<anonymous>)
      at Function.transform (node_modules/@babel/core/lib/transform.js:27:41)
          at transform.next (<anonymous>)
      at evaluateSync (node_modules/gensync/index.js:244:28)
      at Function.sync (node_modules/gensync/index.js:84:14)
      at Object.transform (node_modules/@babel/core/lib/transform.js:36:54)
      at Object.process (scripts/jest/preprocessor.js:86:20)
      at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:446:35)
      at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:525:40)
      at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:563:25)
      at Object.<anonymous> (packages/react/src/React.js:38:25)
      at Object.<anonymous> (packages/react/index.js:59:14)
      at Object.<anonymous> (packages/react-dom/src/events/plugins/__tests__/EnterLeaveEventPlugin-test.js:21:13)

---

When I tried mutate on packages/react-devtools-extensions:

  mutate: [
    'packages/react-devtools-extensions/**/*.js',
    '!packages/react-devtools-extensions/node_modules/**/*.js',
    '!packages/react-devtools-extensions/__tests__/**/*.js',
  ],

it gave me this error:

21:01:38 (45459) ERROR Stryker an error occurred Error: Error while placing mutants of type(s) "StringLiteral" on 52:15 with conditionalExpressionMutantPlacer. TypeError: Property id of DeclareModule expected node to be of a type ["Identifier","StringLiteral"] but instead got "SequenceExpression"
    at Object.validate (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/types/lib/definitions/utils.js:132:11)
    at validateField (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/types/lib/validators/validate.js:24:9)
    at Object.validate (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/types/lib/validators/validate.js:17:3)
    at NodePath._replaceWith (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/replacement.js:172:7)
    at NodePath.replaceWith (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/replacement.js:156:8)
    at conditionalExpressionMutantPlacer (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/dist/src/mutant-placers/conditional-expression-mutant-placer.js:17:14)
    at Object.placeMutant (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/dist/src/mutant-placers/index.js:20:21)
    at exit (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/dist/src/transformers/babel-transformer.js:28:34)
    at NodePath._call (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:55:20)
    at NodePath.call (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:38:14)
    at Object.placeMutant (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/dist/src/mutant-placers/index.js:25:23)
    at exit (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/dist/src/transformers/babel-transformer.js:28:34)
    at NodePath._call (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:55:20)
    at NodePath.call (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:38:14)
    at NodePath.visit (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:101:8)
    at TraversalContext.visitQueue (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/context.js:112:16)
    at TraversalContext.visitSingle (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/context.js:84:19)
    at TraversalContext.visit (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/context.js:140:19)
    at Function.traverse.node (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/index.js:82:17)
    at NodePath.visit (/home/brodybits/react/node_modules/@stryker-mutator/instrumenter/node_modules/@babel/traverse/lib/path/context.js:99:18)

---

Unfortunately I am not all that familiar with the implementation of mutation switching and would appreciate any pointers if there is anything I can do to help get these issues resolved.

I hope this is not too much error output for this discussion and would be happy to move things into separate issues if needed.

[brodybits]

For Prettier, the initial test run went much faster when switching from Jest to the command runner. But for some reason I am getting many test failures from the initial test run. I have a feeling that something is going very wrong with what the mutation switching does to the JavaScript and will try to investigate it another day.

[nicojs]

@brodybits

I complement your bravery, running Stryker on such huge open source projects. But yes, why shouldn't we?

Unsupported type ConditionalExpression

See #2400

TypeError: Property id of DeclareModule expected node

See #2399 (this one should not be that hard, string literal mutator shouldn't mutate DeclareModule identifiers.

For Prettier, the initial test run went much faster when switching from Jest to the command runner. But for some reason I am getting many test failures from the initial test run.

See #2401

🎉

[Lakitna]

I've managed to get it working too, though I did encounter a few bugs. See #2398 and #2402 for details.

Some background. I'm working on a .js project without Typescript, bundling, or other instrumentation on the source code. I'm using Mocha for my test runner.

Things I noticed:

• There are fewer mutations. This is not necessarily a bad thing though, as long as it's intended.
• Function attribute defaults are now properly mutated and tested (ie: function(hello=true) {}). I believe this is a known issue with Stryker@3 that has now been solved.
• I am now running 15 instances, where I used to run 16. Again not necessarily a bad thing, it feels like a new default for max concurrent processes option.
• The biggest thing for me though is that it's actually a lot slower. Below I've added the progress reporter output for both Stryker@3 and [email protected]. Note how the new one is 2 whole minutes slower, that's a time increase of +70%.

Stryker@3
11:08:32 (16712) WARN InputFileResolver Globbing expression "" did not result in any files.
11:08:32 (16712) WARN InputFileResolver Globbing expression "" did not result in any files.
11:08:33 (16712) INFO InputFileResolver Found 87 of 279 file(s) to be mutated.
11:08:33 (16712) INFO InitialTestExecutor Starting initial test run. This may take a while.
11:08:44 (16712) INFO InitialTestExecutor Initial test run succeeded. Ran 1028 tests in 12 seconds (net 1379 ms, overhead 5139 ms).
11:08:45 (16712) INFO MutatorFacade 3196 Mutant(s) generated
11:08:45 (16712) INFO SandboxPool Creating 16 test runners (based on CPU count)
Mutation testing  [] 8% (elapsed: <1m, remaining: ~3m) 266/3064 tested (3 survived, 0 timed out)11:09:08 (16712) WARN Sandbox Failed find coverage data for this mutant, running all tests. This might have an impact on performance: StringLiteral: ("") file://{{projectRoot}}\src\cli\translators\detail.js:97:36
Mutation testing  [] 72% (elapsed: ~2m, remaining: <1m) 2223/3064 tested (73 survived, 2 timed out)11:10:49 (16712) WARN Sandbox Failed find coverage data for this mutant, running all tests. This might have an impact on performance: BooleanLiteral: (false) file://{{projectRoot}}\src\core\xml\xml-path.js:44:31
Mutation testing  [] 73% (elapsed: ~2m, remaining: <1m) 2251/3064 tested (73 survived, 2 timed out)11:10:50 (16712) WARN Sandbox Failed find coverage data for this mutant, running all tests. This might have an impact on performance: BooleanLiteral: (false) file://{{projectRoot}}\src\core\xml\xml-path.js:87:24
Mutation testing  [] 74% (elapsed: ~2m, remaining: <1m) 2297/3064 tested (73 survived, 2 timed out)11:10:53 (16712) WARN Sandbox Failed find coverage data for this mutant, running all tests. This might have an impact on performance: BooleanLiteral: (false) file://{{projectRoot}}\src\core\xml\xml-path.js:192:31
Mutation testing  [] 100% (elapsed: ~2m, remaining: n/a) 3064/3064 tested (89 survived, 7 timed out)
11:11:29 (16712) INFO MutationTestReportCalculator Final mutation score of 92.30 is greater than or equal to break threshold 50
11:11:29 (16712) INFO HtmlReporter Your report can be found at: file:///{{projectRoot}}/reports/mutation/html/index.html
11:11:29 (16712) INFO Stryker Done in 2 minutes 56 seconds.

[email protected]
11:38:50 (18168) WARN InputFileResolver Globbing expression "" did not result in any files.
11:38:50 (18168) WARN InputFileResolver Globbing expression "" did not result in any files.
11:38:50 (18168) INFO InputFileResolver Found 87 of 279 file(s) to be mutated.
11:38:52 (18168) INFO Instrumenter Instrumented 87 source file(s) with 2890 mutant(s)
11:38:52 (18168) INFO ConcurrencyTokenProvider Creating 15 test runner process(es).
11:38:52 (18168) INFO DryRunExecutor Starting initial test run. This may take a while.
11:38:55 (18168) INFO DryRunExecutor Initial test run succeeded. Ran 1028 tests in 5 seconds (net 1282 ms, overhead 797 ms).
Mutation testing  [=========] 100% (elapsed: ~4m, remaining: n/a) 2890/2890 tested (199 survived, 10 timed out)
11:43:49 (18168) INFO MutationTestReportHelper Final mutation score of 93.10 is greater than or equal to break threshold 50
11:43:50 (18168) INFO HtmlReporter Your report can be found at: file:///{{projectRoot}}/reports/mutation/html/index.html
11:43:50 (18168) INFO MutationTestExecutor Done in 4 minutes 59 seconds.

[brodybits]

Thanks @nicojs. I think the other major thing with Prettier is the resource usage and long prep time I had encountered on a large cloud instance, as I listed in https://github.com/stryker-mutator/stryker/issues/2401#issuecomment-676436431. I did clean up my attempt in https://github.com/brodybits/prettier/pull/42, in case it may help at all.

I can try to test new beta releases from time to time, otherwise leaving it in your hands for now. Yeah what a monster mutating Prettier!

P.S. I did minimize my long comment as "duplicate", since it is now covered by separate issues.

[nicojs]

@Lakitna thanks for giving the beta a spin!

There are fewer mutations

This is probably because we don't generate mutants anymore that we couldn't place (i.e. babel doesn't allow invalid syntax). A good example is property keys:

const foo = {
  'bar': baz
}

In this example, the string literal bar was mutated in 3.0 but no longer in 4, because inserting a ternary operator there isn't allowed there. We could place the mutant with a switch case for example, but I decided against it, because we don't mutate regular property names, so why mutate them when they appear as literals?

I believe this is a known issue with Stryker@3 that has now been solved.

Awesome! 🍺

I am now running 15 instances, where I used to run 16.

Yes, during the refactoring of the concurrency (for the checker api), I've decided to reserve 1 core for the stryker main process by default. You can override it with --concurrency. Note: the max concurrency is no longer capt by Stryker, so don't use a too high concurrency.

The biggest thing for me though is that it's actually a lot slower.

Hmm interesting. Let's take a closer look at your use case and the possible performance improvements with mutation switching:

• Less I/O. This improvement might not have a big impacting on your use case. 🤷‍♂️
• One-time transpiling. You don't use transpiling ❌
• Hot reload. This would be a big performance improvement I think, but alas, this isn't implemented yet. ❌
• Test coverage analysis. This is something Stryker 3 already supported with istanbul for your use case, correct? So no performance gain here. ❌

I was planning to pickup hot reload after the 4.0 release, but with this performance impact in mind, maybe we should already implement it in the beta? What do you think?

Just to be sure, you are running with --coverageAnalysis perTest correct?

[Lakitna]

I decided against it, because we don't mutate regular property names, so why mutate them when they appear as literals?

That was actually a minor annoyance for me. A good decision as far as I'm concerned

I've decided to reserve 1 core for the stryker main process by default. You can override it with --concurrency

I'll give it a run with --concurrency as a performance check. I'll also keep an eye on my CPU usage while it's running. My gut tells me that the reserved core makes sense, but I'm curious anyway 😊

I was planning to pickup hot reload after the 4.0 release, but with this performance impact in mind, maybe we should already implement it in the beta? What do you think?

The way you phrased it makes me think that hot reload is not a breaking change, right?

The thing is that it's worse for me, but a lot better for others in this thread. In fact, Stryker suddenly became feasible for bigger projects.

Thinking about this makes me curious about what the impact is on a small project with transpiling. Something like a React/Angular/Vue/... front-end written in Typescript. If someone knows about an open-source project like this, I'll be glad to run it in 3 and 4.beta.

In fact, I've been wondering about how Stryker scales in general. The relation between mutation count and runtime feels exponential-ish. Would you be up for including some benchmarks in the repo? I'm not sure how to set it up yet, but I have some ideas.

If the only projects that are worse off are those like mine, then I think its a no-brainer to release asap. When It's not as clear, and it rarely is, you can also release 4 with the note in the changelog that it comes with a performance hit for smaller projects.

Just to be sure, you are running with --coverageAnalysis perTest correct?

Yes, I am.

[nicojs]

Hot reload [...] The way you phrased it makes me think that hot reload is not a breaking change, right?

Well ... it might break some test suites that don't clean up nicely between test runs. See #2413

The thing is that it's worse for me, but a lot better for others in this thread. In fact, Stryker suddenly became feasible for bigger projects.

That's the goal yes! 😊

The relation between mutation count and runtime feels exponential-ish. Would you be up for including some benchmarks in the repo?

Yes, please! Benchmarking would be awesome! I started trying out some of them in the perf directory. My goal was to add use cases there, but I stopped at one (angular project).

Feel free to add more use cases there! A great example of a big project that might look like your use case is express. If someone wants to add, please make the PR against the current master branch, that way it will make it easy for us to test it in both versions. 🤘

[Lakitna]

Feel free to add more use cases there! A great example of a big project that might look like your use case is express.

Express would be an interesting one for sure. I've created an issue for it.

Though I've been thinking even further. I mentioned the relation between mutation count and runtime, I think it's interesting to explore that. So I was thinking of benchmarks that can be run with a variable amount of mutations at regular intervals. This would provide us with a performance graph, rather than a single number. I feel like something like that could help tremendously with decision making.