Stryker: Arithmetic Mutator seems to be broken

[JoeGaebel]

Summary

having a function

const multiplyTwoNumbers = (a, b) => a * b

and a test

it('works', () => {
  expect(multiplyTwoNumbers(2,2)).toEqual(4)
});

You would expect the mutant of a + b to survive, however he doesn't appear to be created.

Stryker config

module.exports = function(config) {
  config.set({
    mutator: "javascript",
    packageManager: "yarn",
    reporters: ["clear-text", "progress"],
    testRunner: "jest",
    coverageAnalysis: "off",
    mutate: [
      'src/**/*.js',
      '!src/*spec.js'
    ],
    babel: {optionsFile: 'babel.config.js'},
    transpilers: [
      'babel'
    ],
  });
};

Test runner config
Default (also no) jest configuration

Stryker environment

[email protected] /Users/joe/workspace/stryker-example
├─┬ @stryker-mutator/[email protected]
│ ├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected]
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
└─┬ [email protected]
npm ERR! extraneous: [email protected] /Users/joe/workspace/stryker-example/node_modules/bindings
npm ERR! extraneous: [email protected] /Users/joe/workspace/stryker-example/node_modules/nan
npm ERR! extraneous: [email protected] /Users/joe/workspace/stryker-example/node_modules/node-pre-gyp

"jest": "24.9.0"

Test runner environment

"test": "yarn jest src"

Your Environment

| software | version(s)
| ---------------- | -------
| node | v10.16.3
| npm | 6.9.0
| Operating System | Mac 10.15.3

Add stryker.log

stryker.log

I've made an example repo here: https://github.com/JoeGaebel/stryker-example, to see the issue:

yarn install
yarn mutate

[kmdrGroch]

It should create mutator 'a / b' not 'a+b'...
And indeed it did:
21:43:40 (14623) TRACE JavaScriptMutator Generated mutant for mutator ArithmeticOperator in file /Users/joe/workspace/stryker-example/src/multiplyTwoNumbers.js with replacement code "a / b"

Conclusion: it works as intended.
More info: https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#arithmetic-operator

[kmdrGroch]

@JoeGaebel if that solved your problem, consider closing this issue :) unless you have some question

[JoeGaebel]

@kmdrGroch Ah, I see. I suppose then it is operating as intended, however, this leaves a big hole in the mutation testing.

In that case I'll mark this as a feature request.

[kmdrGroch]

I don't see a big hole 🤔 can you provide some examples?

[simondel]

Hi @JoeGaebel why would you expect a * b to be mutated to a + b? Pretty much every mutation is meant to be the oposite of the original code. So a - b would be mutated to a + b and a * b is mutated to a / b.

[JoeGaebel]

@simondel, I see it this way. A mutant is defined as code that's been modified in a way that your tests should catch, but in fact your tests do not catch.

In the example I've provided, the test covers a case where multiplication and addition operations give the same result, yet the function is quite explicitly intending multiplication.

In this case, a mutant exists, simply by changing the operation to addition, yet Stryker does not make or validate these mutants.

Why restrict the mutations to be only the opposite of the real implementation operation? This reduces the coverage of mutation testing for code that operates on numerics.

[simondel]

In the example I've provided, the test covers a case where multiplication and addition operations give the same result, yet the function is quite explicitly intending multiplication.

As far as I know, there are only two numbers where this is the case: 0 and 2. In all other cases you would have a different outcome. Since your function accepts the numbers as inputs, I don't expect you to always work with 2*2

Why restrict the mutations to be only the opposite of the real implementation operation? This reduces the coverage of mutation testing for code that operates on numerics.

This is true. However, if you start going this route, you should also change a*b into a-b because they are equal when a and b are 0. This causes you to make a lot more mutations in the code, which are only valid in a small percentage of cases. But it would slow down every run of Stryker.

This mutation is not something that I want Stryker to make, so I'm closing this issue for now. If there are any other mutations that you think Stryker misses, please let us know!