Stryker: Strykers detect TranspileError only

[Mrna1]

Hi Everyone,
thank you for the great framework. I will be very glad if you help me with my issue.
I have got thru issues, but I did not find an answer. :(

I set my Stryker, but it accepts TransspileError only.

My config:

module.exports = function (config) {
config.set({
coverageAnalysis: "off",
files: [
"test//.ts",
"src/common/fn.ts",
],
mochaOptions: {
files: "test/spec//.spec.js",
opts: "test/mocha.opts"
},
mutate: [
"src/*/.ts"
],
mutator: "typescript",
reporters: ["progress", "clear-text", "html"],
htmlReporter: {
baseDir: 'reports/mutation/html'
},
testFramework: "mocha",
testRunner: "mocha",
transpilers: ["typescript"],
tsconfigFile: "tsconfig.json",
maxConcurrentTestRunners: 10,
//timeoutMS: 60000,
});
};

If I do not set timeoutMS test runs longer time and all non-TranspileError are Time out:

If I set timeoutMS to some high value, test runs lower time and all non-TranspileError are Survived:

When I add some mutations manually. Right tests fail as they should be, but stryker doesn't detect it.

Thank you
Best Regards
Franta

[nicojs]

Ho @Mrna1 thanks for reporting this issue. It seems that Stryker has problems to transpile your mutants. Could you please provide the requested information from the issue template:

Stryker environment

Please list your stryker plugins + versions here (output of npm ls | grep stryker)

+-- [email protected]
+-- [email protected]

Please also add the test runner you are using. npm ls | grep mocha and npm ls | grep typescript for you.

Your Environment

| software | version(s)
| ---------------- | -------
| node |
| npm |
| Operating System |

Add stryker.log

Please add your stryker.log file. This file can be generated using stryker run --fileLogLevel trace. You can drag and drop it here.
Your source code is never logged to this file, however file names are logged. Feel free to obfuscate those log messages if you think it is a problem

[nicojs]

If your project is open source, could you please share the URL?

[Mrna1]

Hi @nicojs,
thank you for the quick answer.

Your Environment

software version(s)
node v10.13.0
npm 6.6.0
Operating System Windows10

"stryker": "^0.34.0",
"stryker-api": "^0.23.0",
"stryker-html-reporter": "^0.17.0",
"stryker-mocha-framework": "^0.14.0",
"stryker-mocha-runner": "^0.16.0",
"stryker-typescript": "^0.17.0",

Runner

npm test

Add stryker.log:

Because it is not open source I could not provide you original logs. But I took all the setting and one file name fn.ts and put to a new project. Now I'm not able to reproduce the timeouts. But The second option is still present. It looks like I had two problems in one :)
stryker.log

[nicojs]

Hmm, we need to find out what the typescript error message is. Appearently, we're not logging that :| I'm a bit perplexed. I'll send you a patched version of the typescript transpiler tomorrow (as well as create a PR to add the logging).

[Mrna1]

@nicojs Thank you very much.
if you want I can remove all company dependencies and send the project to your email.

[nicojs]

@Mrna1 sending me the project via email would be great, I can help you most effectively that way.

I've changed the MutationTestExecutor file locally and compiled it to TypeScript. It now should log transpile errors at debug level. Could you replace your node_modules/stryker/src/process/MutationTestExecutor.js file with this one:
MutationTestExecutor.zip ?

That way we should be able to figure out what's wrong.

[Mrna1]

@nicojs: Hi here are the logs. I will send you the project to email. It will be better.
stryker.log

One more time, thank you for help.
Franta

[nicojs]

Thanks for sending. I've debugged and finally found the issue. It has to do with your setup.

You can fix it in 2 ways,

1. Use relative imports: import * as SUT from "../../../src/common/fn"; instead of import * as SUT from "@common/fn";,
2. Add tsconfig.json to the files array in your stryker config, remove the "outDir" from your tsconfig.json file and change files: "dist/test/spec/**/*.spec.js", to files: "test/spec/**/*.spec.js", in your Stryker config file.

Here is an explanation of what is wrong:

Stryker is working as expected. It mutates your source code as expected. For example: it changes i > 0 to i <= 0. It is even transpiled correctly using typescript and put in the Stryker sandbox (for example .stryker-tmp/sandbox4076165). Awesome!

Then Stryker instructs mocha to run a test. Your mocha.opts file has these settings (amongst others): --require tsconfig-paths/register. I've looked it up, it basically loads your tsconfig and adds your "paths" setting and monkey patches Module._resolveFilename. You're also adding --require ts-node, which means that nodejs will transpile *.ts files to javascript on the fly.

_Note: This is incredibly ugly and i would advice against using tsconfig-paths. It might break at any moment if nodejs team decides to change the name or implementation of their private API._

So whenever you require @common/fn it will translate that to src/common/fn. There it finds the original src/common/fn.ts. It loads it using ts-node. You see what the problem is here? Even though Stryker mutated the source file, it is never required from your test 😅

All in all you have a very complicated setup. I would personally advice to not use magic like tsnode and tsconfig-paths. Just compile your code before you run your tests.

[nicojs]

Also: all transpile errors are errors that are expected. So no problems there

[nicojs]

Maybe we should support a relative NODE_PATH environment variable. So you can use magic imports like @common/fn (but without the magic of tsconfig-paths). More info on $NODE_PATH: https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

[Mrna1]

@nicojs: Thank you for the help. It is working now.
The second part was my fold. I wrongly removed dist from the paths.
About paths: Our platform supports ES5 only so we have to transpile ts to js. There is no other way.
I would be glad for support of a relative path. I can not remove it from our project.

I will probably modify the relative path just to check the tests but unfortunately cannot implement without the support. :(

[nicojs]

@Mrna1

I will probably modify the relative path just to check the tests but unfortunately cannot implement without the support. :(

That's not allowed! We've come this far 😄 I'll help some more

The problem is with tsconfig-paths. I've created an issue over there:
https://github.com/dividab/tsconfig-paths/issues/75

I've also created a workaround. With this, you don't have to alter any source files:

Add this file in the root of your project:

// register-tsconfig-paths.js
const tsConfig = require("./tsconfig.json");
const tsConfigPaths = require("tsconfig-paths");
const path = require("path");

tsConfigPaths.register({
  baseUrl: path.resolve(tsConfig.compilerOptions.baseUrl || '', tsConfig.compilerOptions.outDir || ''),
  paths: tsConfig.compilerOptions.paths
});

Update your stryker config:

module.exports = function (config) {
    config.set({
        coverageAnalysis: "perTest",
        files: [
            // "src/**/*.ts",
            "test/**/*.ts",
            "tsconfig.json",
            "register-tsconfig-paths.js",
            // "!src/entrypoints/*.ts",
            // "!src/**/index.ts",
            // "!src/**/types.ts",
            // "!src/**/di/*.ts",
            // "!src/**/constants.ts",
            // "!src/translator/translations.ts",
            // "!src/common/dom.ts",
            "src/common/fn.ts",
        ],
        mochaOptions: {
            files: "dist/test/spec/**/*.spec.js",
            opts: 'none',
            require: ["./register-tsconfig-paths.js"]
        },
        mutate: [
            "src/common/fn.ts"
        ],
        mutator: "typescript",
        reporters: ["progress", "clear-text", "html"],
        htmlReporter: {
            baseDir: 'reports/mutation/html'
        },
        testFramework: "mocha",
        testRunner: "mocha",
        transpilers: ["typescript"],
        tsconfigFile: "tsconfig.json"
    });
};

That should do the trick

About paths: Our platform supports ES5 only so we have to transpile ts to js. There is no other way.
I would be glad for support of a relative path. I can not remove it from our project.

Well, you could also use a bundler like webpack. It plays nice with relative paths.

I've also enabled coverageAnalysis to give you a 50% performance boost 😀

[nicojs]

Unfortunately, it is impossible to disable loading the mocha.opts file at the moment (which is what is needed for your use case). I want to fix that in the branch created for the transpiler logging.

As a workaround, I've pointed the mocha.opts file to "none". This results in an error being logged, but Stryker will run fine, so you can at least already start using.

14:06:44 (26204) ERROR MochaOptionsLoader Could not load opts from "C:\z\tmp\StrykerTest\none". Please make sure opts file exists.

[Mrna1]

Great thank you, It's working now. I wanted to do a similar thing, but you did it for me.
You are my hero 💃, can close the issue if you want.

Franta

[nicojs]

Thanks for the kind words 😅 . Let's fix the 2 cosmetic problems in Stryker before closing this issue (PR #1345)