Stryker: Support mutating TypeScript code

[simondel]

Stryker is only capable of mutating JavaScript. For TypeScript projects this results in an unprecise mutation score as generated code is being mutated and because some mutations would normally cause compilation errors.

So as an extension of issue #343, the `stryker-typescript plugin will be developed. @thomasp24 has already done a great deal for this and the current code can be viewed on the support-typescript branch. Based on that @nicojs created a typescript-mutation-poc in various features are demonstrated needed for the changes here.

Todo list

We're working on this in the stryker-typescript branch.

• [x] Gather info on projects that want this plugin so we can test it.
  
  
  
  • Stryker itself
  
  
  • Typed html: https://github.com/nicojs/typed-html
  
  
  • Link parent bin: https://github.com/nicojs/node-link-parent-bin/
  
  
  • Install local: https://github.com/nicojs/node-install-local
  
  
  • Moneyboom ch frontend: https://github.com/wearesho-team/moneyboom-ch-frontend (see comment 327761543)
  
  
  • Wearesho-site https://github.com/wearesho-team/wearesho-site (see comment 327761543)
  
  
• [x] Update stryker-api for a Transpiler api
• [x] Update stryker-api's Mutator plugin. Update stryker's Mutator for this structure.
• [x] Update stryker to use the Transpiler api.
• [x] Create TypescriptConfigEditor responsible for enhancing stryker.conf.js with typescript's config.
• [x] Create TypescriptTranspiler responsible for transpiling ts code and generating sourcemaps
• [x] Write unit tests for TypescriptTranspiler
• [x] Create TypescriptMutator responsible for mutating typescript code and validating that it can still compile.
• [x] Write unit tests for TypescriptMutator.
• Add Typescript mutator equivalents for every js mutator in stryker.
  
  
  
  • [x] ArrayDeclaratorMutator
  
  
  • [x] BinaryOperatorMutator
  
  
  • [x] BlockStatementMutator
  
  
  • [x] BooleanSubstitutionMutator
  
  
  • [x] LogicalOperatorMutator
  
  
  • [x] RemoveConditionalsMutator
  
  
  • [x] UnaryOperatorMutator
  
  
  • [x] UpdateOperatorMutator
  
  
• Refactoring in Stryker:
  
  
  
  • [x] Refactor InputFileResolver to now also read in the input files initialially
  
  
  • [x] Make InputFileResolver responsible for reporting OnSourceFileRead and onAllSourceFilesRead
  
  
  • [x] Refactor current Mutators in Stryker to be implemented using the new MutantGenerator api
  
  
  • [x] Refactor - check if code coverage analysis still works. Stryker is now calculating its own line/column offset based on position in a file. It's doing that 0-based, so the first line is 0 and first column is 0. By istanbul inspected code is first line 1 and first column 0.
  
  
  • [x] Notify users if they specify a transpiler and a coverage analysis level, because this is not yet supported.
  
  
  • [x] Add unit tests for the ChildProcessProxy
  
  
  • [x] Don't start a transpiler ChildProcessProxy if there is no transpiler
  
  
  • [x] Rename mutantGenerator to mutator everywhere (see comment 328269389)
  
  
  • [x] Add 'transpiled' property to the FileDescriptor so users can select if a file should be transpiled. Defaults to 'true'.
  
  

Way it will work

When running Stryker on a TypeScript project, we won't mutate your javascript anymore. Please read the blog article about the Stryker 1.0 roadmap to know why. Instead, we'll mutate your TypeScript code. We split the responsibility of TypeScript mutation testing in 2 blocks:

1. Mutate typescript code.

The mutating will be done basically the same way as it is now for javascript code. We'll walk the AST and create mutations on copies of the typescript nodes. This will keep the process really fast. It is OK for these mutations to cause compilation errors as they'll be caught during transpiling of the typescript code.

1. Transpile typescript code.

All code will be transpiled for the initial test run, and then for each mutant after that. This has some advantages over other ways of working:

• By first generating all mutants, we can later decide to devide the transpiling workload over multiple processes (however, we'll start with using the host process for this PR)
• By running the transpiler for each mutant, we can mutate our typescript code as complex or simple as we want. We don't have to worry about the javascript code it will produce. This contrary to transpiling once and than go find-and-replacing bits of javascript code (using source maps). That might be faster, but we would be limited in what we can mutate, as complex typescript constructs (like decorators for example) can result in complex javascript, different for each target (es5, es2015, es2016, ...).

There is some added complexity though: we want to use coverage analysis on transpiled code too. This means that we need to actively keep track of source maps if we want to know which line, column in the source code is hit by a particular javascript statement. Fortunately, TypeScript has excellent source map support. We might skip coverage analysis for this PR. Lets make it work first.

The transpile api will be added to accommodate this process. You'll be able to configure multiple transpiler plugins. Each transpiler gets the output of the previous transpiler. This makes it possible to first transpile using TypeScript and next transpile that output using a bundler like web pack. Each transpiler is responsible for its own source maps.

A picture to illustrate:

foo.ts   ==> Typescript  ==> foo.js ==> Webpack ==> foobar.js
bar.ts   ==> Transpiler  ==> bar.js ==> Transpiler

So foo.ts:3:5 might map to foo.js:5:6 which maps to foobar.js:52:46.

Pretty complex stuff, but makes it really pluggable.

In order to keep things as fast as possible, transpilers know exactly which file is changed for a mutant. This makes it possible to use the TypeScript "watch source files" feature to not do too much work. It goes without saying that we want to do as much work as we can in-memory.

Usage

Your future stryker.conf.js file might look like this:

// stryker.conf.js
module.exports = functions(config) {
  config.set({
    tsconfigFile: 'tsconfig.json',
    mutate: 'src/**/*.ts',
    mutatorGenerator: 'typescript',
    transpilers: [
      'typescript'
    ]
  });
}

Do you want to use this on your project? Please let us know if we can access the source code for testing purposes!

Note: This post will be updated to always show the latest info.

[nicojs]

Do some of the people who have upvoted this issue also have typescript projects to test this on?

@omar10594 @dmitry-korolev @thekiba @Horat1us

[Horat1us]

I have two open-source projects, where I want to test this

https://github.com/wearesho-team/moneyboom-ch-frontend
https://github.com/wearesho-team/wearesho-site

[nicojs]

I have two open-source projects, where I want to test this

Very cool! I can see both projects are using webpack. As we're mutating the typescript source code, instead of the resulting javascript, Stryker will also do the transpiling + bundeling. We won't support webpack out of the box... at first. Just plain typescript transpiling. So you might have to do some more work to get this working.

[Horat1us]

This projects using webpack only for creating bundle for web, not tests. (webpack is too slow to bundle every time I want to run unit tests).

For tests we use Mocha:

cross-env TS_NODE_PROJECT=tsconfig.test.json ./node_modules/.bin/nyc mocha -r jsdom-global/register -r ts-node/register -r source-map-support/register tests/unit/bootstrap.ts tests/unit/**-specs.tsx tests/unit/**-specs.ts

So I use ts-node to transpile typescript before testing.
Is it ok?

[nicojs]

So I use ts-node to transpile typescript before testing.
Is it ok?

TLDR: Yes. If you choose to configure the TypescriptMutantGenerator and not the TypescriptTranspiler.

Theoretically it should work. As you can see, we choose to split mutating of the ts code and transpiling of ts code into 2 separate plugins. If you choose to configure the TypescriptMutantGenerator and not the TypescriptTranspiler, you should be able to use mocha with ts-node. Stryker will just write the mutated source files to the test sandboxes.

However, the TypescriptMutantGenerator will also generate invalid mutants. That means: mutants that cannot compile. These will probably be reported as runtime errors. They won't influence your mutation score, as the mutants are not valid, so its not a big deal.

[mthmulders]

The first bits & pieces of the Badge API are in place, _and_ written in TypeScript. Curious to see how good the tests are. Its tests are currently written in Jest, so that adds another challenge :).

[omar10594]

I have a private project where we use typescript and webpack (I will make a new public small project with similar configuration).

For test with stryker, we first build with webpack, and then use stryker with the js code.

[nicojs]

About the name MutantGenerator. I think we should rename it back to Mutator.

It better explains what the thing does: mutating code. The problem was this is that we have a Mutator plugin point right now that we are removing with this feature. With that, you could write a plugin that can mutate a single javascript AST node. It was kind of blocking this feature, that's why we removed it. To prevent confusion, we decided to name the new plugin differently.

The way I see it now: its better to have some confusion now, than have a confusing name for the rest of the time.

@simondel do you agree?

[simondel]

@nicojs As discussed, I like the name Mutator more.

What kind of tags shall we use for the plugins? I would say stryker-transpiler for the transpilers. Mutators are a bit more difficult. We could use stryker-mutator but we already use that one. We could aos use stryker-mutator-plugin but that's also really generic..

[simondel]

Good news! We don't use the tag stryker-mutator yet 👍

[nicojs]

@simondel
In that case, let's use stryker-mutator. 👍

[ ] Add mutators and transpilers to the initializer code

Do we really need to add the initializer code in this branch? I think that can be a separate feature. Let's first release it as a "soft launch". Test it in multiple projects and iron out any kinks along the way. Add initializer code in an other feature and release it big.

[nicojs]

@mthmulders @Horat1us @omar10594

Yesterday, we finally launched typescript support in stryker. Before we write a blog article highlighting the changes and new features, we want to know if the solution is working for you.

The best place to get started with stryker & typescript is the stryker-typescript readme at the moment. You can also review commit 617f54 i did on install-local for adding stryker to my build.

It works as expected for me. Of course, it would be pretty schizophrenic of me if it did not do so. Please share your experiences with is. Any questions can be posted here as well of course.

[james-r17]

This is just the thing I need for my current project... But I keep coming up against one particular error:

[2017-09-21 14:22:04.988] [ERROR] IsolatedTestRunnerAdapter - /Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:22
    return new Error(stack ? msg + ' (Resolving: ' + stack + ')' : msg);
           ^
Error: No provider for "files"! (Resolving: framework:jasmine -> files)
    at error (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:22:12)
    at Object.get (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:9:13)
    at get (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:54:19)
    at /Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:71:14
    at Array.map (<anonymous>)
    at Array.invoke (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:70:31)
    at Injector.get (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:48:43)
    at /Users/[...]/src/main/webapp/node_modules/karma/lib/server.js:143:20
    at Array.forEach (<anonymous>)
    at Server._start (/Users/[...]/src/main/webapp/node_modules/karma/lib/server.js:142:21)
    at Injector.invoke (/Users/[...]/src/main/webapp/node_modules/di/lib/injector.js:75:15)
    at Server.start (/Users[...]/src/main/webapp/node_modules/karma/lib/server.js:103:18)
    at new KarmaTestRunner (/Users/[...]/src/main/webapp/node_modules/stryker-karma-runner/src/KarmaTestRunner.js:47:22)
    at TestRunnerFactory.Factory.create (/Users/[...]/src/main/webapp/node_modules/stryker-api/src/core/Factory.js:46:20)
    at IsolatedTestRunnerAdapterWorker.start (/Users/[...]/src/main/webapp/node_modules/stryker/src/isolated-runner/IsolatedTestRunnerAdapterWorker.js:59:80)
    at process.<anonymous> (/Users/[...]/src/main/webapp/node_modules/stryker/src/isolated-runner/IsolatedTestRunnerAdapterWorker.js:20:27)

I'm using stryker with jasmine and karma, and the stacktrace appears to suggest an issue with those... except the unit tests do run fine by themselves. I'm only having the issue when I run stryker.

My stryker config is as follows:

module.exports = function(config) {
  config.set({
    files: ['src/**/*.ts'],
    testRunner: "karma",
    reporter: ["clear-text", "progress"],
    testFramework: 'jasmine',
//    coverageAnalysis: "perTest",
    karmaConfigFile: "karma.conf.js",
    mutate: ['src/**/*.ts'],
    coverageAnalysis: 'off', // Coverage analysis with a transpiler is not supported a.t.m.
    tsconfigFile: 'tsconfig.json', // Location of your tsconfig.json file
    mutator: 'typescript', // Specify that you want to mutate typescript code
    logLevel: 'trace'
//    transpilers: [
//        'typescript' // Specify that your typescript code needs to be transpiled before tests can be run. Not needed if you're using ts-node Just-in-time compilation.
//    ]
  });
};

and karma config is as follows:

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular/cli'],
    plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-jasmine-html-reporter'),
      require('karma-coverage-istanbul-reporter'),
      require('@angular/cli/plugins/karma')
    ],
    client:{
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    coverageIstanbulReporter: {
      reports: [ 'html', 'lcovonly' ],
      fixWebpackSourcePaths: true
    },
    angularCli: {
      environment: 'dev'
    },
    reporters: ['progress', 'kjhtml'],
    port: 9876,
    colors: true,
    logLevel: config.DEBUG,
    autoWatch: true,
    browsers: ['Chrome'],
    singleRun: false
  });
};

I think the error somehow relates to how stryker-typescript handles files. When I took out the stryker-typescript plugin and then ran stryker, I didn't get the error.

Any help would be greatly appreciated.

[nicojs]

@james-dsb thanks for trying out our typescript support!

This sounds like an issue with karma. Do you mind opening a separate issue for this? Could you also set logLevel: 'debug' in your stryker.conf.js and report the logging somewhere?

[omar10594]

I made a small repo with similar configuration from a large private project.

https://github.com/omar10594/nutri-starter

We first use webpack to generate three files(dependencies code, main code (to mutate) and specs code), then we run karma only, at last run stryker, and some mutants survive.

Now i'm trying to use stryker-typescript but i think, we should change a lot of code, because we use webpack, and for example, to include a template we import it and webpack use a loader to return the file content (or the url), but if we compile the the file with typescript, we will get a error like:

Cannot find module 'xxx.template.html'

To avoid that kind of errors we should include the templates on typescript files (i suppose that), but we import other files like images, json files, etc., so for projects like ours i think it will be hard implement stryker-typescript.

Edit: I just read right now the road to 1.0, so if i can help with the stryker-webpack-transpiler tell me. 😄

[nicojs]

@omar10594 i've tried your repo and i think stryker works as well as can be expected. I see that you are using webpack to transpile and bundle all typescript files (amongst others) into a main.js. Later you use stryker to mutate the main,js code. This is a good approach seeing as we don't support webpack as a transpiler yet.

I just read right now the road to 1.0, so if i can help with the stryker-webpack-transpiler tell me.

Archcry is an intern at our company and is working on the webpack support. @Archcry maybe you can use @omar10594 nutri-starter project as an example?

I'm closing this issue. We have now formally communicated the support for TypeScript in our latest blog article https://stryker-mutator.github.io/blog/2017-10-06/typescript-support.html

[nicojs]

We forgot to even reference the PR here: it was #376 😄