Stryker: Use stryker with @open-wc karma-esm plugin

[vlaraort]

Summary

I am trying to make stryker work in a currently developed project. It's testing development is based in karma as a runner, and mocha as a framework. But it has a caveat, it uses karma-webpack to preproccess the files.

I have tryied to make it work with a stryker.conf.js like this

module.exports = config => {
  config.set({
    testRunner: 'karma',
    testFramework: 'mocha',
    // coverageAnalysis: 'perTest',
    logLevel: 'debug',
    mutator: 'javascript',
    reporters: ['html', 'progress'],
    mutate: [
      'packages/**/*.js',
      '!packages/**/*.test.js',
    ],
    karma: {
      projectType: 'custom',
      configFile: './karma.conf.js',
      config: {
        browsers: ['ChromeHeadless']
      },
    }
  });
};

But it never kills a mutant, my feeling is that is not modifying the transpiled code, only the source one, so karma is not using the mutants.

My karma.conf is generated on-the-fly, but more or less this is the configuration

{
    "frameworks": ["mocha", "snapshot", "mocha-snapshot", "source-map-support", "webpack"],
    "files": ["packages/*/test/*.test.js", "node_modules/@babel/polyfill/dist/polyfill.js"],
    "reporters": [ "mocha", "coverage-istanbul"],
    "singleRun": true,
    "browsers": ["ChromeHeadlessNoSandbox"],
    "preprocessors": {
        "**/*.test.js": ["webpack", "sourcemap"],
        "**/*.spec.js": ["webpack", "sourcemap"]
    },
    "plugins": ["/Workspaces/MY_PROJECT/node_modules/karma-mocha/lib/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-mocha-reporter/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-source-map-support/lib/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-webpack/lib/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-sourcemap-loader/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-coverage-istanbul-reporter/src/reporter.js", "/Workspaces/MY_PROJECT/node_modules/karma-static/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-snapshot/lib/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-mocha-snapshot/lib/index.js", "/Workspaces/MY_PROJECT/node_modules/karma-chrome-launcher/index.js", "karma-*"],
    "coverageIstanbulReporter": {
        "reports": ["html", "lcovonly", "text-summary"],
        "dir": "coverage",
        "combineBrowserReports": true,
        "skipFilesWithNoCoverage": false,
        "thresholds": {
            "global": {
                "statements": 70,
                "branches": 70,
                "functions": 70,
                "lines": 70
            }
        }
    },
    "webpack": {
        "devtool": "inline-cheap-module-source-map",
        "resolve": {
            "mainFields": ["module", "jsnext:main", "browser", "main"]
        },
        "module": {
            "rules": [{
                "test": {},
                "loader": "/Workspaces/MY_PROJECT/node_modules/@open-wc/webpack-import-meta-loader/webpack-import-meta-loader.js"
            }, {
                "test": {},
                "loader": "/Workspaces/MY_PROJECT/node_modules/istanbul-instrumenter-loader/dist/cjs.js",
                "enforce": "post",
                "include": "/Workspaces/MY_PROJECT",
                "exclude": {},
                "options": {
                    "esModules": true
                }
            }, {
                "test": {},
                "use": ["file-loader"]
            }, {
                "test": {},
                "use": ["babel-loader"],
                "exclude": {}
            }]
        }
    }
}

I have tryied to use the webpack-transpiler, and the babel-transpiler, but I receive lots of errors and I am unable to make it work.

Is there any sample or demo project about how to set up a karma-webpack-babel environment with stryker? I have been struggling for two days and I can't find a solution

Stryker environment

├─┬ @stryker-mutator/[email protected]
│ ├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected]
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ └── @stryker-mutator/[email protected] deduped
├─┬ [email protected]

Also, If it's needed I can create a sample project so you can play around.

Thanks!

[nicojs]

Thanks @vlaraort for reporting this issue. Your configuration should work as far as I can tell. Indeed, Stryker _always mutates sources_ (albeit in a sandbox directory), never transpiled code.

With your current setup, Stryker should copy your project to the sandbox and mutate code from there while running karma to verify the mutants. Problems might occur because you have absolute paths inside your karma.conf.js file (/Workspaces/...).

Creating a sample project is the fastest way to get you up-and-running, so I'm all for that 👍

[vlaraort]

Sure! here you have it :)

https://github.com/vlaraort/sample-riker-karma-webpack

The information that I found creating the sample project, is that if I remove the gitignore, all the node_modules folder are included into the riker-tmp folder, and it works properly. But this shouldn't be the way, as in big projects this can be a real memory issue.

Thank you for all the help and the awesome project @nicojs 😄

(Sorry about the nested karma configuration, but this is the real dependency that I have in my project and I would like to have it working with this one)

[nicojs]

Problems seem to arise from @open-wc/testing-helpers. You're importing that and it fails to resolve if your running from a place other than right next to the node_modules folder. Any ideas?

Uncaught TypeError: Failed to resolve module specifier "@open-wc/testing-helpers/index.js". Relative references must start with either "/", "./", or "../".

I will need to investigate this further but I'll probably don't have more time today...

[nicojs]

Probably has to do with this middleware (loaded by @open-wc/testing-helpers):

https://github.com/open-wc/open-wc/blob/master/packages/karma-esm/src/middleware.js

[vlaraort]

I have upgraded the project downgrading that dependency, now it don't have that issue. Sorry about that. Now it launches some weird errors about jasmine (not included in this project at all), but it runs the mutations

[nicojs]

I've opened https://github.com/open-wc/open-wc/issues/481.

[vlaraort]

oh cool! thank you for your support!.
In any case, with the last change you don't have the dependency with that karma-esm :)
Thanks again!

[nicojs]

I've opened another issue with the _actual problem_ with karma-esm.

https://github.com/open-wc/open-wc/issues/484

I've also updated the title of this issue to more accurately represent the problem.

[vlaraort]

Hi nicojs! thank you for all the support!

In any case, I am updating my karma dependencies, to use the legacy config of the @open-wc library, so in my use case I am not using the karma-esm file, I am sticking to a webpack-karma configuration.

With this configuration, the issue is the original defined in the first post, the mutation testing runs without problem, but the mutations are applied on the source code, not the transpiled. Probably is just a configuration problem with the webpack plugin, but I don't find how to make it work.

I have updated the sample repository, do you want me to open another issue, as this one is now related with the karma-esm plugin?

Thank you again!

[LarsDenBakker]

Sorry this took a bit longer, but we've rewritten our testing testing setup. It now properly makes use of karma's base path, can you see if it works for you now?

[igomezal]

It doesn't seems to be solved.

I just run it in my project and it gives me the following error:

17:41:46 (62575) TRACE ChildProcessProxy 19 08 2020 17:41:46.948:WARN [web-server]: 404: /node_modules/@open-wc/testing/index.js

17:41:46 (62575) TRACE ChildProcessProxy Chrome Headless 84.0.4147.135 (Mac OS 10.15.5) ERROR: 'Error loading test file: /packages/hello-world-webcomponent/test/HelloWorldWebcomponent.test.js'

17:41:46 (62575) TRACE ChildProcessProxy Chrome Headless 84.0.4147.135 (Mac OS 10.15.5) ERROR: 'TypeError: Failed to fetch dynamically imported module: http://localhost:9876/base/packages/hello-world-webcomponent/test/HelloWorldWebcomponent.test.js?3f37b2cb4bb36b2ff74b97635d7212ab87fb7635'

So it seems it moves or try to find the files in the root path instead of trying to get it from the relative path.

karma start works without any issue but stryker run doesn't run correctly.