Stryker: Support VueCli3 with Jest

[Zorin95670]

Summary

Unable to run stryker because it fail on import in test.

Stryker config

module.exports = function(config) {
  config.set({
    mutate: ["src/**/*.js", "src/**/*.ts", "src/**/*.vue"],
    mutator: "vue",
    testRunner: "jest",
    jest: {
      config: require('./jest.config.js')
    },
    reporters: ["progress", "clear-text", "html"],
    coverageAnalysis: "off"
  });
};

Stryker environment

    "@stryker-mutator/core": "^1.1.1",
    "@stryker-mutator/html-reporter": "^1.1.1",
    "@stryker-mutator/javascript-mutator": "^1.1.1",
    "@stryker-mutator/jest-runner": "^1.1.1",
    "@stryker-mutator/vue-mutator": "^1.1.1",
    "@vue/cli-plugin-e2e-cypress": "^3.2.0",
    "@vue/cli-plugin-unit-jest": "^3.2.0",
    "@vue/cli-service": "^3.2.0",
    "@vue/test-utils": "^1.0.0-beta.20",
    "babel-jest": "^23.6.0",
    "jest-junit": "^5.2.0",
    "vue-jest": "^3.0.2",
    "core-js": "^3.0.0",

Test runner environment

Npm command :

"test:e2e": "vue-cli-service test:e2e",
 "test:unit": "vue-cli-service test:unit"

Jest config:

module.exports = {
  moduleFileExtensions: [
    'js',
    'jsx',
    'json',
    'vue',
  ],
  transform: {
    '^.+\\.js$': 'babel-jest',
    '^.+\\.vue$': 'vue-jest',
    '.+\\.(css|styl|less|sass|scss|svg|png|jpg|ttf|woff|woff2)$': 'jest-transform-stub',
    '^.+\\.jsx?$': 'babel-jest',
  },
  moduleNameMapper: {
    '^@/(.*)$': '<rootDir>/src/$1',
  },
  snapshotSerializers: [
    'jest-serializer-vue',
  ],
  testMatch: [
    '**/tests/unit/**/*.spec.(js|jsx|ts|tsx)|**/__tests__/*.(js|jsx|ts|tsx)',
  ],
  testURL: 'http://localhost/',
  reporters: [
    'default',
    'jest-junit',
  ],
  collectCoverage: true,
  collectCoverageFrom: [
    'src/**/*.{js,vue}',
    '!src/main.js',
    '!src/index.js'
  ],
  coverageReporters: [
    'lcov',
    'cobertura',
  ],
  coverageDirectory: './target',
};

My Environment

| software | version(s)
| ---------------- | -------
| node | v11.0.0
| npm | v6.9.0

Stryker.log

14:37:36 (9999) ERROR InitialTestExecutor One or more tests resulted in an error:
      ● Test suite failed to run

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/AwesomeTable.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import "core-js/modules/es6.array.sort";
                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    SyntaxError: Unexpected string

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/models/PaginationUtils.class.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import PaginationUtils from '@/models/PaginationUtils.class';
                                                                                                    ^^^^^^^^^^^^^^^

    SyntaxError: Unexpected identifier

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/components/filters/FilterDate.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { shallowMount } from '@vue/test-utils';
                                                                                                    ^

    SyntaxError: Unexpected token {

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/components/filters/FilterText.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import _Promise from "/home/vmoittie/Workspaces/awesome-table/node_modules/@babel/runtime-corejs2/core-js/promise";
                                                                                                    ^^^^^^^^

    SyntaxError: Unexpected identifier

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/components/RowData.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { shallowMount, createLocalVue } from '@vue/test-utils';
                                                                                                    ^

    SyntaxError: Unexpected token {

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/components/FilterManager.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { shallowMount } from '@vue/test-utils';
                                                                                                    ^

    SyntaxError: Unexpected token {

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/App.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { shallowMount } from '@vue/test-utils';
                                                                                                    ^

    SyntaxError: Unexpected token {

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

      ● Test suite failed to run

    /home/vmoittie/Workspaces/awesome-table/.stryker-tmp/sandbox3319273/tests/unit/components/filters/FilterList.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import "core-js/modules/es6.array.find";
                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    SyntaxError: Unexpected string

      at ScriptTransformer._transformAndBuildScript (../../node_modules/jest-runtime/build/script_transformer.js:403:17)

14:37:36 (9999) ERROR StrykerCli an error occurred Error: Something went wrong in the initial test run
    at InitialTestExecutor.validateResult (/home/vmoittie/Workspaces/awesome-table/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:117:15)
    at InitialTestExecutor.<anonymous> (/home/vmoittie/Workspaces/awesome-table/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:46:30)
    at step (/home/vmoittie/Workspaces/awesome-table/node_modules/tslib/tslib.js:133:27)
    at Object.next (/home/vmoittie/Workspaces/awesome-table/node_modules/tslib/tslib.js:114:57)
    at fulfilled (/home/vmoittie/Workspaces/awesome-table/node_modules/tslib/tslib.js:104:62)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Thank for your support :)

[simondel]

Thanks for opening this issue! Is it possible for you to create a repository with an example of this issue? (I assume the code is not open source)

[Zorin95670]

Yes, I do that, please wait =D

[Zorin95670]

https://github.com/Zorin95670/stryker-test

It's a simple project create with vue cli 3, then I add stryker.

[nicojs]

I've looked into this. When you use jest directly, you get the same error:

$ npx jest -c jest.config.js
 FAIL  tests/unit/example.spec.js
  ● Test suite failed to run

    Jest encountered an unexpected token

    This usually means that you are trying to import a file which Jest cannot parse, e.g. it's not plain JavaScript.

    By default, if Jest sees a Babel config, it will use that to transform your files, ignoring "node_modules".

    Here's what you can do:
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/en/configuration.html

    Details:

    C:\z\github\Zorin95670\stryker-test\tests\unit\example.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { shallowMount } from '@vue/test-utils';

Apparently, your unit test command is: vue-cli-service test:unit

So, yet another CLI to ruin this beautiful day.

The magic this CLI does is incredibly simple. It sets 2 environment variables and then calls jest. Apparently, those environment variables are what does the magic 🙄
https://github.com/vuejs/vue-cli/blob/dev/packages/%40vue/cli-plugin-unit-jest/index.js#L13-L15

Workaround:

1. Install cross-env: npm i -D cross-env
2. Add this script: "stryker": "cross-env VUE_CLI_BABEL_TARGET_NODE=true VUE_CLI_BABEL_TRANSPILE_MODULES=true stryker run"
3. Run Stryker: npm run stryker

That should do the trick.

Note: Apparently: the HTML reporter cannot handle </script> tags in user code yet. Please see #1482

[Zorin95670]

Thank a lot for your response, I go test it =D

[nicojs]

@simondel I'm not sure what we should do here. Adding a projectType: 'vue-cli-service' doesn't seem maintainable. Who knows what they will add in the future. On the other hand, how else would we support it?

[simondel]

I think Vue 3 will change things quite a bit. Unfortunately we can't really wait on that because it's currently planned for Q3 2019.

I think we should add a project type for the vue -cli-service in this case.

[Zorin95670]

Or, I can also report the bug in vu-cli project. Because is link to vue cli .

[InfoSec812]

@nicojs This got rid of all of the transform errors, but it now results in a different set of errors:

> [email protected] stryker /home/dphillips/Documents/RedHat/Workspace/ui-bootstrap
> cross-env VUE_CLI_BABEL_TARGET_NODE=true VUE_CLI_BABEL_TRANSPILE_MODULES=true stryker run

21:39:33 (9666) INFO ConfigReader Using stryker.conf.js in the current working directory.
21:39:33 (9666) INFO InputFileResolver Found 7 of 49 file(s) to be mutated.
21:39:33 (9666) INFO InitialTestExecutor Starting initial test run. This may take a while.
21:39:34 (9666) ERROR InitialTestExecutor One or more tests resulted in an error:
          ● Test suite failed to run

    TypeError: $export is not a function

      at Object.<anonymous> (../../node_modules/core-js/library/modules/es6.object.define-property.js:6:1)
      at Object.<anonymous> (../../node_modules/core-js/library/fn/object/define-property.js:3:1)
      at Object.<anonymous> (../../node_modules/@babel/runtime-corejs2/core-js/object/define-property.js:3:18)
      at Object.<anonymous> (../../node_modules/core-js/library/modules/_object-dp.js:5:46)
      at Object.<anonymous> (../../node_modules/core-js/library/modules/_hide.js:3:10)
      at Object.<anonymous> (../../node_modules/core-js/library/modules/_export.js:9:12)
      at Object.<anonymous> (../../node_modules/core-js/library/modules/_iter-define.js:5:15)
      at Object.<anonymous> (../../node_modules/core-js/library/modules/es6.string.iterator.js:6:1)
      at Object.<anonymous> (../../node_modules/core-js/library/fn/promise.js:5:1)
      at Object.<anonymous> (../../node_modules/@babel/runtime-corejs2/core-js/promise.js:3:18)
      at Object.<anonymous> (../../node_modules/regenerator-runtime/runtime.js:5:39)

I'll see if I can jigger the tranform ignores a little and make some progress.

[InfoSec812]

Woot!!! Success!

I had transtormIgnorePatterns: ['<rootDir>/node_modules/'] in my jest.config.js. I changed it to transtormIgnorePatterns: ['/node_modules/'] and Styker runs perfectly!!!

Thanks again for all of the help!!!

[nicojs]

I had transtormIgnorePatterns: ['<rootDir>/node_modules/'] in my jest.config.js. I changed it to transtormIgnorePatterns: ['/node_modules/'] and Styker runs perfectly!!!

@InfoSec812 Great to hear you got it working. However, we do want it to work out-of-the-box with your jest config. Could you open a separate issue detailing how to reproduce it? That would be most welcome!

@simondel
I agree, let's add it.

[InfoSec812]

@nicojs So, like most others on this thread, I created a new Vue app using @vue/cli. Other than that, the only special things I have done (and it doesn't seem to make any difference) is that I sometimes use quasar-framework on top of Vue (vue add quasar). The instructions I just gave my colleague which seem to work are as follows:

Setting up Stryker for mutation testing:

• Install stryker CLI: npm install -g stryker-cli
• Initialize Stryker inside the project: stryker init
• Install cross-env: npm install --save-dev cross-env
• Add a script to package.json to run Stryker: "stryker": "cross-env VUE_CLI_BABEL_TARGET_NODE=true VUE_CLI_BABEL_TRANSPILE_MODULES=true stryker run"

[InfoSec812]

I have created an example project to demonstrate the config, but the HTML reports are garbled for some reason:

https://github.com/InfoSec812/example-stryker-vue-project

Clone the project, cd into the root, run npm run stryker, view the reports in reports/mutation/html/index.html.

You will see a page with a bunch of JSON littered inside of it.

[stephen-smith]

I'm running into #1229 when I try and use vue+ts. Tried using the cross-env setup from above, but I still get the import/export/module error.

[kmdrGroch]

Is this issue still present? I think we support 3x versions?

[stephen-smith]

Is this issue still present? I think we support 3x versions?

https://github.com/stryker-mutator/stryker/issues/1229#issuecomment-646393363

[kmdrGroch]

@stephen-smith it seems to be working in 4.0.0 version :)
take a look at https://github.com/stryker-mutator/stryker/tree/epic/mutation-switching/e2e/test/vue-cli-javascript-jest
With small tweaks I could make it work as standalone
tests.zip

Although its still JS project not TS

[nicojs]

I think we should add a project type for the vue-cli-service in this case.

This should improve vue support. Something like this.

{
  "jest": {
    "projectType": "vue-cli"
  }
}