Stryker: Since upgrading to Stryker version 1.3 it can't find any tests to run

Created on 25 Apr 2019 · 17Comments · Source: stryker-mutator/stryker

Summary

I upgraded the following from version 1.2 to 1.3

@stryker-mutator/core
@stryker-mutator/javascript-mutator
@stryker-mutator/mocha-framework
@stryker-mutator/mocha-runner

and now running npx stryker run reports "No tests were executed. Stryker will exit prematurely. Please check your configuration."

Stryker config

module.exports = function(config) {
  config.set({
    mutate: [
      'src/**/*.js',
      '!src/defaults.js',
      '!src/errors.js',
      '!src/index.js'
    ],
    mutator: 'javascript',
    packageManager: 'npm',
    reporters: ['clear-text', 'progress'],
    testRunner: 'mocha',
    mochaOptions: {
      files: ['test/unit/**/*.test.js'],
      require: ['test/unit/testHelper.js']
    },
    transpilers: [],
    testFramework: 'mocha',
    coverageAnalysis: 'perTest',
    thresholds: { high: 80, low: 70, break: null }
  })
}

Stryker environment

├─┬ @stryker-mutator/[email protected]
│ ├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected]
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ └── @stryker-mutator/[email protected] deduped
├─┬ @stryker-mutator/[email protected]
│ ├── @stryker-mutator/[email protected] deduped

[email protected]

Test runner environment

    "test:unit": "find ./test/unit -name '*.test.js' | NODE_ENV=test xargs mocha --require ./test/unit/testHelper.js",
    "test:mutants": "NODE_ENV=test NODE_PATH=. npx stryker run"

Your Environment

| software | version(s)
| ---------------- | -------
| node | 11.14.0
| npm | 6.9.0
| Operating System | macOS 10.14.4

Add stryker.log

11:22:36 (5300) INFO ConfigReader Using stryker.conf.js in the current working directory.
11:22:36 (5300) DEBUG ConfigReader Loading config stryker.conf.js
11:22:36 (5300) DEBUG PluginLoader Loading @stryker-mutator/* from /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator
11:22:36 (5300) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/javascript-mutator" (matched with expression @stryker-mutator/*)
11:22:36 (5300) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-framework" (matched with expression @stryker-mutator/*)
11:22:36 (5300) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-runner" (matched with expression @stryker-mutator/*)
11:22:36 (5300) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/javascript-mutator
11:22:36 (5300) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-framework
11:22:36 (5300) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-runner
11:22:37 (5300) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/core/src/reporters/index.js
11:22:37 (5300) DEBUG MochaOptionsLoader Mocha >= 6 detected. Using mocha's `loadOptions` to load mocha options
11:22:37 (5300) DEBUG MochaOptionsLoader Loaded options: {
  "require": [
    "test/unit/testHelper.js"
  ],
  "extension": [
    "js"
  ],
  "timeout": 2000,
  "ui": "bdd"
}
11:22:37 (5300) DEBUG BroadcastReporter Broadcasting to reporters ["clear-text","progress"]
11:22:37 (5300) INFO InputFileResolver Found 3 of 29 file(s) to be mutated.
11:22:37 (5300) DEBUG InputFileResolver All input files: [
  "/Users/davesag/src/personal/amqp-delegate/.circleci/config.yml",
  "/Users/davesag/src/personal/amqp-delegate/.editorconfig",
  "/Users/davesag/src/personal/amqp-delegate/.eslintrc.js",
  "/Users/davesag/src/personal/amqp-delegate/.gitignore",
  "/Users/davesag/src/personal/amqp-delegate/.prettierignore",
  "/Users/davesag/src/personal/amqp-delegate/CONTRIBUTING.md",
  "/Users/davesag/src/personal/amqp-delegate/LICENSE",
  "/Users/davesag/src/personal/amqp-delegate/README.md",
  "/Users/davesag/src/personal/amqp-delegate/config.yml",
  "/Users/davesag/src/personal/amqp-delegate/docker-compose.yml",
  "/Users/davesag/src/personal/amqp-delegate/package-lock.json",
  "/Users/davesag/src/personal/amqp-delegate/package.json",
  "/Users/davesag/src/personal/amqp-delegate/src/attachEvents.js",
  "/Users/davesag/src/personal/amqp-delegate/src/defaults.js",
  "/Users/davesag/src/personal/amqp-delegate/src/errors.js",
  "/Users/davesag/src/personal/amqp-delegate/src/index.js",
  "/Users/davesag/src/personal/amqp-delegate/src/makeDelegator.js",
  "/Users/davesag/src/personal/amqp-delegate/src/makeWorker.js",
  "/Users/davesag/src/personal/amqp-delegate/stryker.conf.js",
  "/Users/davesag/src/personal/amqp-delegate/test/.eslintrc.js",
  "/Users/davesag/src/personal/amqp-delegate/test/integration/delegateAsyncTask.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/integration/delegateManyTasks.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/integration/delegateTask.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/integration/testHelper.js",
  "/Users/davesag/src/personal/amqp-delegate/test/unit/attachEvents.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/unit/fakes.js",
  "/Users/davesag/src/personal/amqp-delegate/test/unit/makeDelegator.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/unit/makeWorker.test.js",
  "/Users/davesag/src/personal/amqp-delegate/test/unit/testHelper.js"
]
11:22:37 (5300) DEBUG InputFileResolver Files to mutate: [
  "/Users/davesag/src/personal/amqp-delegate/src/attachEvents.js",
  "/Users/davesag/src/personal/amqp-delegate/src/makeDelegator.js",
  "/Users/davesag/src/personal/amqp-delegate/src/makeWorker.js"
]
11:22:37 (5300) DEBUG TestFrameworkOrchestrator Using testFramework mocha based on `testFramework` setting
11:22:37 (5300) INFO InitialTestExecutor Starting initial test run. This may take a while.
11:22:37 (5300) DEBUG Sandbox Creating a sandbox for files in /Users/davesag/src/personal/amqp-delegate/.stryker-tmp/sandbox7133032
11:22:37 (5300) DEBUG Sandbox Creating test runner 0
11:22:37 (5300) DEBUG ChildProcessProxy Starting /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/core/src/test-runner/ChildProcessTestRunnerWorker.js in child process 5304
11:22:37 (5300) DEBUG InitialTestExecutor Adding test hooks for coverageAnalysis "perTest".
11:22:37 (5300) DEBUG TimeoutDecorator Starting timeout timer (300000 ms) for a test run
11:22:37 (5304) DEBUG ChildProcessProxyWorker Changing current working directory for this process to /Users/davesag/src/personal/amqp-delegate/.stryker-tmp/sandbox7133032
11:22:37 (5304) DEBUG PluginLoader Loading @stryker-mutator/* from /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator
11:22:37 (5304) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/javascript-mutator" (matched with expression @stryker-mutator/*)
11:22:37 (5304) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-framework" (matched with expression @stryker-mutator/*)
11:22:37 (5304) DEBUG PluginLoader Loading plugin "/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-runner" (matched with expression @stryker-mutator/*)
11:22:37 (5304) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/javascript-mutator
11:22:37 (5304) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-framework
11:22:37 (5304) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/mocha-runner
11:22:37 (5304) DEBUG PluginLoader Loading plugins /Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/core/src/reporters/index.js
11:22:37 (5304) DEBUG MochaTestRunner Mocha >= 6 detected. Using mocha's `handleFiles` to load files
11:22:37 (5304) DEBUG MochaTestRunner Starting Mocha test run
11:22:37 (5304) DEBUG MochaTestRunner Mocha test run completed: 0/0 passed
11:22:37 (5300) DEBUG ChildProcessProxy Disposing of worker process 5304
11:22:37 (5300) DEBUG ChildProcessProxy Kill 5304
11:22:37 (5300) WARN InitialTestExecutor No tests were executed. Stryker will exit prematurely. Please check your configuration.
11:22:37 (5300) INFO MutatorFacade 72 Mutant(s) generated

🐛 Bug

Source

davesag

👍1

Most helpful comment

@davesag it has to do with the fix introduced in https://github.com/stryker-mutator/stryker/pull/1520. As a workaround, you can explicitly set maxConcurrentTestRunners: 2 in your stryker.conf.js file. I'll be working on a fix.

@MHeironimus your issue should be fixed by renaming files to spec. Since v1.3 we're actively supporting mocha 6, which uses spec. Before that, we didn't support mocha 6, although it did work by chance.

nicojs on 26 Apr 2019

🎉1 👍1

All 17 comments

Notes:

Changing files to spec as per mocha-runner#mochaoptionsspec-string-or-string kind of works but I get this output now and the tests never complete.

Ran 9.93 tests per mutant on average.
------------------|---------|----------|-----------|------------|----------|---------|
File              | % score | # killed | # timeout | # survived | # no cov | # error |
------------------|---------|----------|-----------|------------|----------|---------|
All files         |   59.65 |       34 |         0 |         23 |        0 |      15 |
 attachEvents.js  |  100.00 |        8 |         0 |          0 |        0 |       0 |
 makeDelegator.js |   48.28 |       14 |         0 |         15 |        0 |       5 |
 makeWorker.js    |   60.00 |       12 |         0 |          8 |        0 |      10 |
------------------|---------|----------|-----------|------------|----------|---------|
11:29:49 (5349) INFO Stryker Done in 3 seconds.
(node:5349) UnhandledPromiseRejectionWarning: Error: Child process [pid 5365] exited unexpectedly with exit code 1 (without signal). Last part of stdout and stderr was:
    /Users/davesag/src/personal/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:35
                throw new Exception_1.Exception(`Could not inject "${Class.name}"`, error);
                ^

    Error: Could not inject "ChildProcessTestRunnerWorker". Inner error: Could not inject "MochaTestRunner". Inner error: Cannot find module 'test/unit/testHelper.js'
        at ValueProvider.injectClass (/Users/davesag/src/personal/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:35:19)
        at ValueProvider.injectClass (/Users/davesag/src/personal/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:95:22)
        at ChildProcessProxyWorker.handleMessage (/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/core/src/child-proxy/ChildProcessProxyWorker.js:43:45)
        at process.emit (events.js:193:13)
        at emit (internal/child_process.js:848:12)
        at processTicksAndRejections (internal/process/task_queues.js:81:17)

    at ChildProcessProxy.handleUnexpectedExit (/Users/davesag/src/personal/amqp-delegate/node_modules/@stryker-mutator/core/src/child-proxy/ChildProcessProxy.js:150:33)
    at ChildProcess.emit (events.js:193:13)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:255:12)
(node:5349) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:5349) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

davesag on 25 Apr 2019

FYI pinning it back to version 1.2 everything works fine, so I've done that for now.

Ran 4.66 tests per mutant on average.
-----------------------|---------|----------|-----------|------------|----------|---------|
File                   | % score | # killed | # timeout | # survived | # no cov | # error |
-----------------------|---------|----------|-----------|------------|----------|---------|
All files              |  100.00 |       55 |         0 |          0 |        0 |      19 |
 utils                 |  100.00 |       15 |         0 |          0 |        0 |       2 |
  invoker.js           |  100.00 |        4 |         0 |          0 |        0 |       1 |
  messageCorrelator.js |  100.00 |        7 |         0 |          0 |        0 |       1 |
  taskRunner.js        |  100.00 |        4 |         0 |          0 |        0 |       0 |
 attachEvents.js       |  100.00 |        8 |         0 |          0 |        0 |       0 |
 makeDelegator.js      |  100.00 |       18 |         0 |          0 |        0 |       6 |
 makeWorker.js         |  100.00 |       14 |         0 |          0 |        0 |      11 |
-----------------------|---------|----------|-----------|------------|----------|---------|

davesag on 25 Apr 2019

@davesag thanks for reporting this issue. I've found your project here: https://github.com/davesag/amqp-delegate, so I'll take a look later. Seems that the new mocha 6 file discovery is not working correctly

nicojs on 25 Apr 2019

👍1

FYI: I am seeing the same issue.

Stryker Config

module.exports = function(config) {
  config.set({
    mutator: "typescript",
    packageManager: "npm",
    reporters: ["html", "clear-text", "progress"],
    testRunner: "mocha",
    transpilers: ["typescript"],
    testFramework: "mocha",
    mochaOptions: {
      opts: "test/mocha.opts",
      files: ["build/test/**/*.js"]
    },
    coverageAnalysis: "off",
    tsconfigFile: "tsconfig.json",
    mutate: ["src/**/*.ts"],
    htmlReporter: {
      baseDir: "coverage/mutation/html"
    }
  });
};

Stryker Versions

  "devDependencies": {
    "@stryker-mutator/core": "1.3.0",
    "@stryker-mutator/html-reporter": "1.3.0",
    "@stryker-mutator/mocha-framework": "1.3.0",
    "@stryker-mutator/mocha-runner": "1.3.0",
    "@stryker-mutator/typescript": "1.3.0",
    "chai": "4.2.0",
    "mocha": "6.1.4",
    "mocha-junit-reporter": "1.22.0",
    "mocha-typescript": "1.1.17",
    "nyc": "14.0.0",
    "sinon": "7.3.2",
    "ts-node": "8.1.0",
    "tslint": "5.16.0",
    "typescript": "3.4.5"
  },

MHeironimus on 25 Apr 2019

@davesag thanks for reporting this issue. I've found your project here: https://github.com/davesag/amqp-delegate, so I'll take a look later. Seems that the new mocha 6 file discovery is not working correctly

Yep thanks. Also this one https://github.com/davesag/amqp-simple-pub-sub

davesag on 25 Apr 2019

Okay I have made some progress on this. I've changed my test script in package.json to use the --spec option that mocha 6 provides, like so:

"test:unit": "NODE_ENV=test mocha --require ./test/unit/testHelper.js --spec ./test/unit/**/*.test.js",

and I have added a leading ./ to the paths in the stryker.conf.js file.

The updated stryker.conf.js is

module.exports = function(config) {
  config.set({
    mutate: [
      'src/**/*.js',
      '!src/defaults.js',
      '!src/errors.js',
      '!src/index.js'
    ],
    mutator: 'javascript',
    packageManager: 'npm',
    reporters: ['clear-text', 'progress'],
    testRunner: 'mocha',
    mochaOptions: {
      spec: ['./test/unit/**/*.test.js'],
      require: ['./test/unit/testHelper.js']
    },
    transpilers: [],
    testFramework: 'mocha',
    coverageAnalysis: 'perTest',
    thresholds: { high: 80, low: 70, break: null }
  })
}

This works fine with stryker 1.2 but still in 1.3 I get

(node:30947) UnhandledPromiseRejectionWarning: Error: Child process [pid 30970] exited unexpectedly with exit code 1 (without signal). Last part of stdout and stderr was:
    /Users/davesag/src/personal/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:35
                throw new Exception_1.Exception(`Could not inject "${Class.name}"`, error);
                ^

    Error: Could not inject "ChildProcessTestRunnerWorker". Inner error: Could not inject "MochaTestRunner". Inner error: Cannot find module '/Users/davesag/src/personal/amqp-delegate/.stryker-tmp/sandbox5841625/test/unit/testHelper.js'

and the test runner hangs rather than exiting cleanly.

davesag on 26 Apr 2019

@davesag thanks for the update. I can reproduce that as well. Stryker runs like normal, but once it's done it logs the strange error. Can you confirm that this happens for you as well?

12:22:01 (78) INFO ConfigReader Using stryker.conf.js in the current working directory.
12:22:02 (78) INFO InputFileResolver Found 6 of 35 file(s) to be mutated.
12:22:02 (78) INFO InitialTestExecutor Starting initial test run. This may take a while.
12:22:07 (78) INFO InitialTestExecutor Initial test run succeeded. Ran 48 tests in 5 seconds (net 63 ms, overhead 1839 ms).
12:22:08 (78) INFO MutatorFacade 74 Mutant(s) generated
12:22:08 (78) INFO SandboxPool Creating 8 test runners (based on CPU count)
Mutation testing  [==================================================] 100% (ETC n/a) 74/74 tested (0 survived)

Ran 4.66 tests per mutant on average.
-----------------------|---------|----------|-----------|------------|----------|---------|
File                   | % score | # killed | # timeout | # survived | # no cov | # error |
-----------------------|---------|----------|-----------|------------|----------|---------|
All files              |  100.00 |       55 |         0 |          0 |        0 |      19 |
 utils                 |  100.00 |       15 |         0 |          0 |        0 |       2 |
  invoker.js           |  100.00 |        4 |         0 |          0 |        0 |       1 |
  messageCorrelator.js |  100.00 |        7 |         0 |          0 |        0 |       1 |
  taskRunner.js        |  100.00 |        4 |         0 |          0 |        0 |       0 |
 attachEvents.js       |  100.00 |        8 |         0 |          0 |        0 |       0 |
 makeDelegator.js      |  100.00 |       18 |         0 |          0 |        0 |       6 |
 makeWorker.js         |  100.00 |       14 |         0 |          0 |        0 |      11 |
-----------------------|---------|----------|-----------|------------|----------|---------|
12:22:35 (78) INFO Stryker Done in 32 seconds.
(node:78) UnhandledPromiseRejectionWarning: Error: Child process [pid 176] exited unexpectedly with exit code 1 (without signal). Last part of stdout and stderr was:
        /mnt/c/y/github/davesag/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:35
                    throw new Exception_1.Exception(`Could not inject "${Class.name}"`, error);
                    ^

        Error: Could not inject "ChildProcessTestRunnerWorker". Inner error: Could not inject "MochaTestRunner". Inner error: Cannot find module '/mnt/c/y/github/davesag/amqp-delegate/.stryker-tmp/sandbox6405031/test/unit/testHelper.js'
            at ValueProvider.injectClass (/mnt/c/y/github/davesag/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:35:19)
            at ValueProvider.injectClass (/mnt/c/y/github/davesag/amqp-delegate/node_modules/typed-inject/src/InjectorImpl.js:95:22)
            at ChildProcessProxyWorker.handleMessage (/mnt/c/y/github/davesag/amqp-delegate/node_modules/@stryker-mutator/core/src/child-proxy/ChildProcessProxyWorker.js:43:45)
            at process.emit (events.js:197:13)
            at emit (internal/child_process.js:834:12)
            at processTicksAndRejections (internal/process/next_tick.js:76:17)

    at ChildProcessProxy.handleUnexpectedExit (/mnt/c/y/github/davesag/amqp-delegate/node_modules/@stryker-mutator/core/src/child-proxy/ChildProcessProxy.js:150:33)
    at ChildProcess.emit (events.js:197:13)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:254:12)
(node:78) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:78) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Note: I can only reproduce this in linux, not windows

nicojs on 26 Apr 2019

🎉1 👍1

Thanks @nicojs — yes I get that error. I'm running this on macOS locally and linux on CI. I'll try your maxConcurrentTestRunners: 2 suggestion, thanks.

davesag on 26 Apr 2019

I can confirm adding maxConcurrentTestRunners: 2 fixes my issues. I have 8 cores on my laptop and 18 on my iMac Pro so it would be nice to fix this but at least I can unpin the versions. Thanks for the workaround. (FYI I am loving Stryker! Have added it to all my open source projects now and am about to run an explainer session at work to demo why it's so good. It's really improved my tests and improved my code by forcing me to make it more testable.)

davesag on 26 Apr 2019

(FYI I am loving Stryker! Have added it to all my open source projects now and am about to run an explainer session at work to demo why it's so good. It's really improved my tests and improved my code by forcing me to make it more testable.)

❤️ Thanks a lot! Keep spreading the word 🥇

PR #1527 should fix it. After a green CI build I'll merge it and release a patch version

nicojs on 26 Apr 2019

@nicojs Thank you for your help. Renaming the files property in the mochaOptions property of my stryker.config.js to spec fixed the issue I as having.

@MHeironimus your issue should be fixed by renaming files to spec. Since v1.3 we're actively supporting mocha 6, which uses spec. Before that, we didn't support mocha 6, although it did work by chance.

MHeironimus on 26 Apr 2019

@davesag
Updating to 1.3.1 should do the trick

Have fun with the explainer session! Don't hesitate to PM me or send me an email if you need any info or feedback from our side 👍

nicojs on 26 Apr 2019

Hi @nicojs I have updated to 1.3.1 and removed the maxConcurrentRunners: 2 line from the config. This works fine locally on my Mac but it's still failing on CircleCI with the mutation tests timing out.

see https://circleci.com/gh/davesag/amqp-delegate/147

davesag on 27 Apr 2019

Note when I put maxConcurrentRunners: 2 back in it runs fine again. Also I now see it exhibiting the same behaviour on my local Mac. Weird.

davesag on 27 Apr 2019

Definitely weird. I will investigate further. Synchronisation problems during cleanup.

nicojs on 27 Apr 2019

I note this is all fixed in Version 2. Thanks.

davesag on 18 May 2019

Was this page helpful?

0 / 5 - 0 ratings