Stryker: All tests timing out when using Stryker & JEST with Typescript?

[sswales]

Question
Have you seen all tests timeout before when using Stryker & JEST with Typescript?

Stryker environment

    "@stryker-mutator/core": "^2.0.2",
    "@stryker-mutator/html-reporter": "^2.0.2",
    "@stryker-mutator/jest-runner": "^2.0.2",
    "@stryker-mutator/typescript": "^2.0.2",
    "@stryker-mutator/babel-transpiler": "^2.0.2",

    "jest": "^24.8.0",
    "jest-emotion": "^10.0.11",
    "jest-fetch-mock": "^2.1.2",
    "jest-html-reporter": "^2.5.0",
    "@types/graphql": "^14.2.0",
    "@types/jest": "^24.0.13",
    "@types/react": "^16.8.19",
    "@types/react-dom": "^16.8.4",
    "@types/react-router-dom": "^4.3.3",
    "@types/single-spa-react": "^2.8.3",
    "@typescript-eslint/eslint-plugin": "^1.10.2",
    "@typescript-eslint/parser": "^1.10.2",
    "babel-eslint": "^10.0.1",
    "babel-loader": "^8.0.6",
    "babel-plugin-emotion": "^10.0.9",

Additional context

I have used the same version of plugins on a repository which is just targeting normal JS files and it works fine.

I only seem to be having this issue with Typescript.

Also, I have tried doing minimal files and I still get the same results, timeouts.

[kmdrGroch]

@shaunswales thanks for reporting this. Is it possible for you to provide us some repository, example code to check this?

[simondel]

Hi! Could you post your Stryker.conf.js?

If there's a translilers option in there, could you try running it without it?

[sswales]

mutate: [...thefileList],
reporters: ['html', 'clear-text', 'progress', 'dashboard'],
mutator: 'typescript',
tsconfigFile: 'tsconfig.json',
packageManager: 'yarn',
testRunner: 'jest',
testFramework: 'jest',
coverageAnalysis: 'off',
jest: { enableFindRelatedTests: false },
htmlReporter: { baseDir: 'reports/mutation' },
thresholds: { high: 90, low: 75, break: 70 },
allowConsoleColors: true,
timeoutMs: 60000,

That is as it stands at the moment, with the mutate files I have tried everything and a limited set of files but still getting the same issues.

An here is the output:

+ yarn run test:mutation
yarn run v1.13.0
$ yarn workspaces run test:mutation
$ ./stryker.sh
11:42:28 (59) INFO ConfigReader Using stryker.conf.js in the current working directory.
11:42:29 (59) INFO TypescriptConfigEditor Loading tsconfig file ./tsconfig.json
11:42:29 (59) INFO BroadcastReporter Detected that current console does not support the "progress" reporter, downgrading to "progress-append-only" reporter
11:42:29 (59) WARN InputFileResolver Globbing expression "src/__mocks__/" did not result in any files.
11:42:29 (59) INFO InputFileResolver Found 16 of 138 file(s) to be mutated.
11:42:29 (59) INFO InitialTestExecutor Starting initial test run. This may take a while.
11:42:54 (59) INFO InitialTestExecutor Initial test run succeeded. Ran 113 tests in 25 seconds (net 15244 ms, overhead 9058 ms).
11:42:54 (59) INFO MutatorFacade 192 Mutant(s) generated
11:42:54 (59) INFO SandboxPool Creating 8 test runners (based on CPU count)
Mutation testing 0% (ETC n/a) 0/192 tested (0 survived)
Mutation testing 0% (ETC n/a) 0/192 tested (0 survived)
Mutation testing 0% (ETC 1h, 35m, 30s) 1/192 tested (0 survived)
Mutation testing 2% (ETC 24m, 56s) 5/192 tested (0 survived)
Mutation testing 4% (ETC 16m, 56s) 9/192 tested (0 survived)
Mutation testing 4% (ETC 20m, 20s) 9/192 tested (0 survived)
Mutation testing 4% (ETC 23m, 43s) 9/192 tested (0 survived)
Mutation testing 8% (ETC 13m, 43s) 17/192 tested (0 survived)
Mutation testing 8% (ETC 15m, 26s) 17/192 tested (0 survived)
Mutation testing 8% (ETC 17m, 9s) 17/192 tested (0 survived)
Mutation testing 8% (ETC 18m, 52s) 17/192 tested (0 survived)
Mutation testing 13% (ETC 13m, 21s) 25/192 tested (0 survived)
Mutation testing 13% (ETC 14m, 28s) 25/192 tested (0 survived)
Mutation testing 13% (ETC 15m, 35s) 25/192 tested (0 survived)
Mutation testing 15% (ETC 13m, 30s) 30/192 tested (5 survived)
Mutation testing 17% (ETC 12m, 50s) 33/192 tested (8 survived)
Mutation testing 17% (ETC 13m, 39s) 33/192 tested (8 survived)
Mutation testing 17% (ETC 14m, 27s) 33/192 tested (8 survived)
Mutation testing 21% (ETC 11m, 39s) 41/192 tested (9 survived)
Mutation testing 21% (ETC 12m, 16s) 41/192 tested (9 survived)
Mutation testing 21% (ETC 12m, 30s) 42/192 tested (9 survived)
Mutation testing 23% (ETC 11m, 58s) 45/192 tested (9 survived)
Mutation testing 25% (ETC 11m, 11s) 49/192 tested (9 survived)
Mutation testing 25% (ETC 11m, 40s) 49/192 tested (9 survived)
Mutation testing 26% (ETC 11m, 50s) 50/192 tested (9 survived)
Mutation testing 29% (ETC 10m, 31s) 56/192 tested (11 survived)
Mutation testing 29% (ETC 10m, 39s) 57/192 tested (11 survived)
Mutation testing 30% (ETC 10m, 46s) 58/192 tested (12 survived)
Mutation testing 32% (ETC 10m, 8s) 62/192 tested (16 survived)
Mutation testing 33% (ETC 9m, 46s) 65/192 tested (18 survived)
Mutation testing 33% (ETC 10m, 5s) 65/192 tested (18 survived)
Mutation testing 34% (ETC 9m, 57s) 67/192 tested (18 survived)
Mutation testing 38% (ETC 8m, 57s) 73/192 tested (18 survived)
Mutation testing 38% (ETC 9m, 14s) 73/192 tested (18 survived)
Mutation testing 39% (ETC 9m, 6s) 75/192 tested (18 survived)
Mutation testing 40% (ETC 8m, 57s) 77/192 tested (19 survived)
Mutation testing 41% (ETC 8m, 38s) 80/192 tested (19 survived)
Mutation testing 42% (ETC 8m, 29s) 82/192 tested (19 survived)
Mutation testing 43% (ETC 8m, 32s) 83/192 tested (19 survived)
Mutation testing 44% (ETC 8m, 23s) 85/192 tested (19 survived)
Mutation testing 45% (ETC 8m, 4s) 88/192 tested (19 survived)
Mutation testing 46% (ETC 7m, 56s) 90/192 tested (19 survived)
Mutation testing 47% (ETC 7m, 57s) 91/192 tested (19 survived)
Mutation testing 48% (ETC 7m, 48s) 93/192 tested (19 survived)
Mutation testing 49% (ETC 7m, 39s) 95/192 tested (19 survived)
Mutation testing 51% (ETC 7m, 21s) 98/192 tested (19 survived)
Mutation testing 51% (ETC 7m, 21s) 99/192 tested (19 survived)
Mutation testing 52% (ETC 7m, 12s) 101/192 tested (19 survived)
Mutation testing 53% (ETC 7m, 12s) 102/192 tested (19 survived)
Mutation testing 55% (ETC 6m, 45s) 106/192 tested (19 survived)
Mutation testing 55% (ETC 6m, 53s) 106/192 tested (19 survived)
Mutation testing 55% (ETC 6m, 53s) 107/192 tested (19 survived)
Mutation testing 57% (ETC 6m, 35s) 110/192 tested (19 survived)
Mutation testing 59% (ETC 6m, 9s) 114/192 tested (19 survived)
Mutation testing 59% (ETC 6m, 16s) 114/192 tested (19 survived)
Mutation testing 59% (ETC 6m, 14s) 115/192 tested (19 survived)
Mutation testing 61% (ETC 5m, 57s) 118/192 tested (19 survived)
Mutation testing 63% (ETC 5m, 32s) 122/192 tested (19 survived)
Mutation testing 63% (ETC 5m, 38s) 122/192 tested (19 survived)
Mutation testing 64% (ETC 5m, 29s) 124/192 tested (19 survived)
Mutation testing 65% (ETC 5m, 19s) 126/192 tested (19 survived)
Mutation testing 67% (ETC 4m, 55s) 130/192 tested (19 survived)
Mutation testing 67% (ETC 5m, 0s) 130/192 tested (19 survived)
Mutation testing 68% (ETC 4m, 50s) 132/192 tested (19 survived)
Mutation testing 69% (ETC 4m, 48s) 133/192 tested (19 survived)
Mutation testing 71% (ETC 4m, 18s) 138/192 tested (19 survived)
Mutation testing 71% (ETC 4m, 22s) 138/192 tested (19 survived)
Mutation testing 72% (ETC 4m, 12s) 140/192 tested (19 survived)
Mutation testing 73% (ETC 4m, 9s) 141/192 tested (19 survived)
Mutation testing 75% (ETC 3m, 46s) 145/192 tested (19 survived)
Mutation testing 76% (ETC 3m, 43s) 146/192 tested (19 survived)
Mutation testing 77% (ETC 3m, 34s) 148/192 tested (19 survived)
Mutation testing 77% (ETC 3m, 30s) 149/192 tested (19 survived)
Mutation testing 79% (ETC 3m, 8s) 153/192 tested (19 survived)
Mutation testing 80% (ETC 3m, 5s) 154/192 tested (19 survived)
Mutation testing 81% (ETC 2m, 55s) 156/192 tested (19 survived)
Mutation testing 81% (ETC 2m, 57s) 156/192 tested (19 survived)
Mutation testing 83% (ETC 2m, 36s) 160/192 tested (19 survived)
Mutation testing 84% (ETC 2m, 26s) 162/192 tested (19 survived)
Mutation testing 85% (ETC 2m, 16s) 164/192 tested (19 survived)
Mutation testing 85% (ETC 2m, 18s) 164/192 tested (19 survived)
Mutation testing 87% (ETC 1m, 57s) 168/192 tested (19 survived)
Mutation testing 88% (ETC 1m, 47s) 170/192 tested (19 survived)
Mutation testing 89% (ETC 1m, 43s) 171/192 tested (19 survived)
Mutation testing 89% (ETC 1m, 38s) 172/192 tested (19 survived)
Mutation testing 91% (ETC 1m, 18s) 176/192 tested (19 survived)
Mutation testing 92% (ETC 1m, 8s) 178/192 tested (19 survived)
Mutation testing 93% (ETC 1m, 3s) 179/192 tested (19 survived)
Mutation testing 93% (ETC 59s) 180/192 tested (19 survived)
Mutation testing 95% (ETC 39s) 184/192 tested (19 survived)
Mutation testing 96% (ETC 29s) 186/192 tested (19 survived)
Mutation testing 97% (ETC 24s) 187/192 tested (19 survived)
Mutation testing 97% (ETC 19s) 188/192 tested (19 survived)

----------------------|---------|----------|-----------|------------|----------|---------|
File                  | % score | # killed | # timeout | # survived | # no cov | # error |
----------------------|---------|----------|-----------|------------|----------|---------|
All files             |   90.05 |       26 |       146 |         19 |        0 |       1 |
 components           |  100.00 |       10 |        12 |          0 |        0 |       0 |
  ButtonNav           |  100.00 |        1 |         6 |          0 |        0 |       0 |
   index.tsx          |  100.00 |        1 |         6 |          0 |        0 |       0 |
  Subheader           |  100.00 |        9 |         6 |          0 |        0 |       0 |
   index.tsx          |  100.00 |        9 |         6 |          0 |        0 |       0 |
 pages                |   92.47 |       16 |       119 |         11 |        0 |       1 |
  Errors              |  100.00 |        1 |         2 |          0 |        0 |       0 |
   Error404.tsx       |  100.00 |        1 |         2 |          0 |        0 |       0 |
  EventForm           |   87.78 |       15 |        64 |         11 |        0 |       1 |
   components         |   87.64 |       15 |        63 |         11 |        0 |       1 |
    CategorySection   |   73.81 |       14 |        17 |         11 |        0 |       0 |
     CategoryItem.tsx |   80.00 |        3 |         1 |          1 |        0 |       0 |
     CategoryList.tsx |   70.59 |       10 |        14 |         10 |        0 |       0 |
     index.tsx        |  100.00 |        1 |         2 |          0 |        0 |       0 |
    FixtureSection    |  100.00 |        1 |        17 |          0 |        0 |       1 |
     index.tsx        |  100.00 |        1 |        17 |          0 |        0 |       1 |
    FormActionsBottom |  100.00 |        0 |        29 |          0 |        0 |       0 |
     index.tsx        |  100.00 |        0 |        29 |          0 |        0 |       0 |
   data-handlers      |  100.00 |        0 |         1 |          0 |        0 |       0 |
    save.ts           |  100.00 |        0 |         1 |          0 |        0 |       0 |
  EventList           |  100.00 |        0 |        53 |          0 |        0 |       0 |
   components         |  100.00 |        0 |        53 |          0 |        0 |       0 |
    EventTable        |  100.00 |        0 |         5 |          0 |        0 |       0 |
     EventRow.tsx     |  100.00 |        0 |         5 |          0 |        0 |       0 |
    Search            |  100.00 |        0 |        48 |          0 |        0 |       0 |
     index.tsx        |  100.00 |        0 |        48 |          0 |        0 |       0 |
 styles               |  100.00 |        0 |         3 |          0 |        0 |       0 |
  breakpoints.ts      |  100.00 |        0 |         3 |          0 |        0 |       0 |
 validators           |  100.00 |        0 |         7 |          0 |        0 |       0 |
  validators.ts       |  100.00 |        0 |         7 |          0 |        0 |       0 |
 module.acc.ts        |   38.46 |        0 |         5 |          8 |        0 |       0 |
----------------------|---------|----------|-----------|------------|----------|---------|
11:58:31 (59) INFO DashboardReporter Dashboard report is not sent when not running on a build server
11:58:31 (59) INFO ScoreResultCalculator Final mutation score of 90.05 is greater than or equal to break threshold 70
11:58:31 (59) INFO Stryker Done in 16 minutes 2 seconds.
cleaning up .stryker-tmp directory
$ echo 'graphql: not implemented #todo'
graphql: not implemented #todo
Done in 966.64s.

[mike-dazn]

We faced this issue. We are using ts-jest and it was taking longer than the default stryker timeout to do it's transpilation. Setting timeoutMS: 30000 seems to solve the issue.

[sswales]

Sorry yeah, I forgot I had this issue still open...

We found a similar solution where we tweaked the following options to get it to stop timing out:

{
    timeoutMs: 30000,
    timeoutFactor: 4,
    maxConcurrentTestRunners: 6
}

[brodybits]

I think it would be nice to document this as a hint somewhere. Just saying this as an external user and occasional contributor.