Stryker: Potential Memory Leak

[nosideeffects]

Summary

Allocation failed - JavaScript heap out of memory during mutation testing on moderately-sized project module. I've tried increasing heap space to 8GB, but it still ultimately OOMs.

Stryker config

module.exports = function(config) {
  config.set({
    testRunner: "mocha",
    mutator: "javascript",
    transpilers: [],
    reporter: ["clear-text", "progress"],
    testFramework: "mocha",
    coverageAnalysis: "perTest",
    mutate: ["src/**/!(*.spec).js"],
    mochaOptions: {
      files: ["index.js", "test/dependencies.js", "src/**/*.spec.js"],
    },
  });
};

Stryker environment

├─┬ [email protected]
├─┬ [email protected]
├─┬ [email protected]
├─┬ [email protected]
├─┬ [email protected]
├─┬ [email protected]

Your Environment

| software | version(s)
| ---------------- | -------
| node | v9.9.0
| npm | 6.0.0
| Operating System | MacOS 10.12.6

Stack Trace

```[2018-06-28 13:29:08.894] [INFO] InitialTestExecutor - Initial test run succeeded. Ran 17 tests in 15 seconds (net 436 ms, overhead 3563 ms).
[2018-06-28 14:01:23.999] [INFO] Stryker - 86764 Mutant(s) generated
[2018-06-28 14:01:25.129] [INFO] SandboxPool - Creating 4 test runners (based on CPU count)

<--- Last few GCs --->

[97626:0x103800000] 126169 ms: Mark-sweep 8184.6 (8216.9) -> 8184.6 (8216.9) MB, 88.1 / 0.0 ms allocation failure GC in old space requested
[97626:0x103800000] 126261 ms: Mark-sweep 8184.6 (8216.9) -> 8184.6 (8215.9) MB, 92.0 / 0.0 ms last resort GC in old space requested
[97626:0x103800000] 126423 ms: Mark-sweep 8184.6 (8215.9) -> 8184.6 (8215.9) MB, 162.1 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1c8f1b025529
1: fromString(aka fromString) [buffer.js:336] [bytecode=0x1c8fa68fb241 offset=161](this=0x1c8fddf822d1 ,string=0x1c8f12923071 )
2: nextMutant(aka nextMutant) [/Users/jacobbiggs/repos/akitabox/packages/server-modules/node_modules/stryker/src/transpiler/MutantTranspiler.js:~36] [pc=0x3717e3bfad5f](this=0x1c8fdd...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
3: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [/usr/local/bin/node]
4: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
5: v8::internal::String::SlowFlatten(v8::internal::Handle, v8::internal::PretenureFlag) [/usr/local/bin/node]
6: v8::String::WriteUtf8(char, int, int, int) const [/usr/local/bin/node]
7: node::StringBytes::Write(v8::Isolate, char, unsigned long, v8::Local, node::encoding, int) [/usr/local/bin/node]
8: node::Buffer::New(v8::Isolate, v8::Local, node::encoding) [/usr/local/bin/node]
9: node::Buffer::(anonymous namespace)::CreateFromString(v8::FunctionCallbackInfo const&) [/usr/local/bin/node]
10: v8::internal::FunctionCallbackArguments::Call(void ()(v8::FunctionCallbackInfo const&)) [/usr/local/bin/node]
11: v8::internal::MaybeHandle v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::BuiltinArguments) [/usr/local/bin/node]
12: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate) [/usr/local/bin/node]
13: 0x3717e37842fd
Abort trap: 6
```

[JoshuaKGoldberg]

+1, I get this with Jest and 9486 mutations.

Console output:

$ stryker run
[2018-06-29 10:26:47.574] [INFO] ConfigReader - Using stryker.conf.js in the current working directory.
[2018-06-29 10:26:49.890] [INFO] TypescriptConfigEditor - Loading tsconfig file D:\Code\WebClient\tsconfig.json
[2018-06-29 10:26:50.420] [INFO] InputFileResolver - Found 450 of 560 file(s) to be mutated.
[2018-06-29 10:26:50.424] [INFO] InitialTestExecutor - Starting initial test run. This may take a while.
[2018-06-29 10:28:55.227] [INFO] InitialTestExecutor - Initial test run succeeded. Ran 871 tests in 2 minutes 5 seconds (net 12499 ms, overhead 108760 ms).
[2018-06-29 10:28:56.924] [INFO] Stryker - 9486 Mutant(s) generated
[2018-06-29 10:29:00.835] [INFO] SandboxPool - Creating 12 test runners (based on CPU count)
Mutation testing  [==                                                ] 3% (ETC 43267.9s) 301/9486 tested (147 survived)[2018-06-29 10:54:55.030] [ERROR] TestRunnerChildProcessAdapter - FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Mutation testing  [==                                                ] 3% (ETC 43172.5s) 302/9486 tested (147 survived)[2018-06-29 10:54:55.312] [ERROR] TestRunnerChildProcessAdapter -  1: node::DecodeWrite
 2: node_module_register
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::FatalProcessOutOfMemory
 5: v8::internal::Heap::MaxHeapGrowingFactor
 6: v8::internal::ScavengeJob::operator=
 7: v8::internal::ScavengeJob::operator=
 8: v8::internal::Heap::MaxHeapGrowingFactor
 9: v8::internal::Factory::NewFillerObject
10: v8::internal::ParseInfo::GetFlag
11: 000000871F1841C1

[2018-06-29 10:54:55.551] [ERROR] TestRunnerChildProcessAdapter - Child process exited with non-zero exit code 134. Last 10 message from the child process were:
        [2018-06-29 10:45:34.053] [TRACE] JestPromiseTestAdapter - Invoking Jest with config {"collectCoverageFrom":["src/*/**/*.ts*","!src/*/**/*.stubs.ts*","!src/*","!src/*/Adapters/**","!src/*/External/**","!src/Browser/*","!src/BrowserMain/*","!src/Boot/*","!src/ServerSideRendering/*","!src/typings/**"],"coverageDirectory":"./reports/coverage","globals":{"ts-jest":{"tsConfigFile":"./tsconfig.test.json"}},"moduleFileExtensions":["js","ts","tsx"],"rootDir":"D:\\Code\\WebClient\\.stryker-tmp\\sandbox9197369","setupFiles":["<rootDir>/config/polyfills/nodePolyfills.js"],"setupTestFrameworkScriptFile":"<rootDir>/test/unit/setup","testMatch":["<rootDir>/src/**/?(*.)test.ts?(x)"],"transform":{"^.+\\.tsx?$":"ts-jest"},"collectCoverage":false,"verbose":false,"bail":false,"reporters":[]}

stryker.conf.js:

module.exports = function(config) {
    config.set({
        coverageAnalysis: "off",
        files: [
            "config/**/*.js",
            "config/**/*.js.map",
            "config/**/*.ts",
            "src/**/*.js",
            "src/**/*.ts",
            "src/**/*.tsx",
            "test/unit/setup.js",
            "tsconfig.json",
            "tsconfig.test.json",
        ],
        mutate: [
            "src/**/*.ts",
            "src/**/*.tsx",
            "!src/**/*.d.ts",
        ],
        mutator: "typescript",
        reporter: ["baseline", "clear-text", "dashboard", "html", "progress"],
        testRunner: "jest",
        tsconfigFile: "tsconfig.json",
    });
};

jest.conf.js:

/*!
 * Copyright Microsoft Corporation. All rights reserved.
 */

const path = require("path");

module.exports = {
    collectCoverageFrom: [
        "src/*/**/*.ts*",
        "!src/*/**/*.stubs.ts*",
        "!src/*",
        "!src/*/Adapters/**",
        "!src/*/External/**",
        "!src/Browser/*",
        "!src/BrowserMain/*",
        "!src/Boot/*",
        "!src/ServerSideRendering/*",
        "!src/typings/**",
    ],
    coverageDirectory: "./reports/coverage",
    globals: {
        "ts-jest": {
            tsConfigFile: "./tsconfig.test.json",
        },
    },
    moduleFileExtensions: [
        "js",
        "ts",
        "tsx",
    ],
    rootDir: path.join(__dirname, "../../"),
    setupFiles: [
        "<rootDir>/config/polyfills/nodePolyfills.js",
    ],
    setupTestFrameworkScriptFile: "<rootDir>/test/unit/setup",
    testMatch: [
        "<rootDir>/src/**/?(*.)test.ts?(x)",
    ],
    transform: {
        "^.+\\.tsx?$": "ts-jest",
    },
};

[simondel]

moderately-sized project module.

86764 Mutant(s) generated

Nice.

Seriously though. This is something we'll have to look at. @nosideeffects @JoshuaKGoldberg I don't suppose either of your projects is open source, right?

[nosideeffects]

@simondel Correct. This is a mostly closed source repository.

The only other potentially helpful information I have is that regardless of the amount of heap space I provide to node, the process runs out of memory in the same time frame (within a minute of starting test runners).

[JoshuaKGoldberg]

Correct, closed source. Though I will use this thread and others when I attempt to convince my managers to open source it! 🙌

A _very_ old version of my setup that _might_ help repro is: https://github.com/JoshuaKGoldberg/wallabyjs-office-ui-fabric-react-bug-repro

I doubt any of the problems are specific to my particular repo, so if you copy & paste or autogenerate around 500 files with ~2-3 tests on average per file, that might help? If that doesn't do it for you, you ping me and I can try to make a version of my repo with closed source specifics stripped out.

[simondel]

@nosideeffects your mutate array contains "src/**/!(*.spec).js" how many files does that resolve to? And how big (KB/MB) are those files together? In your logs I don't see any progress bar like @JoshuaKGoldberg does have, correct?

@JoshuaKGoldberg thanks for the repo! I'll give it a shot.

[nosideeffects]

All JavaScript in this module (including spec) amounts to 7.3M;

[andrepm06]

I'm facing this OOM issue too. I have 1366 mutants running.

Also, my very last test hangs (99%). Then, the suite is never finished.

[nicojs]

@andrepm06

Also, my very last test hangs (99%). Then, the suite is never finished.

Could you create a separate issue for that? I'm very curious about your stryker configuration and test runner. It might be that one of the reporters has problems finishing, or the test runner itself keeps running and you have a high timeout setting.

We're going to investigate this issue. We have trouble finding big open source projects to test stryker on. For example, expressjs (sounds like a big one) only generates1841 mutants (if you're curious, mutation score: 83, time: 8 minutes). The suggestion of @JoshuaKGoldberg might help, I'll definitely give it a shot. Hope to reproduce this issue with around 10000 mutants

I'm also planning a session with @simondel to go through the code and spot any potential memory leaks.

[andrepm06]

@nicojs wow, 1841 in 8 minutes? I'm interested on that.

I've created an the issue #932 @nicojs

[nicojs]

@andrepm06 I've created a branch for it: see https://github.com/nicojs/express/tree/add-mutation-testing

[simondel]

@nosideeffects @JoshuaKGoldberg Could you try to replace the content of node_modules/stryker/src/TestableMutant.js and see if this makes a difference? I've looked at this with @nicojs and found a possible leak.

This may break the report that stryker generates, but this is just for testing purposes.

With:

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
var SourceFile_1 = require("./SourceFile");
var objectUtils_1 = require("./utils/objectUtils");
var TestSelectionResult;
(function (TestSelectionResult) {
    TestSelectionResult[TestSelectionResult["Failed"] = 0] = "Failed";
    TestSelectionResult[TestSelectionResult["FailedButAlreadyReported"] = 1] = "FailedButAlreadyReported";
    TestSelectionResult[TestSelectionResult["Success"] = 2] = "Success";
})(TestSelectionResult = exports.TestSelectionResult || (exports.TestSelectionResult = {}));
var TestableMutant = /** @class */ (function () {
    function TestableMutant(id, mutant, sourceFile) {
        this.id = id;
        this.mutant = mutant;
        this.sourceFile = sourceFile;
        this._selectedTests = [];
        this.specsRan = [];
        this._timeSpentScopedTests = 0;
        this.testSelectionResult = TestSelectionResult.Success;
    }
    Object.defineProperty(TestableMutant.prototype, "selectedTests", {
        get: function () {
            return this._selectedTests;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "timeSpentScopedTests", {
        get: function () {
            return this._timeSpentScopedTests;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "fileName", {
        get: function () {
            return this.mutant.fileName;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "mutatorName", {
        get: function () {
            return this.mutant.mutatorName;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "range", {
        get: function () {
            return this.mutant.range;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "replacement", {
        get: function () {
            return this.mutant.replacement;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "location", {
        get: function () {
            if (!this._location) {
                this._location = this.sourceFile.getLocation(this.range);
            }
            return this._location;
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "mutatedCode", {
        get: function () {
            return this.sourceFile.content.substr(0, this.range[0]) +
                this.replacement +
                this.sourceFile.content.substr(this.range[1]);
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "originalCode", {
        get: function () {
            return this.sourceFile.content;
        },
        enumerable: true,
        configurable: true
    });
    TestableMutant.prototype.selectAllTests = function (runResult, testSelectionResult) {
        var _this = this;
        this.testSelectionResult = testSelectionResult;
        runResult.tests.forEach(function (testResult, id) { return _this.selectTest(testResult, id); });
    };
    TestableMutant.prototype.selectTest = function (testResult, index) {
        this._selectedTests.push({ id: index, name: testResult.name });
        this._timeSpentScopedTests += testResult.timeSpentMs;
    };
    Object.defineProperty(TestableMutant.prototype, "originalLines", {
        get: function () {
            var _a = this.getMutationLineIndexes(), startIndex = _a[0], endIndex = _a[1];
            return this.sourceFile.content.substring(startIndex, endIndex);
        },
        enumerable: true,
        configurable: true
    });
    Object.defineProperty(TestableMutant.prototype, "mutatedLines", {
        get: function () {
            var _a = this.getMutationLineIndexes(), startIndex = _a[0], endIndex = _a[1];
            return this.sourceFile.content.substring(startIndex, this.mutant.range[0]) + this.mutant.replacement + this.sourceFile.content.substring(this.mutant.range[1], endIndex);
        },
        enumerable: true,
        configurable: true
    });
    TestableMutant.prototype.getMutationLineIndexes = function () {
        var startIndexLines = this.mutant.range[0], endIndexLines = this.mutant.range[1];
        while (startIndexLines > 0 && !SourceFile_1.isLineBreak(this.originalCode.charCodeAt(startIndexLines - 1))) {
            startIndexLines--;
        }
        while (endIndexLines < this.sourceFile.content.length && !SourceFile_1.isLineBreak(this.originalCode.charCodeAt(endIndexLines))) {
            endIndexLines++;
        }
        return [startIndexLines, endIndexLines];
    };
    TestableMutant.prototype.result = function (status, testsRan) {
        return objectUtils_1.freezeRecursively({
            id: this.id,
            sourceFilePath: this.fileName,
            mutatorName: this.mutatorName,
            status: status,
            replacement: this.replacement,
            originalLines: '',
            mutatedLines: '',
            testsRan: [],
            location: this.location,
            range: this.range
        });
    };
    TestableMutant.prototype.toString = function () {
        return this.mutant.mutatorName + ": (" + this.replacement + ") file://" + this.fileName + ":" + (this.location.start.line + 1) + ":" + this.location.start.column;
    };
    return TestableMutant;
}());
exports.default = TestableMutant;
//# sourceMappingURL=TestableMutant.js.map

[JoshuaKGoldberg]

Well, it went from 144/4299 tested to 145/4299 tested, with a different error message. I assume the error is nondeterministic and it didn't fix it 😢:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Mutation testing  [==                                                ] 3% (ETC 29973.2s) 145/4299 tested (41 survived)[2018-07-16 14:37:41.820] [ERROR] TestRunnerChildProcessAdapter - FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[2018-07-16 14:37:42.057] [ERROR] TestRunnerChildProcessAdapter -  1: node::DecodeWrite
 2: node_module_register
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::FatalProcessOutOfMemory
 5: v8::internal::Heap::MaxHeapGrowingFactor
 6: v8::internal::ScavengeJob::operator=
 7: v8::internal::ScavengeJob::operator=
 8: v8::internal::Heap::MaxHeapGrowingFactor
 9: v8::internal::Factory::NewFillerObject
10: v8::internal::ParseInfo::GetFlag
11: 000002AC950041C1

[2018-07-16 14:37:42.295] [ERROR] TestRunnerChildProcessAdapter - Child process exited with non-zero exit code 134. Last 10 message from the child process were:
        [2018-07-16 14:24:16.394] [TRACE] JestPromiseTestAdapter - Invoking Jest with config
...

[nicojs]

Ah, this is slightly different from the original error, as this happens in a _test runner child process_.

Stryker should be able to recover from that. We could identify this error and retry to run mutant in a fresh Testrunner process. In fact, there already is a decorator to do just that, but it isn't handeling this particular case: the RetryDecorator. I think we should do that, as well as investigate the jest runner

[nicojs]

Some news on this front. We can now reproduce this when running stryker on stryker itself. This was definitely _not_ the case earlier, so it seems to have happened due to some change on our side or one of our dependencies.

Unfortunately, figuring that out is not easy, as we don't run stryker on stryker in our CI build (it takes too long). We still need to configure a nightly build to mutation test all our plugins, which could double as a performance /load test as well. That way we would have caught this sooner.

18:35:20 (11696) WARN Sandbox Failed find coverage data for this mutant, running all tests. This might have an impact on performance: StringLiteral: ("") file://C:\z\github\stryker-mutator\stryker\packages\stryker\src\initializer\StrykerInitializer.ts:11:9
Mutation testing 11% (ETC 731s) 286/2399 tested (32 survived)
Mutation testing 14% (ETC 635s) 351/2399 tested (51 survived)
Mutation testing 18% (ETC 541s) 432/2399 tested (77 survived)
Mutation testing 23% (ETC 413s) 570/2399 tested (100 survived)
18:36:02 (11696) ERROR log4js A worker log process hung up unexpectedly { Error: read ECONNRESET
    at TCP.onread (net.js:659:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
18:36:02 (11696) ERROR TestRunnerChildProcessAdapter Child process exited with non-zero exit code 1. Last 10 message from the child process were:

Mutation testing 24% (ETC 422s) 597/2399 tested (109 survived)
18:36:05 (11696) ERROR TestRunnerChildProcessAdapter FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

18:36:05 (11696) ERROR TestRunnerChildProcessAdapter  1: 00007FF73C8282F5
 2: 00007FF73C804156
 3: 00007FF73C805890
 4: 00007FF73CCFAD5E
 5: 00007FF73CCFAC93
 6: 00007FF73CBD9CB4
 7: 00007FF73CBD0797
 8: 00007FF73CBCED1C
 9: 00007FF73CBD7975
10: 00007FF73CADBB21
11: 00007FF73CADC5E2
12: 00007FF73CABCC0B
13: 00007FF73CABC9B9
14: 00007FF73CABCA53
15: 00007FF73CB3AA42
16: 000000AE8DA041C1

18:36:05 (11696) ERROR log4js A worker log process hung up unexpectedly { Error: read ECONNRESET
    at TCP.onread (net.js:659:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
18:36:06 (11696) ERROR TestRunnerChildProcessAdapter Child process exited with non-zero exit code 134. Last 10 message from the child process were:

18:36:06 (11696) ERROR StrykerCli an error occurred Error: Test runner child process exited with non-zero exit code 134
    at ChildProcess.<anonymous> (C:\z\github\stryker-mutator\stryker\packages\stryker\src\isolated-runner\IsolatedTestRunnerAdapter.js:125:46)
    at ChildProcess.emit (events.js:182:13)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:237:12)

[nosideeffects]

I'm glad you have found a reproduction using Stryker's own source! I've been working around this issue by mutating small subsets of code while writing unit tests, when appropriate. If I run into this in one of these subsets, I'll be sure to post an update here on the code contents, to hopefully help pinpoint the underlying issue.

[simondel]

Ok, I figured something out. We transpile every mutant, which results in one or more source files that will be placed in the sandbox during testing. This transpiling is done in a separate process so we can run it while we run the tests. If you are not using any transpiler, the code just passes through. That means that you end up having a lot of files in memory.

I think this causes the memory issue of @nosideeffects because it crashes before the testing starts. 90.000 source files in memory could easily crash your process.

In the case of @JoshuaKGoldberg this may also cause the issue, but since you use typescript, the process is slower (since actual transpilation happens).

I was not able to reproduce the memory issue that we have with Stryker (which takes place inside a test runner).

[simondel]

@JoshuaKGoldberg Are you running on Windows?

[JoshuaKGoldberg]

Yup! Windows Server 2016.

[simondel]

Nice. Could you select the main process when you run stryker?

At the start, there should be only one NodeJS process in your terminal:

If you click it, it will stay selected as more child-processes are created for testing and transpiling:

Is it the main process that run out of memory or is it one of the child processes?

[JoshuaKGoldberg]

Main is 7412.

While the last message is Creating 12 test runners (based on CPU count):

At 1% (36/4471 tested):

At 3% (119/4471 tested):

[simondel]

Nice, PIDs. I should also turn that on. Unfortunately I don't see the huge spike in memory as I do on my machine so it looks like you're having a different issue.

Nonetheless, we should fix the issue with Stryker transpiling every mutant as fast as it can, without waiting for the test runners to catch up.

[simondel]

So, one solution would be to pause the MutantTranspiler if the process feels it's getting overrun with data. However, since this uses reactive programming, it would require us to give an Observable to the MutantTranspiler so it can provide a value based on the current value of the Observable, indicating if it should pause.

This does feel like a bit of an anti-pattern though..

[nicojs]

@JoshuaKGoldberg We found (one of) the memory leaks in Stryker. Since sinon 5, using sinon.stub() without using sinon.restore() is a memory leak. We used it here and there to create one-of-stubs. If you're doing something similar, you'll probably have a memory leak in your tests too. See my PR where I fix this issue in Stryker: #1044

This makes it kind of interesting. I think a lot of people have memory leaks in there unit tests. Usually, it's not a big deal. After unit testing, the process will end. What should we do to improve this situation? I think we should somehow detect that a test runner crashed _because of a memory leak_ and warn the user about it. For example at the end we could send the user to a web page with some common mistakes and how to fix them.

[JoshuaKGoldberg]

Ha, I was wondering if those tweets applied to this issue. Thanks for the find! We'll try to rework our unit tests and see if it helps.

But - why does a memory leak in tests apply here? Shouldn't the test runner recycle whatever environments are being leaked into? Why is the Node process' memory from e.g. previous Jest runs still a part of the test runner's memory space?

[nicojs]

This is for performance reasons. We reuse the test runner process. With mocha, we clear only _your_ modules from the node cache so Sinon will still be in de cache and happely clog your memory.

For jest, we leave the clearing up to the jest testrunner. Not sure what exactly is cleared there, but I imagine it's the same thing

[nosideeffects]

So is there a current workaround for my specific issue? Is there a way to throttle the test runners, or limit the number of files in memory concurrently?

[simondel]

@nosideeffects Unfortunately, not really. The issue is in the main process of Stryker which is being hammered by the total amount of mutations being made. One thing you could do is limit the amount of files you mutate in order to get less mutants.

@nicojs Thinks he has a way to throttle the code that is causing Stryker to run out of memory.

[JoshuaKGoldberg]

FWIW, I'm able to run Stryker without memory leaks by limiting the amount of files to be mutated and mutations to create, but I still see thee odd "ECONNRESET" log once in a while:

16:33:51 (32164) INFO ConfigReader Using stryker.conf.js in the current working directory.
16:33:54 (32164) INFO TypescriptConfigEditor Loading tsconfig file D:\Code\WebClient\tsconfig.json
16:33:56 (32164) INFO InputFileResolver Found 51 of 1423 file(s) to be mutated.
16:33:56 (32164) INFO InitialTestExecutor Starting initial test run. This may take a while.
16:35:05 (32164) INFO InitialTestExecutor Initial test run succeeded. Ran 261 tests in 1 minute 10 seconds (net 2075 ms, overhead 62403 ms).
16:35:05 (32164) INFO Stryker 499 Mutant(s) generated (148 Mutant(s) excluded)
16:35:05 (32164) INFO SandboxPool Creating 12 test runners (based on CPU count)
16:36:44 (32164) ERROR log4js A worker log process hung up unexpectedly { Error: read ECONNRESET
    at TCP.onread (net.js:656:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
16:36:45 (32164) ERROR log4js A worker log process hung up unexpectedly { Error: read ECONNRESET
    at TCP.onread (net.js:656:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
16:36:45 (32164) ERROR log4js A worker log process hung up unexpectedly { Error: read ECONNRESET
    at TCP.onread (net.js:656:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Mutation testing  [==================================================] 100% (ETC 0.0s) 499/499 tested (251 survived)

...

16:50:33 (32164) INFO HtmlReporter Your report can be found at: file:///D:/Code/NeverGonnaGiveYouUp/reports/mutation/html/index.html
16:50:44 (32164) INFO Stryker Done in 16 minutes 49 seconds.

[simondel]

@nicojs is working on this :)

[nicojs]

This issue should be fixed with #1376. We'll be releasing a version later today. I'll notify here when it is released

[nicojs]

We've just released [email protected], @nosideeffects and @JoshuaKGoldberg if you want, you can try it out.

We'll be releasing 1.0 tomorrow, so you can wait for that as well

[JoshuaKGoldberg]

Oh, congrats on the 1.0 @nicojs & team!

I no longer have access to the repository that demonstrated issues here... but @ian-craig might be interested... 😉