Fork-ts-checker-webpack-plugin: Heap out of memory `memoryLimit` issue in ForkTsCheckerWebpackPlugin

[marcusjwhelan]

Current behavior

Upgraded to start using electron-webpack and upgraded all our packages to newer versions.

Expected behavior

Typescript and webpack should compile code

Steps to reproduce the issue

use electron webpack with typescript?

Issue reproduction repository

Environment

• fork-ts-checker-webpack-plugin: [version from the package.json]
  in node modules under /electron-webpack-ts/package.json
  fork-ts-checker-webpack-plugin: 0.5.2
• typescript: [version from the package.json]
  typescript: 3.4.5
• tslint: [version from the package.json]
  "tslint": "^5.14.0",
• webpack: [version from the package.json]
  "webpack": "4.28.4"
• os: [e.g. Ubuntu 19.04]
  Ubuntu 18.04

<--- Last few GCs --->

[8366:0x3f04ad0]    23325 ms: Mark-sweep 2042.0 (2085.7) -> 2041.3 (2086.2) MB, 734.9 / 0.0 ms  (average mu = 0.102, current mu = 0.013) allocation failure scavenge might not succeed
[8366:0x3f04ad0]    24138 ms: Mark-sweep 2043.2 (2086.2) -> 2042.5 (2087.2) MB, 807.7 / 0.0 ms  (average mu = 0.055, current mu = 0.007) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x309d6aa5be1d]
    1: StubFrame [pc: 0x309d6aa5d1df]
Security context: 0x124119b1e6e9 <JSObject>
    2: getIntersectionType(aka getIntersectionType) [0x1a0a8652b3a1] [/home/marcus/work/ClientElectron/node_modules/typescript/lib/typescript.js:~39985] [pc=0x309d6b65f1d2](this=0x0d5af00026f1 <undefined>,types=0x3352c04a57b9 <JSArray[23]>,aliasSymbol=0x0d5af00026f1 <undefined>,aliasTypeArguments=0x0d5af00026f1 <undefine...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 2: 0x8dc55c  [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 5: 0xec7bf2  [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
12: 0x309d6aa5be1d 

Error from running electron-webpack dev

┏ Renderer -------------------

  Type checking and linting aborted - probably out of memory. Check `memoryLimit` option in ForkTsCheckerWebpackPlugin configuration.

┗ ----------------------------
┏ Main -----------------------

  Type checking and linting aborted - probably out of memory. Check `memoryLimit` option in ForkTsCheckerWebpackPlugin configuration.

┗ ----------------------------

[johnnyreilly]

Could be worth upping your memoryLimit

https://github.com/Realytics/fork-ts-checker-webpack-plugin/blob/master/README.md#options

[marcusjwhelan]

@johnnyreilly I moved the memory to 8192 but still failed with the same error just takes a little longer. I believe after talking to someone from electron-webpack that my issue could be a memory leak having to do with Typescript itself.

[marcusjwhelan]

@johnnyreilly I also made a simple repo with what I used. It works just fine. However as I add more reducers/Actions the application then crashes with the same error.

[phryneas]

@marcusjwhelan could you share that repro so I could take a peek? (No guarantees though.) Sometimes there are just TS patterns that lead to a looot of memory consumption.

[marcusjwhelan]

@phryneas https://github.com/electron-userland/electron-webpack/issues/280
https://github.com/marcusjwhelan/testingproj

While using electron-webpack is how I get this issue. That link to that project, that project works just fine, however, as you add more redux/actions to the structure of the application it will memory leak out, No matter the memoryLimit set.

[phryneas]

First thing I am noticing: you are including all .d.ts in node_modules which will include these in the typecheck - usually those should be automatically included as they are the default value of typeRoots. Does everything keep working when you remove them from include in the tsconfig.json? If not, try using compilerOptions.typeRoots instead of include for those.

PS: just invoking tsc --noEmit takes about 80 seconds on my system. That project has less than 100 files, that should take at most 5 seconds. Something is very off there, but it has nothing to do with fork-ts-checker-webpack-plugin

PPS: tsc --noEmit --skipLibCheck is down to 50 seconds. That's a first step.

[johnnyreilly]

I'm considering a skipLibCheck tattoo. Or I would be if I wasn't already filled with regret about the noImplicitAny one I got one drunken night in 2014

[phryneas]

Apart from skipLibCheck and the node_modules thing I don't see anything glaring. That project is not using much advanced type magic, so I don't understand why it could take so long. But this is definitely a TypeScript problem - maybe you could open an issue over there? A project of that size should not consume that many resources.

[phryneas]

Another thing that just came up in a conversation with colleague: you should not import from @material-ui/icons directly.

Do import Close from "@material-ui/icons/Close"; instead of import {Close} from '@material-ui/icons'

maybe that helps

PS: the same thing, albeit with a lower impact applies to @material-ui/core

[phryneas]

Also, you might consider downgrading to TS 3.3: https://github.com/Microsoft/TypeScript/issues/30663

[FishOrBear]

I just upgraded to the new version today and I have the same problem.

I tried to increase the memory, and roll back the ts version, there is no way to solve it.

  new ForkTsCheckerWebpackPlugin({ checkSyntacticErrors: true, memoryLimit: 5000 }),

[phryneas]

Could you please specify, which packages exactly you updated and from which to which version?

• did you only upgrade fork-ts-checker-webpack-plugin?
• did you also upgrade typescript?

[FishOrBear]

I am testing.

When I upgraded from 0.5.2 to the latest version.
     "fork-ts-checker-webpack-plugin": "^0.5.2",

Originally it took 11s, and it became 13s after the upgrade.

[FishOrBear]

Node12
Typescript 3.3.4000
The latest fork-ts-checker-webpack-plugin

There will be no memory explosion problems.

I tried what it was caused.

[FishOrBear]

This is a magical problem, I also encountered, I tried to roll back the package, but it will also trigger the problem of insufficient memory.

And I can compile smoothly on another computer.

[FishOrBear]

tsc --noEmit
I use this script to get the same result, should it be the error caused by typescript?

[phryneas]

Exactly. That's why I'm asking about your TS version - 3.4 seems to have some problems regarding performance.

tsc --noEmit
I use this script to get the same result, should it be the error caused
by typescript?

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[FishOrBear]

3.3.x If properly installed, it will not cause memory leaks.

Originally I may not have correctly rolled back the typescript version.

Type checking can be done when npm i [email protected] is used correctly.

So I think it might be a typescript 3.4.x and 3.5.x error.

Because I run tsc --noEmit, I get the same result.

[FishOrBear]

It should be the problem of this.
https://github.com/microsoft/TypeScript/issues/31227

use
npm i [email protected] to solve the problem

[marcusjwhelan]

I tried all versions after 3.3.400 and they all had the same issue. So sticking with TS 3.3.4 for now

[edwingustafson]

Working flawlessly for me after upping the memoryLimit on the plugin itself via React App Rewired. (Because it forks, adjusting memory settings on the parent process don't help.)

config-overrides.js
https://gist.github.com/edwingustafson/ee786fe26c6a7fae394321a892698f64

Discovered my confusion by inspecting the command line via ps wwwaux on my development box: it showed the plugin's process getting the default memory limit as a command line option.

[fc]

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

[Chnapy]

Hi,

I also have a memory issue on type checking, since I upgraded one of my project lib (react-scripts from 2.1 to 3.X (tried with 3.0.1, 3.2.0, 3.3.0).

Note that react-scripts uses fork-ts-checker-webpack-plugin 3.1.0 in its webpack config.
I use typescript 3.3.3 (also tried with 3.7.5).

When I try a build, it takes like 5min then fails with this error stack:

<--- Last few GCs --->

[62388:0x10280e000]   171064 ms: Scavenge 2017.5 (2093.5) -> 2002.2 (2095.0) MB, 6.5 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure
[62388:0x10280e000]   171096 ms: Scavenge 2017.7 (2095.0) -> 2002.4 (2097.5) MB, 6.3 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure
[62388:0x10280e000]   171124 ms: Scavenge 2017.9 (2097.5) -> 2002.6 (2099.5) MB, 7.3 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1bf3a39dbe3d]
Security context: 0x112bbff9e6e9 <JSObject>
    1: trySymbolTable(aka trySymbolTable) [0x112b05636ab1] [/Users/b046lj/Projects/afds_ext/afds-webapp/frontend/node_modules/typescript/lib/typescript.js:~33453] [pc=0x1bf3a4be076a](this=0x112bfd8826f1 <undefined>,symbols=0x112bf143fd71 <Map map = 0x112b4f684359>,ignoreQualification=0x112bfd8826f1 <undefined>)
    2: getAccessibleSymbolChainFromSymbolTable(...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003d035 node::Abort() [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 2: 0x10003d23f node::OnFatalError(char const*, char const*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 3: 0x1001b8e15 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 4: 0x100586d72 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 5: 0x100589845 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 6: 0x1005856ef v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 7: 0x1005838c4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 8: 0x10059015c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 9: 0x1005901df v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
10: 0x10055fb24 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
11: 0x1007e7e04 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
12: 0x1bf3a39dbe3d

I tried to increase the memory limit to 9000 as a test, in this case the process takes like 10min before failing (the node process takes until 9go of memory...).

There is an issue with my project types, but I have no idea where to check, the error message doesn't help me at all.

There is a way to add logs during type checking, or anything else that can help me to find the issue root cause ?
Thanks

[ahrbil]

@Chnapy I'm having the same issue
I have turned off

"strict": false,
"strictNullChecks": false,
"strictPropertyInitialization": false,

and it's working
using typescript 3.7.5
and fork-ts-checker-webpack-plugin 4.0.3

[JosipReh]

Did someone managed to fix this issue?

[piotr-oles]

Could you try fork-ts-checker-webpack-plugin@alpha? I've published a new version which potentially could resolve this issue :)

[jorisw]

I unfortunately am unable to utilize fork-ts-checker-webpack-plugin version 5.x to set the memoryLimit in my Create React App TypeScript project, because Create React App presently (3.x, 4.x-alpha) targets the fork-ts-checker-webpack-plugin 4.x configuration schema:

ValidationError: Invalid configuration object. ForkTsCheckerWebpackPlugin has been initialized using a configuration object that does not match the API schema.
 - configuration has an unknown property 'silent'. These properties are valid:
   object { async?, typescript?, eslint?, formatter?, issue?, logger? }
 - configuration.typescript should be one of these:
   boolean | object { enabled?, memoryLimit?, configFile?, context?, build?, mode?, compilerOptions?, diagnosticOptions?, extensions?, profile?, … }
   Details:
    * configuration.typescript should be a boolean.
      -> Enable TypeScript reporter.
    * configuration.typescript should be an object:
      object { enabled?, memoryLimit?, configFile?, context?, build?, mode?, compilerOptions?, diagnosticOptions?, extensions?, profile?, … }

This thread did help me however in setting the memory limit and compiling in isolation:
NODE_OPTIONS=--max-old-space-size=8192 node_modules/.bin/tsc --noEmit

... which helped me determine that I have a memory problem regardless of the memory limit: 2GB, 4GB, 8GB - same result.

So to anyone trying to override the memory limit, investigate the cause rather than to suppress the symptoms.

[seanaye]

@jorisw have you found a solution to this? Getting the same error

[jorisw]

In my case, I was importing style object definitions targeting ThemeUI, each typed as a SxStyleProp, then doing composition with the imported objects to create different styles for different states, like so:

export const someStyle: SxStyleProp {
   maxWidth: 300,
}

import {
  someStyle
} from './styles'

const manyStyles: SxStyleProp = {
  ...someStyle,
  ...(isActive ? someMoreStyles : {}),
  // ...
}

Apparently importing and composing these typed objects made the typechecker go into a memory leak of some sort. Removing the : SxStyleProp typing from each of my exported styles in styles.ts, made the problem go away.

As you can see, a pretty specific use case, so our problems are likely to be very different.

[jorisw]

Turns out the problem I was having, was related to TypeScript being bad at handling use of the spread (...) operator on objects that have union types. Using Object.assign() instead fixes it.

Discussion: https://github.com/system-ui/theme-ui/discussions/1162#discussioncomment-59480

[damienallen]

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

For anyone seeing this issue in cases where it doesn't seem related to a specific commit (i.e. roll-back doesn't work), I would like to highlight this solution. Due to some unrelated permissions issues I had temporarily made a copy of node_modules which led to this out-of-memory issue.

[radiantone]

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

For anyone seeing this issue in cases where it doesn't seem related to a specific commit (i.e. roll-back doesn't work), I would light to highlight this solution. Due to some unrelated permissions issues I had temporarily made a copy of node_modules which led to this out-of-memory issue.

Thank you for sharing this solution. It worked for me.