Typescript: Very high memory usage

[mvestergaard]

I've been having issues with the typescript compiler eating up lots of memory.

A run of tsc in my current project reaches around 1.1-1.2GB memory before completion.

This has been causing issues when using awesome-typescript-loader through webpack. It actually crashes after a little while because it exceeds a memory limit (1.5GB i believe)
This seems to be caused in part by a memory leak in ATL, but the memory use of tsc on its own is bordering on extreme.
I have an issue open on that project as well https://github.com/s-panferov/awesome-typescript-loader/issues/227

I would suspect however, that with a project large enough, tsc could exceed the memory limit on its own.

I can provide an example project if needed.

Node: 6.5.0
TS: 2.0.2

[mhegazy]

can you try typescript@next and see if the issue persists?

[vladima]

yes please, repro project will definitely be very helpful

[mvestergaard]

@mhegazy appears to be the same with typescript@next

@vladima [removed - use gist from ChrisJamesC below]
npm install && typings install then npm start or tsc

npm start uses awesome-typescript-loader as described, it should most likely crash with an out of memory error if you leave it be a while, you may need to do an edit to a file though to cause a re-build... Keep an eye on the memory, it should keep climbing.

[mvestergaard]

Odd thing is, running my tests with npm t both runs faster, and doesn't use very much memory.
The tests run through ts-node. Not sure what that project does differently.

[vladima]

repro can be narrowed down to typings for enzyme, seems that most of the compilation time/memory is spent there

vladima@vladima-W520:~/sources/playground/enzyme-test$ node_modules/.bin/tsc --diagnostics --listFiles node_modules/@types/enzyme/index.d.ts 
/home/vladima/sources/playground/enzyme-test/node_modules/typescript/lib/lib.d.ts
/home/vladima/sources/playground/enzyme-test/node_modules/@types/react/index.d.ts
node_modules/@types/enzyme/index.d.ts
Files:              3
Lines:          22201
Nodes:         106921
Identifiers:    39075
Symbols:      1126162
**Types:         937485**
**Memory used: 1104806K**
I/O read:       0.00s
I/O write:      0.00s
Parse time:     0.26s
Bind time:      0.19s
Check time:    13.42s
Emit time:      0.01s
Total time:    13.88s
vladima@vladima-W520:~/sources/playground/enzyme-test$ popd
~/sources/playground/10719/product-designer
vladima@vladima-W520:~/sources/playground/10719/product-designer$ node_modules/.bin/tsc --diagnostics -p .
Files:             112
Lines:           55706
Nodes:          203971
Identifiers:     78405
Symbols:       1157256
**Types:          949701**
**Memory used:  1083038K**
I/O read:        0.01s
I/O write:       0.01s
Parse time:      0.62s
Bind time:       0.43s
Check time:     16.10s
Emit time:       0.36s
Total time:     17.51s
vladima@vladima-W520:~/sources/playground/10719/product-designer$ 

[mvestergaard]

@vladima Do you have any ideas yet which part of the enzyme types may be causing this? I'm wondering if there's any kind of temporary work around i could implement, since this is kind of a show stopper for me atm.

[mvestergaard]

I've done a bit of digging on my own.
It appears it has something to do with overloads, and possibly generics.

Commenting out this block from CommonWrapper saves about 270MB memory:

    parents<P2>(component: ComponentClass<P2>): CommonWrapper<P2, any>;
    parents<P2>(statelessComponent: StatelessComponent<P2>): CommonWrapper<P2, {}>;
    parents(props: EnzymePropSelector): CommonWrapper<any, any>;
    parents(selector: string): CommonWrapper<HTMLAttributes<{}>, any>;
    parents(): CommonWrapper<any, any>;

Commenting out other similar blocks give similar results more or less.

[mvestergaard]

Usage of StatelessComponent seems to be the main cause.
Removing all usages of that from the enzyme type brought memory usage down from a little over 1GB to around 300MB. There must be other things too because excluding the enzyme type entirely brings it down to 158MB.

[MastroLindus]

I have the same exact issues in the typescript@next version.
Compiling my project takes normally less than 2 seconds.
If I just add a import {shallow} from 'enzyme' compilation time spikes to 20 seconds.
Using webpack + tsloader, but even by invoking tsc directly I can see the difference in time.
tsc -w just gets me a heap out of memory error.

[MastroLindus]

By the way I downgraded to 1.8.10 and adding or removing enzyme doesn't affect so much my building time (around 2seconds in both scenarios).

It's really with the 2.1.x version that it gets 10 times slower and the memories usage goes to the roof, and it is a shame since I am in love with most of the features in the 2.x branch

[ChrisJamesC]

In case someone wants to investigate some more, I have created a gist with the setup that I found was the smallest possible to reproduce the issue: https://gist.github.com/ChrisJamesC/02b6d3394202e1fff8c6664d12cfbf5a

I have deleted as much as I could from the enzyme type declaration without getting rid of the issue. I unfortunately couldn't narrow it down to some more specific fragment of the code.

[ChrisJamesC]

@MastroLindus A quick fix to use typescript 2.0 even with this issue:

• npm uninstall @types/enzyme
• use regular require instead of typescript imports

-import { mount } from "enzyme";
+const { mount }  = require("enzyme")

As expected, the drawback is that you don't have typescript working with enzyme. On the other hand you can use typescript 2.

[avivshafir]

I can confirm the same issue occurred to me when
using enzyme typings with ts-loader for webpack
I got out of memory errors. Using typescript 2.0.0 and installing enzyme typings using npm.

[mhegazy]

I have a work around for this issue in https://github.com/DefinitelyTyped/DefinitelyTyped/pull/11524.

[mvestergaard]

Thanks for the workaround @mhegazy It definitely makes it better.

There still seems to be something about the enzyme types that take up a lot of memory though.
Here's my project with the enzyme types, including your work around:

Files:              84
Lines:           57211
Nodes:          216334
Identifiers:     82652
Symbols:        374444
Types:          204438
Memory used:   340482K
I/O read:        0.02s
I/O write:       0.02s
Parse time:      0.46s
Bind time:       0.27s
Check time:      3.07s
Emit time:       0.16s
Total time:      3.96s

And here it is with a simple placeholder typing file i wrote:

Files:              84
Lines:           56611
Nodes:          214693
Identifiers:     81900
Symbols:        151302
Types:           29029
Memory used:   162083K
I/O read:        0.01s
I/O write:       0.02s
Parse time:      0.46s
Bind time:       0.28s
Check time:      1.35s
Emit time:       0.12s
Total time:      2.21s

The number of symbols and types seem to give an indication.

This is the custom enzyme types used, a simple implementation that makes my build pass:

declare module 'enzyme' {
  export function shallow(...args: any[]): any;
  export function mount(...args: any[]): any;
  export function mount<T, K>(...args: any[]): any;

  interface ReactWrapper<T, K> {
    unmount(): void;
    ref(...args: any[]): any;
    instance(): any;
    setProps(props: any): any;
  }
}

[mhegazy]

Your declaration is much simpler than the one on definitely typed. so i am not surprised. the one on definitely typed as many generic types and that results in instantiations when comparing structures. so more types, more memory and more time. however the time and memory you show seems reasonable and matches my expectation.

The original issue with the @types package was that it used in multiple places HTMLAttributes<{}>, currently the compiler does not cache these instantiations, as it thinks every {} is a different type. we will need to fix this in the compiler, but my change to the declaration file removes the different uses of HTMLAttributes<{}> and avoids the issue.

If this declaration file for enzym matches your needs, by all means use it. but i do not think we can change the one on definitely typed.

[mvestergaard]

@mhegazy I think you missed my point.

What I'm trying to illustrate is that I still think there's something about the enzyme types causing a high memory footprint.

Here are all of the types installed in my project:

    "@types/axios": "^0.9.32",
    "@types/chai": "^3.4.33",
    "@types/chai-as-promised": "0.0.29",
    "@types/enzyme": "^2.4.35",
    "@types/jquery": "^2.0.32",
    "@types/lodash": "^4.14.36",
    "@types/mocha": "^2.2.32",
    "@types/moment": "^2.13.0",
    "@types/mousetrap": "^1.5.32",
    "@types/nock": "^8.0.33",
    "@types/react": "^0.14.37",
    "@types/react-addons-css-transition-group": "^0.14.17",
    "@types/react-dom": "^0.14.17",
    "@types/sinon": "^1.16.31",
    "@types/sinon-chai": "^2.7.27"

So i find it suspicious that without the enzyme types memory is at about 160MB and with it 340MB.

Almost half of the used memory is coming from enzyme alone. I was only using the placeholder as a workaround.

[mhegazy]

I understand. if you look at the enzyme declaration file you will find multiple layers of abstractions using generic types. (It feels too complicated; I do not think I can pass a judgment here though since i am not familial with the framework).

TypeScript has a structural type system. so comparing two entities require the type system to do a full structural comparison. every time two types need to be compared (for inheriting or for checking overloads, or just assignability etc..) the compiler needs to check their structure, that means the source and the target need to have the same set of properties with matching types; if sees a method, it needs to check all its signatures between the source and the target, for every signature, every parameter type as well as return type should be checked; every time there is a reference to a generic type, it needs to be instantiated with the correct generic type arguments; then the whole algorithm is repeated recursively, and so on and so forth. hope that makes sense.

[vladima]

Closing this issue since #11934 resolves the main issue with enzyme typings. #11941 is filed to track solution for the problem in general