Jest: Controlling memory usage of node-haste when loading tests

[jeremiahrhall]

I have an app that is built in CircleCI, which has a 4GB memory limit that terminates the build when the threshold is exceeded.

My build hits this threshold when Jest runs. According to the output from CircleCI, there are 32 processes open at the time running node-haste/lib/analyze-changed.js, cumulatively using about 3.5GB of memory. Add in the Jest process which uses around 350mb and some OS overhead and I'm over.

Jest has the -w option but, node-haste's maxProcesses option is set to os.cpus().length in HasteModuleLoader.

Can this be made configurable?

[jeremiahrhall]

Here's a minimal implementation: https://github.com/facebook/jest/compare/master...jeremiahrhall:dev/node-haste-max-processes-arg

[cpojer]

We already have a maxWorkers option that should take care of this, no? It is passed on to the TestRunner.

[jeremiahrhall]

The HasteModuleLoader isn't actually passed that option right now. In fact, TestRunner does not pass the second options param to the loader.

[jeremiahrhall]

I ended up changing my code to use the maxWorkers option but, it still needed to be passed through to the module loader: https://github.com/facebook/jest/compare/master...jeremiahrhall:dev/node-haste-max-workers-arg

[cpojer]

That's weird. Feel free to send a Pull Request :)

[jeremiahrhall]

:+1:

[chriddyp]

@jeremiahrhall - did you have any luck in reducing the memory usage besides limiting maxWorkers? I've gone over Circle CI's 4GB of RAM with 4 workers. Reducing to 3 workers seems to lower the memory enough to not cause circleci to crash (died unexpectedly).

[cpojer]

Also check out #599, which replaces node-haste with an entirely new implementation.

[jeremiahrhall]

I only tried setting maxWorkers to 4 and that dropped memory usage to well below 4GB for me.