Reaction: Allocation failed - process out of memory

[joshuacox]

Expected behavior

Build a custom image from a default checkout of reaction with no modifications

Actual Behavior

CircleCI documents my build failure here

that is on the same commit as 8984 here which passed, yet mine fails.

I have another branch which is actually passing. However, it is using meteor 1.5.1

There are many other builds there too. I believe the only ones that have passed so far have been using meteor 1.5.1, like this one, which is identical to this one except the latter is 1.5.2.1

Steps to Reproduce the Behavior

Fork reaction, make a new circleCI account, add docker env vars, build from master

-or-

reaction init
cd reaction
reaction build mycustom

Versions

Node: 8.6.0
NPM: 5.3.0
Meteor Node: 4.8.4
Meteor NPM: 4.6.1
Reaction CLI: 0.14.2
Reaction: 1.5.0
Reaction branch: reaction-8984
Docker: 17.07.0-ce

There is also a forum post where I've been documenting this issue for a few weeks now.

I've got a good number of machines running ubuntu zesty and xenial, debian stretch, and arch linux that all exhibit this behavior (most of which have tons of available RAM), with circleCI also doing the same.

CirclleCI

branch | 1.5.1 | 1.5.2.1 | 1.5.2.2 |
------------ | :-----------: | -----------: | -----------: |
master | success | fail | fail |
marketplace | success | fail | fail |
v1.4.3 | success | fail | fail |
v1.5.0-rc.1 | fail | fail | fail |
v1.4.1 | success | fail | fail |

Travis-CI

branch | 1.5.1 | 1.5.2.1 | 1.5.2.2 |
------------ | :-----------: | -----------: | -----------: |
v1.4.1 | success | fail | fail |

[jshimko]

Try adding this to your Circle config... https://github.com/reactioncommerce/reaction/commit/cb6e9e1c80a0fe971a133958b1104f6a28c4dd88

And when local...

docker build --build-arg TOOL_NODE_FLAGS="--max-old-space-size=2048" -t custombuild .

[joshuacox]

Thanks to @jshimko we've got some successes in the circleCI builds:
https://github.com/jshimko/reaction/commit/1b45cdb96d93e4a874e515dba80afda9e5ae7416
I'm investigating further on the local builds

[joshuacox]

These numbers definitely need to be tweaked, I have a machine that will build with 2047, but not 2048. We have certainly got a working workaround. There is an npm module here that just adds this flag to all node calls. But I'm not ready to close this issue yet. There is a memory leak in building one of the node modules and that might get worse and prevent builds in the future, and is certainly going to affect our newcomers as they try out reaction.

[yairtal]

This workaround did the trick for me

[joshuacox]

the above works for reaction build, but for the other reaction commands you need to set that variable directly in your environment like so:

TOOL_NODE_FLAGS="--max-old-space-size=2048"  reaction test

[aaronjudd]

@jshimko is your recommendation to use docker build? Just wondering as we have reaction build and curious why / should we support both?

[joshuacox]

it appears so. Though, that is not a deprecation _per se_, which would be letting people know that this feature would not be supported in the future while it continues to work.

[jshimko]

reaction build sounded like a good idea at the time I added it because it was just an alias... back when no special options were required to get the build to work. At this point, it's way easier to just recommend everyone use the standard docker build command so we don't have to parse the --build-arg flags that are apparently now a requirement to get builds to work.

As of 0.16.1, the build command has been deprecated and there is a message that tells you to use docker build instead.

[jshimko]

To summarize this issue, it appears to be happening because the v8 engine in Node has fairly low default memory allotments for a single Node process. If you take a look at the Node source for v4.x (which the current release of Reaction uses), you can see the default is 700MB for 32bit machines and twice that (1.4GB) for 64bit machines (max_old_space_size is used to configure max_old_generation_size). Those defaults aren't terrible in most cases, but they're understandably not enough when a single Node process needs to build a giant app with tens of thousands of lines of code. The Reaction build is processing a _lot_ of files/code and that requires a lot of memory.

So, the current solutions for adjusting this in Reaction are either setting this environment variable in development...

# You can try changing the amount, but FYI, more is not necessarily better
export TOOL_NODE_FLAGS="--max-old-space-size=2048"

# start Reaction
reaction

# or all in one line
TOOL_NODE_FLAGS="--max-old-space-size=2048" reaction

Or this when you run a Docker build...

docker build --build-arg TOOL_NODE_FLAGS="--max-old-space-size=2048" -t mybuild .

Also worth noting is that some people have said that the update to Meteor 1.6 (with Node 8) has made this issue go away for them. So you could also try running our work-in-progress PR branch for Meteor 1.6 and see if the app starts without setting these flags. To test...

reaction init -b meteor-1.6 reaction-1.6

cd reaction-1.6

reaction

More info on max-old-space-size...

https://futurestud.io/tutorials/node-js-increase-the-memory-limit-for-your-process

https://stackoverflow.com/questions/16586705/configuring-v8s-memory-management-to-be-smart-for-a-node-js-process/16591881#16591881

[jshimko]

Also worth noting is Meteor didn't add support for 64bit Windows until a few weeks ago. This means the default Node memory allocation on Windows machines (700MB) is half of what everyone on 64bit machines gets by default. That certainly may explain why Windows users appear to run into this issue more often.

https://github.com/meteor/meteor/pull/9218

It is now available in Meteor 1.6, but Windows users should be aware that you need to reinstall Meteor from scratch to enable 64bit support. See the release blog post for more detail (Windows section is toward the bottom)...

https://blog.meteor.com/announcing-meteor-1-6-abf30f29e2c6

_Note that merely running meteor update from a 32-bit version of Meteor will still install a 32-bit version of Meteor 1.6, so you should use choco install meteor to get a fresh 64-bit version._

And note the updated install instructions for Windows...

https://www.meteor.com/install

[joshuacox]

@jshimko that definitely explains the low memory end on windows. However, there are still many issues where I've seen a 64-bit linux machine with monstrous amounts of ram fail with a ridiculous line like:

docker build --build-arg TOOL_NODE_FLAGS="--max-old-space-size=32048" -t mybuild .

of note approaching 32 gigs of ram should be way beyond what is necessary here, and yet I've seen that exact setup fail.

I think the best solution is to follow @jshimko and the reaction team with their circleCI builds, once you get it setup, it works just fine. But what happens when circleCI changes out their underlying architecture or switch server farms and all of our builds stop?

I'm certainly investigating other build solutions.

[jshimko]

@joshuacox yeah, as I mentioned in the code comments above, more is not necessarily better. For example, when it set it to 4096 on my MacBook Pro, it would just hang for over an hour, but when I set it to 2048 it would pass every time. So it appears that the trick is to only set it as high as necessary for the machine you're on. Not really an ideal user experience, but hopefully Meteor 1.6 solves that for most of us.

[joshuacox]

just a little extra info I've got two shells open at the moment and one is a reaction and during the process it will show:

cfs:gridfs: updating npm dependencies -- mongodb, gridfs-stream...

in my other shell I can watch -n1 'free -m' or htop or similar and watch memory continuously being consumed, it surpassed 5 GB on this machine. At some point gridfs appears to be satisfied and let's loose. @jshimko this was just done on a fresh checkout of the meteor-1.6 branch:

commit cd1322c77c5f77fe998c3924d4bc8e18296554d0 (HEAD -> meteor-1.6, origin/meteor-1.6)
Author: Jeremy Shimko <[email protected]>
Date:   Tue Nov 7 10:19:59 2017 -0500

    handle unhandled promise rejections

[machikoyasuda]

Documented here: https://docs.reactioncommerce.com/reaction-docs/master/deploying-reaction-using-docker#custom-build

[jshimko]

Since this issue has the above workarounds and is essentially a Meteor build tool issue, we're going to close this issue.