Rocket.chat: Javascript Heap Memory Overload when Building with Meteor

Created on 7 Sep 2018  路  15Comments  路  Source: RocketChat/Rocket.Chat

Description:

When building out locally with meteor I am running into a Javascript heap memory overload when trying to build out this package.

Here is the script I'm using to build this out:

```#!/bin/bash
set -x
set -euvo pipefail
IFS=$'\n\t'

Requires Node.js version 4.x

Do not run as root

DEPLOY_DIR=~/Desktop/convergent/build
TOOL_NODE_FLAGS=--max-old-space-size=4096

BUILD

meteor npm install --save babel-runtime --save bcrypt
meteor npm run postinstall

on the very first build, meteor build command should fail due to a bug on emojione package (related to phantomjs installation)

the command below forces the error to happen before build command (not needed on subsequent builds)

set +e
meteor add rocketchat:lib
set -e

meteor build --server-only --directory $DEPLOY_DIR

RUN

cd $DEPLOY_DIR/bundle/programs/server
npm install

cd $DEPLOY_DIR/bundle
NODE_ENV=production \
PORT=3000 \
ROOT_URL=http://localhost:3000 \
MONGO_URL=mongodb://localhost:27017/rocketchat \
MONGO_OPLOG_URL=mongodb://localhost:27017/local \
node main.js
```

Steps to reproduce:

  1. Enter develop branch of rocket chat
  2. Run build bash script

Expected behavior:

Meteor bundle created

Actual behavior:

Javascript heap memory overload

Server Setup Information:

  • Version of Rocket.Chat Server: 0.69.1
  • Operating System: osx high sierra
  • Deployment Method: meteor local
  • Number of Running Instances: local
  • DB Replicaset Oplog:
  • NodeJS Version: 8.11.4
  • MongoDB Version: 4.0.1

Additional context

Relevant logs:

```+ set -euvo pipefail
IFS=$'\n\t'

  • IFS='
    '

Requires Node.js version 4.x

Do not run as root

DEPLOY_DIR=~/Desktop/convergent/build

  • DEPLOY_DIR=/Users/jamesbarlow/Desktop/convergent/build
    TOOL_NODE_FLAGS=--max-old-space-size=4096
  • TOOL_NODE_FLAGS=--max-old-space-size=4096

BUILD

meteor npm install --save babel-runtime --save bcrypt

  • meteor npm install --save babel-runtime --save bcrypt
    npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but npm-shrinkwrap.json was generated for lockfileVersion@0. I'll try to do my best with it!

[email protected] install /Users/jamesbarlow/Desktop/convergent/rc2/node_modules/bcrypt
node-pre-gyp install --fallback-to-build

[bcrypt] Success: "/Users/jamesbarlow/Desktop/convergent/rc2/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
npm WARN [email protected] requires a peer of chai@>= 2.1.2 < 4 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of postcss-syntax@^0.10.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of postcss-syntax@^0.10.0 but none is installed. You must install peer dependencies yourself.

  • [email protected]
  • [email protected]
    updated 2 packages and audited 13647 packages in 15.963s
    found 119 vulnerabilities (20 low, 87 moderate, 12 high)
    run npm audit fix to fix them, or npm audit for details
    meteor npm run postinstall
  • meteor npm run postinstall

Rocket.[email protected] postinstall /Users/jamesbarlow/Desktop/convergent/rc2
cd packages/rocketchat-katex && npm i

audited 2 packages in 1.027s
found 0 vulnerabilities

on the very first build, meteor build command should fail due to a bug on emojione package (related to phantomjs installation)

the command below forces the error to happen before build command (not needed on subsequent builds)

set +e

set -e

  • set -e

meteor build --server-only --directory $DEPLOY_DIR

  • meteor build --server-only --directory /Users/jamesbarlow/Desktop/convergent/build
    up to date in 1.499se plugins \
    Building for web.browser |
    <--- Last few GCs --->

[39210:0x103800000] 255673 ms: Mark-sweep 1452.8 (1534.9) -> 1452.8 (1503.9) MB, 2310.3 / 0.1 ms last resort GC in old space requested
[39210:0x103800000] 257984 ms: Mark-sweep 1452.8 (1503.9) -> 1452.8 (1503.9) MB, 2311.3 / 0.1 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x16fc4d3a5879
1: ReferencedIdentifier [/Users/jamesbarlow/.meteor/packages/ecmascript/.0.10.9.1xi3zkn.ato9m++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/@babel/traverse/lib/scope/index.js:~128] [pc=0x3e0d47ce8d1](this=0x16fc7c27fcd9 ,path=0x16fc5e03bcc1 ,state=0x16fc7c27fcd9

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
build.sh: line 22: 39210 Abort trap: 6 meteor build --server-only --directory $DEPLOY_DIR

```

picture Jdesk

Most helpful comment

we might have a troubleshooting section on our README and add things like this.. wdyt?

picture sampaiodiego on 15 Oct 2018
👍2

All 15 comments

How long after it starts there is it crashing? Also are you running this build script directly on osx? I'd recommend building on a Linux box. While yes some of us develop on osx our build scripts are really targeted for linux

geekgonecrazy picture geekgonecrazy on 7 Sep 2018

Any system specs required as meteor seems to be real memory and cpu intensive. I am trying to build out directly on osx.

Jdesk picture Jdesk on 7 Sep 2018

It looks like you are setting the memory limit to 4gb. Do you actually have that much free?

On osx I think it does consume more memory when building. Linux is really your best bet when building

geekgonecrazy picture geekgonecrazy on 7 Sep 2018

I am setting it to 4gb - definitely have it free. I'm going to try building out on linux and will see how that goes. When you guys build, what are the specs of your linux boxes? how much memory do you have?

Jdesk picture Jdesk on 7 Sep 2018

I'm getting the same thing on linux - created an ec2 instance for this build. T2-Large with 8gb of ram and 4cpus.

Note, I've also removed the memory limit completely after setting it at 4gb the first run. Both builds resulted in the error below.

```<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1fbe9f025879
1: ReferencedIdentifier [/home/ubuntu/.meteor/packages/ecmascript/.0.10.9.10owkt8.g8vj++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/@babel/traverse/lib/scope/index.js:~128] [pc=0x279f623546b1](this=0x1a718d60fd89 ,path=0x19c2e93eb309 ,state=0x1a718d60fd89

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
2: 0x8c21ec [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
3: v8::Utils::ReportOOMFailure(char const, bool) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const, bool) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
5: v8::internal::Factory::NewUninitializedFixedArray(int) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
6: 0xd4b133 [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object*, v8::internal::Isolate) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
8: 0x279f61b842fd
build.sh: line 21: 13333 Aborted (core dumped) meteor build --server-only --directory $DEPLOY_DIR```

Jdesk picture Jdesk on 7 Sep 2018

Same issue here, I also built it on a CentOS 7 Box, and facing the same issue.

YuhuaDeng picture YuhuaDeng on 7 Sep 2018

Just a bit of background on why we are trying to do this. We forked the repo and want to customize some things. I'm DevOps so I won't be doing the development. Our deploy method is docker, but this build is giving me a huge headache.

I've got this to build without using the --save-bcrypt flag (which I added as a recommendation from the logs saying that we are using a pure javascript implementation and this runs 3x slower. Try running npm install --save-bcrypt when building"

We have some customizations we want to implement. I'm not sure if adding to the codebase directly or creating applications that interact with the rocketchat api is a better solution. I'm leaning towards the latter. One example of something we are trying to accomplish is having the ability to create scheduled messages that will be broadcast to specific channels. Another example of something we want to do is the ability to audit specific users and be able to export their chats to csv.

Jdesk picture Jdesk on 7 Sep 2018

@rocketchat/core anyone have any ideas?

Regarding fork or use api's. We definitely prefer using integrations, apps, and our API.

The ability to schedule things with our apps framework is on the roadmap for fairly soon I believe. Once that's added you could actually just write an app to add that additional functionality. Idea being customize with out having to maintain a fork.

geekgonecrazy picture geekgonecrazy on 8 Sep 2018

@Rocketchat/core just checking in to see if you guys have any ideas for me :-)

Jdesk picture Jdesk on 10 Sep 2018

recently I had to use this env var to be able to build rocket.chat from within a VM:
```bash
export TOOL_NODE_FLAGS="--max_old_space_size=4096"

sampaiodiego picture sampaiodiego on 10 Sep 2018

@sampaiodiego I actually tried the same thing - and it's still failing when I use the flag --save-bcrypt (which I need apparently to avoid using a pure javascript implementation which is 3x slower).

Jdesk picture Jdesk on 10 Sep 2018

@Jdesk hm.. I don't recall having any of our scripts using --save-bcrypt my understanding is when you do npm install after the bundle is produced it will install and compile bcrypt if it can.

So you should be good to just drop that instead of continuing to be stuck over that.

geekgonecrazy picture geekgonecrazy on 11 Sep 2018

Same here, @sampaiodiego export TOOL_NODE_FLAGS="--max_old_space_size=4096"helped

localguru picture localguru on 25 Sep 2018
👍2

Is this still an issue or can it be closed?

reetp picture reetp on 11 Oct 2018

we might have a troubleshooting section on our README and add things like this.. wdyt?

sampaiodiego picture sampaiodiego on 15 Oct 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mddvul22 picture mddvul22  路  3Comments
djeber picture djeber  路  3Comments
antn89 picture antn89  路  3Comments
danpospisil picture danpospisil  路  3Comments
ghost picture ghost  路  3Comments