Meteor: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

[Jayuda]

Hello Guys,
I have update Meteor to 1.4.2.3 but on server show error :

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

<--- Last few GCs --->

  252470 ms: Mark-sweep 1340.5 (1458.0) -> 1339.6 (1458.0) MB, 566.4 / 0 ms [allocation failure] [scavenge might not succeed].
  253033 ms: Mark-sweep 1339.6 (1458.0) -> 1340.4 (1458.0) MB, 562.2 / 0 ms [allocation failure] [scavenge might not succeed].
  253612 ms: Mark-sweep 1340.4 (1458.0) -> 1337.4 (1458.0) MB, 579.0 / 0 ms [last resort gc].
  254190 ms: Mark-sweep 1337.4 (1458.0) -> 1339.1 (1458.0) MB, 578.4 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2628e1eb4629 <JS Object>
    1: slowToString [buffer.js:413] [pc=0x270af61dbc15] (this=0x1dfd38efdf69 <an Uint8Array with map 0x2f5cd779741>,encoding=0x2628e1ebd761 <String[4]: utf8>,start=3777,end=401427)
    2: toString [buffer.js:~440] [pc=0x270af6595805] (this=0x1dfd38efdf69 <an Uint8Array with map 0x2f5cd779741>)
    3: arguments adaptor frame: 3->0
    4: /* anonymous */(aka /* anonymous */) [/bundle/bundle/progr...

Any sugesstion to fixed this issue ?
Thanks anyway

[vu-bui]

I'm having the same problem with meteor build command. Setting NODE_OPTIONS="--max-old-space-size=4096" does not help.

UPDATE:
Trying to rerun the command several times will eventually work. It looks like a continuous/incremental building process, but I cannot do that on CI.

[Josh-ES]

Having the same problem with the meteor test command, but oddly enough it only occurs from within a docker container that gets used in our tech stack for CI. Reverting to a 1.4.1 release of Meteor resolves the problem.

[ansaries]

I am also getting the same error, my meteor version is 1.4.2.3. The development server is working fine, but when i try to deploy on galaxy, it generates the above error upon minifying code. Any Suggestions?

[sakulstra]

Getting this now too(on ubuntu 16.04) :/

Awkwardly i have a few meteor applications on my system and they all work fine except for one - the same application is working fine for others.

[danlg]

same issue for me with 1.4.2.1

Exception in setInterval callback: MongoError: connection 164 to localhost:27011 timed out
    at Object.Future.wait (/opt/carmel/app.1.2.7.b501/programs/server/node_modules/fibers/future.js:449:15)
    at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1024:47)
    at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1058:22)
    at SynchronousCursor.map (packages/mongo/mongo_driver.js:1068:10)
    at SynchronousCursor.fetch (packages/mongo/mongo_driver.js:1092:17)
    at Cursor.(anonymous function) (packages/mongo/mongo_driver.js:907:44)
    at Cursor.cursorProto.(anonymous function) (packages/meteorhacks_kadira.js:3347:32)
    at Cursor.kadira_Cursor_fetch [as fetch] (packages/meteorhacks_kadira.js:3739:32)
    at packages/smartix:rss/server/parser.js:8:12
    at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
    - - - - -
    at Function.MongoError.create (/opt/carmel/app.1.2.7.b501/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:29:11)
    at Socket.<anonymous> (/opt/carmel/app.1.2.7.b501/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/connection.js:176:20)
    at Socket.g (events.js:260:16)
    at emitNone (events.js:67:13)
    at Socket.emit (events.js:166:7)
    at Socket._onTimeout (net.js:332:8)
    at _runOnTimeout (timers.js:524:11)
    at _makeTimerTimeout (timers.js:515:3)
    at Timer.unrefTimeout (timers.js:584:5)
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

<--- Last few GCs --->

1865813596 ms: Scavenge 982.8 (1075.5) -> 982.7 (1075.5) MB, 2.1 / 0 ms (+ 16.7 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
1865902073 ms: Mark-sweep 982.7 (1075.5) -> 979.7 (1075.5) MB, 88477.4 / 4 ms (+ 720.6 ms in 320 steps since start of marking, biggest step 26.2 ms) [last resort gc].
1865994344 ms: Mark-sweep 979.7 (1075.5) -> 980.7 (1075.5) MB, 92270.0 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x25b122eb4629 <JS Object>
    2: write [/opt/carmel/app.1.2.7.b501/programs/server/npm/node_modules/meteor/dbeath_feedparser/node_modules/feedparser/node_modules/sax/lib/sax.js:227] [pc=0x1c1f3b67a8e5] (this=0x27f5725c9c99 <an SAXStream with map 0x33fdc0929511>,data=0x10284f004101 <Very long string[4720]>)
    3: _transform [/opt/carmel/app.1.2.7.b501/programs/server/npm/node_modules/meteor/dbeath_feedparser/node_module...

[danlg]

And another case (no error nor stack trace before FATAL)

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

<--- Last few GCs --->

1994363930 ms: Scavenge 982.7 (1076.0) -> 982.7 (1076.0) MB, 0.7 / 0 ms (+ 6.6 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
1994366534 ms: Mark-sweep 982.7 (1076.0) -> 982.4 (1076.0) MB, 2603.6 / 0 ms (+ 21.5 ms in 58 steps since start of marking, biggest step 6.6 ms) [last resort gc].
1994368809 ms: Mark-sweep 982.4 (1076.0) -> 919.8 (1076.0) MB, 2275.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3a9812cb4629 <JS Object>
    2: new constructor(aka Request) [/opt/lg/app.1.2.7.b501/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:142] [pc=0x976c2dc4953] (this=0x22b5902b3c51 <a Request with map 0xacaffffd741>,options=0x22b5902b3c19 <an Object with map 0x16ab16e2c111>)
    4: request(aka request) [/opt/lg/app.1.2.7.b501/programs/server/npm/node_modules/meteor/http/node_modules/request/in...

[chrisknu]

Running into this issue with CI on Bitbucket Pipelines. Any resolution (besides rolling back the version) would be most welcome

trace (if helpful)

Talking to Galaxy servers at https://galaxy.meteor.com
Deploying your app...
  (  Determining active plugins                ... )
  (  Building the application                  ... )
  (  Building for web.browser                  ... )
  (  Processing files with fourseven:scss ...  ... )
  (  Linking                                   ... )
  (  Building for web.browser                  ... )
  (  Linking                                   ... )
  (  Building for web.browser                  ... )
  (  Minifying app code                        ... )
<--- Last few GCs --->
  532933 ms: Scavenge 1389.9 (1456.1) -> 1389.9 (1456.1) MB, 5.9 / 0 ms (+ 1.0 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  534315 ms: Mark-sweep 1389.9 (1456.1) -> 1387.6 (1456.1) MB, 1381.5 / 0 ms (+ 2.6 ms in 2 steps since start of marking, biggest step 1.6 ms) [last resort gc].
  535837 ms: Mark-sweep 1387.6 (1456.1) -> 1389.8 (1456.1) MB, 1521.9 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x3f6790b4629 <JS Object>
    1: doit(aka doit) [0x3f6790041b9 <undefined>:5743] [pc=0x2724815f2c4] (this=0x3f6790041b9 <undefined>)
    2: print [0x3f6790041b9 <undefined>:~5736] [pc=0x27257480f39] (this=0x30bdbfd5efd1 <an AST_VarDef with map 0x8917bfc1621>,stream=0xc2bb265e489 <an Object with map 0x8917bfa3749>,force_parens=0x3f6790041b9 <undefined>)
    3: arguments adaptor frame: 1->2
    4: /* anonymous */(aka /* an...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

[alxmhe]

I had this problem since yesterday. I tried to build my app (web.browser with --server-only actually) in a parent directory (../build for example).
I just figured out it failed because I built my app in the current directory (build/ for example) first, which will work the first time, but will crash the following builds.
The directory build/ was empty but it crashed the app nonetheless.
Just remove the bad directory and burn the trash, then retry.
Hope this solution is the one you are looking for.

[NitroBAY]

My problem was fixed by removing console.log()

[chrisknu]

Bumped meteor version to 1.4.2.5, seeing a slightly different error
( Minifying app code ... ) <--- Last few GCs ---> 695508 ms: Scavenge 1389.8 (1456.1) -> 1389.8 (1456.1) MB, 0.9 / 0 ms (+ 0.9 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep]. 696800 ms: Mark-sweep 1389.8 (1456.1) -> 1384.4 (1456.1) MB, 1291.6 / 0 ms (+ 2.2 ms in 2 steps since start of marking, biggest step 1.3 ms) [last resort gc]. 698081 ms: Mark-sweep 1384.4 (1456.1) -> 1389.7 (1456.1) MB, 1280.8 / 0 ms [last resort gc]. <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x1a4439ab4629 <JS Object> 2: _visit [0x1a4439a041b9 <undefined>:~1600] [pc=0x2ba44f698c35] (this=0x1c4c88d6a981 <a TreeWalker with map 0x37846e786f79>,node=0x1a4adde08c99 <an AST_Dot with map 0xc927ab1e149>,descend=0x15608fb08569 <JS Function (SharedFunctionInfo 0x2828f8e09a99)>) 3: _walk [0x1a4439a041b9 <undefined>:~1213] [pc=0x2ba44f6bb62c] (this=0x1a4adde08c99 <an AST_Dot with map 0xc927ab1e149>,visitor=0x1c4... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

[bluefangs]

Any possible resolution for this? I keep running into this even during development (hot code push) and it's getting very frustrating. Each time I'll have to explicitly set !#/bin/env node --max-old-space-size=xxxx
Sometimes it works, sometimes it does not.

[NitroBAY]

@bharathkeshav12 did you try to remove all your console.log() otherwise I don't know.

[bluefangs]

@NitroBAY - Yes. On my client code, i've overridden my console to nothing. That way it does not generate any console messages. On the server i've commented out all the console.log()s. I'd also removed minify js which I'd read elsewhere. But right now, none of these are of much use.

[NitroBAY]

Maybe try to also comment out console.log() client side ?

[mhewett]

I'm having the same issue with 1.4.2.6. No console.log() on the client side. No console.log() on the server side executing. Still have the out of memory issue the first time I make a call from the client to the server.

(more info): I traced the memory and it is repeatedly allocating a large array of objects. The allocation grows larger each time even though the GC is collecting most of the objects. My program is very small and it wasn't even doing anything but Node CPU was at 100% and memory grew to 3GB. The app seriously wasn't supposed to be doing ANYTHING. The only thing that would need to allocate large amounts of objects would be Mongo or data tables. I disabled my largest datable and it had no effect on the behavior so I suspect a bug in the Mongo driver.

[bluefangs]

@mhewett - can you try:

!#/bin/env node --max-old-space-size=4096 --gc-interval=100
Meteor

And see if it helps?

[mhewett]

@bharathkeshav12 ...
I already added that. It didn't help. I've used the v8-profiler to verify that it is indeed Mongo calls that are maxing out CPU and memory. I'm now using the mongo shell to trace which DB is involved. It's the one that I disabled the datable on, which means it should never be accessed at all. Still tracing...

[chrisknu]

Might this issue be related to the linux kernel OOM bug? http://status.meteor.com/incidents/v0nc2kz3z699

[mhewett]

I fixed it with: meteor remove autopublish.

Meteor was trying to read my largest database into memory in order to publish it. I thought Meteor collections were smarter than that! It turns out that with Datatables you don't need to publish/subscribe the collections that Datatables manages. (I'm using the aldeed:tabular module to manage the datatables.)

I removed unnecessary publish/subscribe statements, removed autopublish, and re-enabled the Datable on my largest DB and everything is working great now.

For others, the following article was helpful for CPU profiling: https://www.dynatrace.com/blog/how-to-track-down-cpu-issues-in-node-js/ He (Daniel Khan) also has a good article on tracing memory usage. FYI, for memory tracing you need to use the 'memwatch-next' module because 'memwatch' no longer works with 2017 versions of node.

[anpoonampardeshi]

Facing the same issue -

   Processing files with angular2-compil...  /
<--- Last few GCs --->

  194108 ms: Scavenge 1410.9 (1493.1) -> 1410.9 (1493.1) MB, 1.1 / 0 ms (+ 1.3 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  194984 ms: Mark-sweep 1410.9 (1493.1) -> 1383.2 (1465.4) MB, 875.8 / 0 ms (+ 2.2 ms in 2 steps since start of marking, biggest step 1.3 ms) [last resort gc].
  195862 ms: Mark-sweep 1383.2 (1465.4) -> 1383.2 (1465.4) MB, 878.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x19889a8b4629 <JS Object>
    1: curContext [/home/nexususer/.meteor/packages/meteor-tool/.1.4.2_3.b4rrtk++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/babylon/lib/tokenizer/index.js:~124] [pc=0x2ae83b6d935e] (this=0x2ec602c61d81 <a Parser with map 0x1354eff7c8c1>)
    2: nextToken [/home/nexususer/.meteor/packages/meteor-tool/.1.4.2_3.b4rrtk++os.linux.x86_64+web.browser+web.cor...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Aborted (core dumped)

[yanndebelgique]

me too... exact same error as @anpoonampardeshi

[tzapu]

i was getting that error as well, tried everything i could think of but no go (removed meteor-desktop, removed ios platform that was using cordova, removed .meteor/local, removed node_modules, removed meteor installation)

in the end, here s my error (happens on 1.4.2.3 , 1.4.2.7 and 1.4.3.1)

=> Started proxy.                             
=> A patch (Meteor 1.4.2.7) for your current release is available!
   Update this project now with 'meteor update --patch'.
=> Started MongoDB.                           
   Building for web.browser                  -










<--- Last few GCs --->

  195374 ms: Scavenge 1375.9 (1456.7) -> 1375.9 (1456.7) MB, 1.1 / 0 ms (+ 4.3 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  197285 ms: Mark-sweep 1375.9 (1456.7) -> 1375.8 (1456.7) MB, 1910.8 / 0 ms (+ 12.5 ms in 31 steps since start of marking, biggest step 4.3 ms) [last resort gc].
  199204 ms: Mark-sweep 1375.8 (1456.7) -> 1375.8 (1456.7) MB, 1919.8 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3a642e5b4629 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/Users/alex/.meteor/packages/ecmascript/.0.6.1.puk6rj++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-traverse/lib/path/family.js:112] [pc=0x1534c798f0b3] (this=0x3a642e5041b9 <undefined>,_=0x22a747ada471 <a Node with map 0x2dbd92b1e3b1>,i=0)
    2: arguments adaptor frame: 3->2
    3...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

[tzapu]

guys, in my case it seems i had the production 'build' folder in the meteor's project root folder. i was able to run meteor again without the above error after removing it.
maybe it helps someone

[egoldblum]

I ran into meteor build tool dying when running in an environment without swap (jenkins inside docker on ECS in my case).

Apparently the build toolchain uses a different environment variable to pass arguments to the underlying node.js process than the meteor runtime.

https://github.com/meteor/meteor/blob/22d3fa3f0a8509c3c11ae9aca3b200a6bd4827d4/meteor#L135

For me, setting TOOL_NODE_FLAGS="--max-old-space-size=4096" allowed my build to complete successfully (and use more than the default V8 heap size limit of ~1.4GB).

[guns2410]

unable to build or run meteor even with TOOL_NODE_FLAGS="--max-old-space-size=4096"

   Building for web.browser                  \
<--- Last few GCs --->

  383092 ms: Scavenge 1394.5 (1457.8) -> 1394.5 (1457.8) MB, 8.3 / 0 ms (+ 3.1 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  385386 ms: Mark-sweep 1394.5 (1457.8) -> 1390.6 (1457.8) MB, 2293.4 / 0 ms (+ 8.1 ms in 24 steps since start of marking, biggest step 3.1 ms) [last resort gc].
  387716 ms: Mark-sweep 1390.6 (1457.8) -> 1394.2 (1457.8) MB, 2330.1 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x18fef97b4629 <JS Object>
    1: get [/Users/gunjansoni/.meteor/packages/ecmascript/.0.6.3.cn7ogz++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-traverse/lib/path/index.js:~76] [pc=0x1747f73f675f] (this=0x5b9a53a9419 <JS Function NodePath (SharedFunctionInfo 0x2f59def56b41)>,_ref=0x3fdfb3dc6031 <an Object with map 0x931c3f53ba1>)
    2: node [/Users/gun...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
/bin/sh: line 1:  1848 Abort trap: 6           meteor build ../deploy/app --directory --architecture os.linux.x86_64
make: *** [build] Error 134

[yanickrochon]

Same thing. I need to push some critical updates and bugfix, and I can no longer do so because of this error. I have tried

$ TOOL_NODE_FLAGS=\"--max-old-space-size=2048\" mup deploy

but the build process fail, still.

Update

I have increased the memory limit to 4096... and watched my monitor for a solid 5 minutes before the build was finally successful. What a few updates on npm packages and Meteor 1.4.4.1 has to do with requiring me to octuple my memory usage for building?? :open_mouth:

Update 2

While the deployment seem to have succeeded, the app fails on the client with JavaScript errors. I'll try to revert back a previous version of Meteor because, clearly, 1.4.4.1 fails for me.

Update 3

I confirm that downgrading to 1.4.3.2 did it for me. If needs be, I can investigate further, but sniffing through a few megabytes of minified JS code is not what I desire the most, right now. :stuck_out_tongue:

[abernix]

Based on the versions you're suggesting a problem with, I suspect you're running into a two-factor problem. First, I think UglifyJS is failing to transpile your code due to some more modern JavaScript constructs which it doesn't (yet!) support – again this is due to a deficiency in UglifyJS.

To overcome this (known) shortcoming, in Meteor 1.4.4.1, Babili is used as a fall-back for when Uglify fails. Unfortunately, my experience has been that Babili, while very ECMAScript-feature-complete, is quite heavy (read: CPU-and-memory intensive) when trying to minify large files. So I'm curious, what's the output from the following command, when run from the root of your app project directory:

ls -la .meteor/local/build/programs/*/packages/modules.js

[yanickrochon]

$ ls -la .meteor/local/build/programs/*/packages/modules.js
-r--r--r-- 1 yanick yanick   286868 Apr 12 12:00 .meteor/local/build/programs/server/packages/modules.js
-r--r--r-- 1 yanick yanick 20057853 Apr 12 12:02 .meteor/local/build/programs/web.browser/packages/modules.js

[abernix]

To further test, can you try doing a mup deploy after you've removed standard-minifier-js from your project?

[yanickrochon]

You mean, upgrading to 1.4.4.1, and removing standard-minifier-js, then do a mup deploy?

[Floriferous]

@abernix removing standard-minifier-js from my project does solve the issue, any potential problems with keeping it out?

[Azeures]

Any updates on this? I am still facing this issue while running Meteor 1.4.4.1 with standard-minifier-js removed.

[yanickrochon]

It did not work for me either, and I have no time to look for a fix right now. I downgraded and it works just fine.

[kokokenada]

+1. @stubailo, are you guys on this? Seems pretty serious

[etyp]

This is really crippling our workflow 😞 . Is there anything we can do to help push this forward?

[ffxsam]

Happening with Galaxy deployments too! I have two other projects with standard-minifier-js in them and it doesn't happen there.

Talking to Galaxy servers at https://galaxy.meteor.com
Deploying your app...
   Minifying app code                        |
<--- Last few GCs --->

  280971 ms: Mark-sweep 1393.4 (1454.8) -> 1394.4 (1454.8) MB, 1007.4 / 0 ms [allocation failure] [GC in old space requested].
  281938 ms: Mark-sweep 1394.4 (1454.8) -> 1393.4 (1454.8) MB, 966.6 / 0 ms [allocation failure] [GC in old space requested].
  282908 ms: Mark-sweep 1393.4 (1454.8) -> 1391.8 (1454.8) MB, 970.7 / 0 ms [last resort gc].
  283903 ms: Mark-sweep 1391.8 (1454.8) -> 1393.4 (1454.8) MB, 994.5 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x317239db4629 <JS Object>
    1: addExtra [/Users/samh/.meteor/packages/standard-minifier-js/.2.0.0.1py145r++os+web.browser+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/babylon/lib/index.js:~1640] [pc=0x2398d41653b9] (this=0x2f6002429c31 <a Parser with map 0x3c6aa7b81a39>,node=0x26cc4407e929 <a Node with map 0x1094a122bea9>,key=0x2648474d62b1 <String[13]: parenthesized>,val=0x317...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

[abernix]

@ffxsam What version of Meteor are you using?

[ffxsam]

1.4.4.1. Had standard-minifier-js 2.0.0, removed and it trying deployment now.

I have another project at 1.4.4.1 with standard-minifier-js 1.2.1 and it deploys fine, though I did get a warning about exceeding 32000 characters (???). Another project at 1.4.2.3 has standard-minifier-js 1.2.1, deploys no problem. (these are all on Galaxy)

[ffxsam]

Removing standard-minifier-js did the trick. This won't break anything, right? I wonder why the other projects with this package deploy without issue. It must be some perfect storm of packages or code size or something that's making it break.

-r--r--r--  1 samh  staff    62M Apr 26 22:55 .meteor/local/build/programs/web.browser/packages/modules.js

That's pretty huge. My other project is at about 54MB.. I hope it doesn't get to a point where it croaks too.

[abernix]

@ffxsam You can safely ignore the warning, see #8592. That is fixed upstream, but not released yet.

What's happening is likely what I explained in this comment. To continue the experimentation of the status of the UglifyJS "harmony" branch (which means, it's aiming for ES6 support, something the official harmony package does _not_ support), I've updated abernix:standard-minifier-js to use the new logic which the official Meteor standard-minifier-js package has (specifically, how it falls back to Babili when Uglify fails) but with the bleeding edge harmony branch.

I'm curious if you have success by adding the following (and not using standard-minifier-js):

meteor add abernix:[email protected]

[ffxsam]

@abernix I'm happy to test out your version of the minifier. Though at this point I've ripped out standard-minifier-js and things still seem to work. What happens if I don't add a minifier back in? Is minification simply not going to take place?

[abernix]

@ffxsam Yes, minification will simply not take place.

[ffxsam]

Ok, I deployed using your minifier. Actually, before that, I deployed with no JS minifier.. I imagine that would result in massive unminified code in production (good thing it was only deployed to a private staging server). Not good for bandwidth nor security. 😉 It worked fine.

[kokokenada]

I was able to build with no minifier, but still had the problem with abernix:[email protected]

[chrisknu]

Bumped abernix:[email protected] to 2.0.4, however my galaxy deploy timed out after about 2 hours (no logs) (from bitbucket CI)

Local builds and deploys are fine from my local machine, with either minifier

Has anyone had success integrating recent versions of Meteor with Bitbucket Pipelines CD?

[abernix]

@chrisknu Try running your project locally with meteor --production (which simulates production-style bundling and minification) and see if you can reproduce a similar failure.

(M/t chrisbridgett)

[chrisknu]

@abernix if your last comment was aimed at me, running locally with the production flag is fine. no errors/issue

[abernix]

@chrisknu It was. If it's working with --production locally the problem might lie with something else. I would suggest you contact Galaxy Support!

[sebakerckhof]

I'm also getting this error. Setting TOOL_NODE_FLAGS=\"--max-old-space-size=4096\" seems to solve it. But come on guys:

Processing a bit of javascript shouldn't take up so many gigs of memory.

[sebakerckhof]

Ok, after increasing the memory limit I got other issues from babeli, basically bugs in babili since compilation using uglify-es works.

I also created a butternut minifier plugin (if you want to try it out: seba:[email protected]). Butternut claims to be 3x faster than uglifyjs (and 10x faster than babeli). However, it also uses ridiculous amounts of memory. And since it's so new, you probably want to use it with the check option, which reduces the speed advantage.

I can completely compile my applicaiton using uglify-es without it ever having to go to babili and without memory problems.

[macrozone]

got the same problem on meteor 1.5-rc.11 when building on gitlab-ci:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

<--- Last few GCs --->

  323052 ms: Mark-sweep 1363.2 (1456.7) -> 1365.1 (1456.7) MB, 814.1 / 0 ms [allocation failure] [GC in old space requested].
  323859 ms: Mark-sweep 1365.1 (1456.7) -> 1366.6 (1456.7) MB, 806.9 / 0 ms [allocation failure] [GC in old space requested].
  324676 ms: Mark-sweep 1366.6 (1456.7) -> 1359.7 (1456.7) MB, 816.8 / 0 ms [last resort gc].
  325490 ms: Mark-sweep 1359.7 (1456.7) -> 1362.7 (1456.7) MB, 814.0 / 0 ms [last resort gc].

[igor90007]

as for me (windows 10, meteor 1.5), helps set TOOL_NODE_FLAGS=--max-old-space-size=2047 . Not ...4096, not 2048. Only 2047 or less.

[pandabrand]

Has anyone found a definitive solution for this?
I'm deploying to Heroku on Meteor 1.4.2.7 and it started failing with this. I've tried adding the Node params to no effect. This is the error I'm getting and I'm not well-versed enough to parse where I should look next.

remote: 
remote: <--- Last few GCs --->
remote: 
remote:   360471 ms: Scavenge 1387.1 (1456.2) -> 1387.1 (1456.2) MB, 0.6 / 0 ms (+ 0.9 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
remote:   361776 ms: Mark-sweep 1387.1 (1456.2) -> 1383.3 (1456.2) MB, 1304.7 / 0 ms (+ 2.4 ms in 2 steps since start of marking, biggest step 1.5 ms) [last resort gc].
remote:   363033 ms: Mark-sweep 1383.3 (1456.2) -> 1387.1 (1456.2) MB, 1256.9 / 0 ms [last resort gc].
remote: 
remote: 
remote: <--- JS stacktrace --->
remote: 
remote: ==== JS stack trace =========================================
remote: 
remote: Security context: 0x110a5e6b4629 <JS Object>
remote:     1: def_variable [0x110a5e6041b9 <undefined>:~3415] [pc=0x31f91d786164] (this=0x167f5329a131 <an AST_Defun with map 0x32861d7d0821>,symbol=0x167f5329fe89 <an AST_SymbolFunarg with map 0x173185a2e4d1>)
remote:     2: visit [0x110a5e6041b9 <undefined>:~3241] [pc=0x31f911725b9f] (this=0x369efd622441 <a TreeWalker with map 0x39dd29c15d49>,node=0x167f5329fe89 <an AST_SymbolFunarg with map 0x173185a2e4d1>...
remote: 
remote: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
remote: /app/tmp/buildpacks/2a6b3cd8e392368665e034e859b6dd73daffd5ad6e476e155af3667bd21fdc765346a576e32db1450dd74e597a117a5b1c7e397b50e2d583268b716843bbe97a/bin/compile: line 107:   435 Aborted                 HOME="$METEOR_DIR" "$METEOR_DIR/.meteor/meteor" $ARGS

Any help would be great, my production deploy is broken and I'm past my deadline.

[abernix]

@pandabrand Do you experience the same problem if you replace standard-minifier-js with abernix:standard-minifier-js (more recommended) or if you remove standard-minifier-js entirely (less recommended)?

[pandabrand]

I didn't, this seems to be a red herring for me. My issues were actually in my tests not here but this seemed to be the closest to my error. Sorry about the confusion.

[fadomire]

replace standard-minifier-js with abernix:standard-minifier-js fix the issue

but with abernix:standard-minifier-js we can't use bundle-visualizer it seems

it throws the following error in browser console bundle-visualizer: Couldn't load stats for visualization. Are you using standard-minifier-js >= 2.1.0 as the minifier?

[abernix]

@fadomire (and others!):

I just published abernix:[email protected] which uses the latest uglify-es (this should be the new version of the harmony branch of the uglify-js repo). This new version also brings in the latest changes which support the bundle-visualizer package and should work properly if you update bundle-visualizer to version 1.0.3 (which now supports other minifiers thanks to https://github.com/meteor/meteor/pull/8774).

Please let me know how it works! We still haven't published an official standard-minifier-js which uses uglify-es due to occasional regressions we see, but as the uglify-es project continues to evolve we will continue to reevaluate. See #8698 for updates on that!

[fadomire]

working for me 👍

[pgiani]

Work for me also

[ExcelNet-Public]

Worked for me also

[denodaeus]

Tried this on CircleCI, didn't work for me.

<--- Last few GCs --->

  350065 ms: Scavenge 1394.3 (1453.7) -> 1394.3 (1453.7) MB, 1.8 / 0 ms [allocation failure].
  350066 ms: Scavenge 1394.3 (1453.7) -> 1394.3 (1453.7) MB, 1.6 / 0 ms [allocation failure].
  351178 ms: Mark-sweep 1394.3 (1453.7) -> 1391.1 (1453.7) MB, 1111.6 / 0 ms (+ 5.1 ms in 35 steps since start of marking, biggest step 0.5 ms) [last resort gc].
  352270 ms: Mark-sweep 1391.1 (1453.7) -> 1394.2 (1453.7) MB, 1092.1 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x8ae042b4629 <JS Object>
    1: new constructor(aka Buffer) [buffer.js:~50] [pc=0x91712923f52] (this=0x2881ee2187c9 <a Buffer with map 0x17dc95a176b9>,arg=0x2881ee2187a1 <Very long string[4960]>,encoding=0x8ae042bda41 <String[4]: utf8>)
    3: _combineFiles [/home/ubuntu/.meteor/packages/meteor-tool/.1.5.0.1lxziix++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/isobuild/import-scanner.js:449] [pc=0x917...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

export PATH="$PATH:$HOME/.meteor"
meteor lint
 died unexpectedly

Action failed: meteor lint

[kwuite]

Adding the abernix:standard-minifier-js package worked for us locally but circle ci just wouldn't build our production bundle. Luckily we finally found the problem with a conflict between the fourseven version used by mozfet:autoform-materialize imported package called mozfet:[email protected].

If you are using fourseven and the materialize package in your project then you might want to investigate your version file.

[damonmaria]

I was hoping that Meteor 1.5.1 would fix this (it upgraded some of the relevant packages) but alas... no :(

[damonmaria]

@xonuzofa Agreed. It's an issue with the new minifier that Meteor is using. But the touted advantage of Meteor is that it 'just works' out of the box. Having to set an obscure Node env var (after finding a solution to a weird error dozens of pages down a Github issue) doesn't fit with that. If being unable to build is a common problem then Meteor should handle it in a built-in way.

Why doesn't Meteor have this flag set in its startup script?

[chrisknu]

This solution only works with machines that have higher than 4gig of ram. Unfortunately some CI systems limit the available memory to 2gig (bitbucket pipelines). If your app is moderately large, it'll choke without recourse.

[suezzo]

When I set TOOL_NODE_FLAGS to the 4096 and type meteor in my cmd it immediately exits. I can't even see meteor help. When I set it to the lower value for example 2048 it runs normally and fixes FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory. My computer has 8GB RAM and is using Windows 10 64bit. Does anyone have similar issue?

[monkeyman888]

export TOOL_NODE_FLAGS="--max_old_space_size=4096" worked for me. Thanks!

[danilaml]

Unfortunately using --max_old_space_size=4096 makes meteor exit immediately and using anything less just results in the fatal error. I'm on Win 10 x64 with 16GB of RAM. This made meteor completely unusable for me.

[rhymn]

Removing standard-minifier-js solves the memory problem. But not using it causes a usability problem, making the application too slow. Using abernix:standard-minifier-js instead does not solve the problem for me.

I'm running heroku machines with 1GB memory. Now trying to downgrading from Meteor 1.5.1 back to 1.4.4.1.

[rhymn]

I ended up running Meteor 1.5.1 with standard-minifier simply removed. It does not slow down too much actually

[sebakerckhof]

@xonuzofa Minification should absolutely not be considered a security practice.

[sebakerckhof]

On that I agree, since developing publicly on Github gives you the same level of security like minifying your code.

[yomexzo]

@xonuzofa "security through obscurity is no security"

[danilaml]

YOU ARE PROTECTING YOUR CODE!

No you are not. It's a very dangerous and wrong way to think. You are protecting your code by only exposing the stuff to the client that is not crucial to your business.

If you can't event see the difference between minifying code (those aim should only be just that - reduce the code size without changing it's functionality/performance), obfuscation (which might be viable in some cases as an additional security measure to increase the costs of some action against your property) and using a secure authentications methods, I can only hope that you are not responsible for the information security in your company/whatever field you are working in.

Anyway, this discussion is becoming more and more off-topic to this issue and I feel like it's getting more and more personal with each comment. I only want for this issue to be fixed and I believe that such debates are not contributing to this in any way.

[abernix]

@xonuzofa Please stop continuing the off-topic debate about whether minification is for security or not. Minification is important for reasons other than security and this issue is meant to track a problem with the minification process. Let's keep it about that and not argue about whether or not minification is security. Thank you.

[Floriferous]

Just updated to meteor 1.5.2 and the error has reappeared again on Circle CI.

[abernix]

@Floriferous What other packages were updated in the process and what version of Meteor were you upgrading from? Have you tried to see if the problem goes away if you remove standard-minifier-js? (Just to be clear, this is not recommended as a permanent solution, but just to test.)

[Floriferous]

I was upgrading from 1.5.1, I attached a screenshot of the updated packages, there were a few more core packages as well. I've switched the minifier a few times between yours and the official one.

I was actually able to solve it (I think) by adding the TOOL_NODE_FLAGS environment variable to my circle.yml file.

[abernix]

@Floriferous To narrow down the problem to the minifier though, and to gain confidence that a future update to the official standard-minifier-js (as is slated for Meteor 1.6 thanks to #8698), it would still be useful to know if your problem went away without adding TOOL_NODE_FLAGS and removing standard-minifier-js (completely, not just replacing it with mine).

[Floriferous]

I'll try to do that, but I'd like to have a couple builds work properly to have a control group. Circle CI is now running out of memory with yarn run test-CI died unexpectedly, and a warning from Circle CI:

Your build has exceeded the memory limit of 4G on 1 container. The results of this build are likely invalid. We have taken a snapshot of the memory usage at the time, which you can find in a build artifact named memory-usage.txt. The RSS column in this file shows the amount of memory used by each process, measured in kilobytes.

Here's the memory-usage.txt if it is of any use:

   PID   RSS %CPU COMMAND
 16109 3431880 100 /home/ubuntu/.meteor/packages/meteor-tool/.1.5.2.ecibw9++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node --expose-gc --max_old_space_size=4096 /home/ubuntu/.meteor/packages/meteor-tool/.1.5.2.ecibw9++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/index.js test --settings settings-dev.json --once --driver-package dispatch:mocha
 16860 455008 89.4 /home/ubuntu/.meteor/packages/meteor-tool/.1.5.2.ecibw9++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node /tmp/meteor-test-runb4d8ma/.meteor/local/build/main.js
 12241 183352 0.0 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
 16098 98152  0.1 node /home/ubuntu/.yarn/bin/yarn.js run test-CI
 16138 57292  0.9 /home/ubuntu/.meteor/packages/meteor-tool/.1.5.2.ecibw9++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/mongodb/bin/mongod --bind_ip 127.0.0.1 --port 3001 --dbpath /tmp/meteor-test-runb4d8ma/.meteor/local/db --oplogSize 8 --replSet meteor --nojournal
  1478 51056  0.2 /usr/bin/mongod --config /etc/mongod.conf
 16537 36120  0.4 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch
    42 20052  0.0 Xvfb :99 -screen 0 1280x1024x24
 12308 16940  0.0 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
  1417  4556  0.0 xfwm4 --daemon
  2781  4048  0.0 sshd: ubuntu [priv] 
  2835  3236  0.0 sshd: ubuntu@pts/7  
  1839  3064  0.0 /usr/sbin/sshd -D
     1  2924  0.0 /sbin/init
 12313  2760  0.0 postgres: 9.5/main: autovacuum launcher process                                                                             
  1416  2608  0.0 /usr/lib/x86_64-linux-gnu/xfce4/xfconf/xfconfd
  1647  2412  0.0 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
  5142  1964  0.0 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 116:123
 12310  1916  0.0 postgres: 9.5/main: checkpointer process                                                                                    
 12314  1816  0.0 postgres: 9.5/main: stats collector process                                                                                 
 16523  1708  0.0 su couchdb -c /usr/bin/couchdb
 12311  1704  0.0 postgres: 9.5/main: writer process                                                                                          
 12312  1700  0.0 postgres: 9.5/main: wal writer process                                                                                      
 16565  1516  0.0 /lib/systemd/systemd-logind
  1407  1460  0.0 /lib/systemd/systemd-udevd --daemon
  1483  1220  0.0 /usr/sbin/irqbalance
 16876  1168  0.0 ps -e -o pid,rss,%cpu,command --sort -rss
 16566  1152  0.0 rsyslogd
 16525  1084  0.0 dbus-daemon --system --fork
  1452  1044  0.0 cron
  1414   988  0.0 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
  1549   960  0.0 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.4.1 --dhcp-range 10.0.4.2,10.0.4.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative
  7622   872  0.0 /sbin/getty -8 38400 tty1
  1420   868  0.0 /sbin/getty -8 38400 tty4
  1428   868  0.0 /sbin/getty -8 38400 tty3
  7617   864  0.0 /sbin/getty -8 38400 console
  1427   856  0.0 /sbin/getty -8 38400 tty2
 11989   748  0.0 /bin/sh /usr/bin/mysqld_safe
  1413   732  0.0 dbus-launch --autolaunch 54f9f16cebc74faf7c49864956cb7375 --binary-syntax --close-stderr
 16108   668  0.0 sh -c TEST_BROWSER_DRIVER=nightmare SERVER_TEST_REPORTER=xunit XUNIT_FILE=$CIRCLE_TEST_REPORTS/reports/unit.xml meteor test --settings settings-dev.json --once --driver-package dispatch:mocha
 15618   660  0.0 /bin/sh /home/ubuntu/.yarn/bin/yarn run test-CI
   343   652  0.0 upstart-socket-bridge --daemon
  1403   652  0.0 upstart-udev-bridge --daemon
 16724   652  0.0 sh -s disksup
  1451   644  0.0 acpid -c /etc/acpi/events -s /var/run/acpid.socket
 16561   644  0.0 upstart-file-bridge --daemon
  6798   524  0.0 ssh-agent
  1453   160  0.0 atd

[Floriferous]

Okay, so now I'm stuck, but the minifier doesn't seem to be the problem. I removed it for one build but still got the FATAL ERROR. Here's the CI output:

Since 1.5.2 I now can't build my app with any of the 3 setups I've mentioned in my last comment and this one.

Worth noting that I upgraded to 1.5.2 only because Circle CI was getting stuck on dowloading meteor-tool. With an issue on meteor forums here suggesting to upgrade.

[abernix]

@Floriferous Can you try running the following command _immediately_ before you start whatever meteor command is yielding this error and see if it fixes your problem?

meteor npm install --global [email protected]

Reasoning: https://github.com/mozilla/source-map/blob/master/CHANGELOG.md && https://github.com/mozilla/source-map/pull/260

[Floriferous]

Just tried it, and it didn't fix the error. The error appears to be the same, but I'll paste it nevertheless. Also, a screenshot of the order in which the commands were run.

[abernix]

@Floriferous Ok, since this issue is closed and your issue is not the same as the others reported here: Can you open a new issue with the information we've discussed here and what we've tried already so we can avoid spamming the other 38 participants who are subscribed to this thread? 😄

Also, for the sake of completeness, can you include the output of that meteor npm install --global [email protected] section (i.e. expanded to show console output, instead of collapsed), just to make sure that it was installed into the correct dev_bundle?