Meteor: Source map generation causes build to go out of memory.

[sebakerckhof]

I'm trying to verify if https://github.com/meteor/meteor/issues/9552 is fixed in 1.6.1, but it takes forever and then goes out of memory.

Meteor 1.6.0 starts/builds my test-packages command in about 7 minutes.
Meteor 1.6.1 hangs for about 25 minutes on the Build step, and finally breaks down:

| (#4) Profiling: Build App                  /
   Linking                                   -             
<--- Last few GCs --->

[11441:0x2cf8550]  1948045 ms: Mark-sweep 1402.6 (2063.7) -> 1402.6 (2063.7) MB, 908.3 / 0.2 ms  allocation failure GC in old space requested
[11441:0x2cf8550]  1949102 ms: Mark-sweep 1402.6 (2063.7) -> 1402.6 (2011.7) MB, 1056.4 / 0.1 ms  last resort GC in old space requested
[11441:0x2cf8550]  1950043 ms: Mark-sweep 1402.6 (2011.7) -> 1402.6 (2002.2) MB, 940.2 / 0.2 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x286ac22a5ee1 <JSObject>
    2: _parseMappings(aka SourceMapConsumer_parseMappings) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/source-map/lib/source-map-consumer.js:462] [bytecode=0x1e5cf12fdb89 offset=204](this=0x12a7dc78ea9 <BasicSourceMapConsumer map = 0x3c0cf2167301>,aStr=0x1b346517fc79 <Very long string[3090]>,...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 2: 0x121a2cc [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 6: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 7: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 8: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::PretenureFlag) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 9: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
10: 0x10d434e [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/home/seba/.meteor/packages/meteor-tool/.1.6.1.blw62s.u2x7r++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
12: 0x15729810463d
Aborted (core dumped)

Running the application normally (meteor run) seems to work.

I did have to modify and include some packages to get 1.6.1 to work due to the babel changes (which turned out to be a very painful upgrade for a patch release IMO):

• https://github.com/Urigo/meteor-static-html-compiler (upgraded to babel-compiler 7)
• https://github.com/vsivsi/meteor-job-collection/ (converted from coffeescript, since the old coffeescript compiler also relies on the old babel compiler).

Other than that, the code is the same.

[abernix]

• Does running meteor run --production produce a similar failure?
• Can you reproduce the same occurrence when running Meteor from a checkout? (i.e. ./meteor run --production)
• If you provide the Meteor tool with more memory by setting the environment variable TOOL_NODE_FLAGS=--max-old-space-size=4096 (4GB in this case, adjust as your system permits of course), does it eventually finish?
  
  
  
  • i.e. TOOL_NODE_FLAGS=--max-old-space-size=4096 meteor run --production (or ./meteor from a checkout)
  
  
• If it does, can you provide the output of a successful run with the environment variable METEOR_PROFILE=1 set?
  
  
  
  • i.e. METEOR_PROFILE=1 ./meteor run --production
  
  

If this is, in-fact, a memory management problem as a result of source-map, I'm certainly intrigued by these recent changes they made in 0.7.0 (we use 0.5.3 in meteor-tool), though bumping 0.7.0 requires some other (breaking) changes to be resolved (feel free to investigate!) :smile:

[benjamn]

If someone wants to investigate upgrading to [email protected], I would happily review that pull request! Their usage of Rust/WASM is some of the coolest tech I've seen in a long time!

[hwillson]

If no one else has already started on this, I'll put my ✋ up.

[benjamn]

The great news is that this code will only need to run in Node 8, which natively supports WASM:

% meteor node -p WebAssembly.compile
[Function: compile]

[KoenLav]

Same issue here, 1.6.0 builds in under 10 minutes, 1.6.1 takes over an hour or renders an out of memory error.

[benjamn]

If we spent some time instrumenting meteor build with profiling hooks (see https://github.com/meteor/meteor-feature-requests/issues/239), then it would be easier to diagnose the source of this problem.

[benjamn]

Also, just to be clear, build times of more than a few minutes (even ~10) are unacceptable, and should always be regarded as a bug. These apps aren't written in C++ or Scala or some other language known for long compile times. Something's definitely wrong here.

[jamesmillerburgess]

I also just got a similar error on a rebuild of TINYTEST_FILTER="accounts" ./meteor test-packages.

=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
=> Linted your app. No linting errors.
   Building for web.browser                  -
<--- Last few GCs --->

[89583:0x103000000] 83182677 ms: Mark-sweep 1408.0 (1461.6) -> 1408.0 (1461.6) MB, 1681.8 / 0.1 ms  allocation failure GC in old space requested
[89583:0x103000000] 83184036 ms: Mark-sweep 1408.0 (1461.6) -> 1408.0 (1458.6) MB, 1357.9 / 0.1 ms  last resort GC in old space requested
[89583:0x103000000] 83185383 ms: Mark-sweep 1408.0 (1458.6) -> 1408.0 (1458.6) MB, 1346.6 / 0.1 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3d42453a5ee1 <JSObject>
    1: stringSlice(aka stringSlice) [buffer.js:~555] [pc=0x24287a4257b6](this=0x3d42b7b82311 <undefined>,buf=0x3d426623f969 <Uint8Array map = 0x3d4255241e99>,encoding=0x3d42453b6d31 <String[4]: utf8>,start=0,end=712006)
    2: toString [buffer.js:~609] [pc=0x24287989163e](this=0x3d426623f969 <Uint8Array map = 0x3d4255241e99>,encoding=0x3d42453b6d31 <String[4]: utf8>,start=0x3d42b7b82311 <undefin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Abort trap: 6

[benjamn]

@jamesmillerburgess Do you see any more clues in your stack trace about where the string/buffer slicing might have been happening?

[jamesmillerburgess]

@benjamn Not really, but maybe the error could have something to do with me deleting a project file during a rebuild? I don't remember if this was exactly the build I deleted the file, but that was one of the last things I did before seeing the error.

[hwillson]

With regards to updating source-map to 0.7.0, that might be a bit of a challenge due to this breaking change:

Breaking change: new SourceMapConsumer now returns a Promise object that resolves to the newly constructed SourceMapConsumer instance, rather than returning the new instance immediately.

The Meteor Tool makes a few new SourceMapConsumer calls like:

https://github.com/meteor/meteor/blob/1fec23f3b7a6d2a4ec45ef726884a336722a68c3/tools/fs/files.js#L1091-L1093

Adapting this code to work with Promises is likely not going to be straightforward however, since Fiber yields are blocked by loadIsopackage. This means we don't have a way to get the resolved SourceMapConsumer object before it's needed to feed into others parts of the synchronous Tool code.

Unless I'm missing something? Suggestions are welcome! 🤔

[sebakerckhof]

Does running meteor run --production produce a similar failure?

Yes, didn't check this first since our jenkins was building, but those got the max-old-space-flag set.

Can you reproduce the same occurrence when running Meteor from a checkout? (i.e. ./meteor run --production)

Yes

If you provide the Meteor tool with more memory by setting the environment variable TOOL_NODE_FLAGS=--max-old-space-size=4096 (4GB in this case, adjust as your system permits of course), does it eventually finish?

Yes, and then it finishes in about the same time as 1.6.0 (still about 10 minutes). So the slowdown is probably from the memory getting used up. Might it be that the new uglify-es version in 1.6.1 is producing more errors and therefore we're seeing the old problem with babeli ?

Anyway, there's something in 1.6.1 that results in higher memory usage than 1.6.0

Unfortunately we're working towards a new release of our application and I've got little time to spend on these kind of problems right now. So I'm just reporting my findings now.

One thing (unrelated to the above) I just noticed, is that almost all time in the ProjectContext prepareProjectForBuild step is spend on:
runJavaScript packages/urigo_static-html-compiler.js 38,360 ms (1)
On a total of: Total: 42,481 ms (ProjectContext prepareProjectForBuild)
From what I got of the comments on runJavaScript, this just evaluates the code. So I thought it might be from the npm dependencies in that package, but even if I remove those, it remains that high.

[KoenLav]

Interesting!

I'll try upping the memory limit on our builds as well.

Actual build & deploy times for our app used to be around 6 minutes, the whole CircleCI run would take about 10.

It might also be worth noting that we experienced a similar issue when using standard-minifier-js 2.2.3 instead of 2.2.1 in Meteor 1.6.0.

https://github.com/meteor/meteor/issues/9430

[abernix]

@hwillson Since Meteor's Promise implementation should be in play, is it possible to just call .await() on the result of the new SourceMapConsumer instantiation while keeping everything else relatively unchanged?

[hwillson]

@abernix I had tried, but unfortunately the fiberHelpers.noYieldsAllowed call in loadIsopackage prevents the await from working:

While loading isopacket `combined`:
/tools/utils/fiber-helpers.js:28:9: Can't call yield in a noYieldsAllowed block!
at Function.disallowedYield (/tools/utils/fiber-helpers.js:28:9)
at stackSafeYield (/Users/hwillson/Documents/git/meteor/meteor/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:101:25)
at awaitPromise (/Users/hwillson/Documents/git/meteor/meteor/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:96:12)
at Function.Promise.await (/Users/hwillson/Documents/git/meteor/meteor/dev_bundle/lib/node_modules/meteor-promise/promise_server.js:56:12)
at Profile.time (/tools/fs/files.js:1092:22)
...

[hwillson]

Just to add @abernix - removing the noYieldsAllowed call from loadIsopackage fixes the issue (letting Promise.await work), but the no yields block was added in https://github.com/meteor/meteor/commit/e7167e5257fc152af4723c051c53dec44a421dc5 as part of a massive file tool cleanup. Maybe it's being a bit overzealous, but I'm reluctant to change it without knowing the full impact of it being removed. Maybe we can be a bit more selective about what isn't allowed to yield under the loadIsopackage process. That's the path I've been heading down ...

P.S. > Working through the isobuild code has me yielding for coffee ... a lot. 🙂

[abernix]

Hmm. Commit https://github.com/meteor/meteor/commit/e7167e5257fc152af4723c051c53dec44a421dc5 leaves a lot to be understood (grr, squash!) about the reasoning for the noYieldsAllowed being in the place that it is (it wasn't there before!). The (squashed) commit message hints: Make sure JsImage readFromDisk doesn't yield. With that in mind, it might be reasonable to move the noYieldsAllowed to wrap the bundler.readJsImage call in loadIsopacketFromDisk rather than also enveloping the load call (which I think is where you're running into the yield being prevented):

https://github.com/meteor/meteor/blob/070acc01b03397a18fc259837027fa5e244a790d/tools/tool-env/isopackets.js#L296-L298

I think that would allow the runJavaScript at https://github.com/meteor/meteor/blob/070acc01b03397a18fc259837027fa5e244a790d/tools/isobuild/bundler.js#L1972-L1977 to permit the await() you're needing to add in this case. It may need to be more surgical, but I'd be curious how the test suite behaves with that change. 😸

[hwillson]

Thanks @abernix - that's exactly where I'm headed, and things are looking promising. I'm prepping a PR, so we'll be able to kick the tires shortly.

[KoenLav]

Promising, hehe.

Great to see there's progress here, would love to try it out, currently abroad but planning on testing the latest changes to Cordova and the build process next week.

[PolGuixe]

My case:

• Version: 1.6.0.1
• MacOS 10.13.2

This is a weird error. On a new machine - MBP 15" 2017 -, when trying to deploy to make a build -deploy to Galaxy, --production flag, ...- it runs out of memory:

   Minifying app code                        \
<--- Last few GCs --->

[1225:0x102801e00]   257826 ms: Mark-sweep 1390.5 (1546.1) -> 1390.5 (1546.1) MB, 1153.5 / 0.1 ms  allocation failure GC in old space requested
[1225:0x102801e00]   259049 ms: Mark-sweep 1390.5 (1546.1) -> 1390.5 (1507.1) MB, 1222.2 / 0.1 ms  last resort GC in old space requested
[1225:0x102801e00]   260306 ms: Mark-sweep 1390.5 (1507.1) -> 1390.5 (1501.6) MB, 1256.9 / 0.1 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1b75197a5ee1 <JSObject>
    0: builtin exit frame: slice(this=0x1b751424c9a1 <JSArray[1232183]>)

    1: clone [/Users/polguixe/.meteor/packages/standard-minifier-js/.2.2.3.fd3562.vtpy++os+web.browser+web.cordova/plugin.minifyStdJS.os/npm/node_modules/meteor/babel-compiler/node_modules/babylon/lib/index.js:~630] [pc=0x71f45e352f3](this=0x1b751424c8f9 <State map = 0x1b759956f021>,skipArrays=0x1b7508f02311 <undefined>)
 ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

However, the same exact project builds on older machines without issues 🤷‍♂️ :

• MBP 13” Late 2013 MacOS 10.13.2
• Mac Mini 2013
• MBP 13” Late 2011

Any idea? The logic tells me that must be something with the new machine, however, I have no idea... the only change we did to the older machines was related to this issue which seems to be solved on newer versions of Meteor #6952

[KoenLav]

Upping max-old-space-size to 3GB in CircleCI resolves the issue, the build now completes in +/- 12 minutes on a free instance.

To test this add yourself on CircleCI (v2):

TOOL_NODE_FLAGS: --max-old-space-size=3072

to the environment within your image.

[RobertLowe]

I had this happen to me, meteor ran fine locally on my MBR, however once pushed the app to heroku to build with meteor-buildpack-horse it continually exploded. I was able to resolved the issue by updating the app from 1.6.0.1 to 1.6.1 it appears that 1.6.1 building 1.6.0.1 apps has issues. I'm sorry to the yak I shaved.

[KoenLav]

@benjamn I've read the details about WebAssembly and if my interpretation is correct this could soon open up the possibility of (near) native performance of Javascript applications.

Just checking here, before I create a FR: Meteor could, at some point, update its build proces to compile the bundle into WebAssembly and serve that alongside the current bundle, allowing browsers which support WebAssembly to run Meteor at (near) native speed.

I'm probably missing some steps still, but aside from that this would be possible, or not?

[benjamn]

@KoenLav Compiling JavaScript to WebAssembly isn't something people are likely to do anytime soon, because WebAssembly has no built-in garbage collection, and JS is a language that relies on GC. Languages like Rust and C++ that don't rely on GC are currently the most attractive options for compiling to WebAssembly.

Also, JavaScript works as-is in the browser, so there's no need to compile it to something else (besides the usual compilation of non-native syntax, a la Babel).

[KoenLav]

I see, so compiling the entire bundle isn't something that we'll see soon, but compiling certain functions or portions of code in Meteor/React to WebAssembly could speed things up, I guess?

If not then I guess I am missing the point completely... The use of WebAssembly, as far as I understood, comes from providing Javascript with a way to execute (faster) native code.

Suppose there are certain Meteor packages (or more generally Npm packages) which have little dependencies they could probably be converted into C at some point, either manually or partially automated (https://github.com/andrei-markeev/ts2c).

I guess this is not (necessarily) a Meteor battle to be fought, but it would be interesting to explore the possibility of rewriting a Meteor/Npm package to C and only including a wrapper/proxy JS file to map the Javascript functions to their C counterparts.

[sebakerckhof]

@KoenLav , it's not really native code, it's portable bytecode and needs a VM (basically the same one that also runs javascript) to run it. In that regard it's similar to java and the JVM, but without GC and some other stuff at the moment.
It can be faster since not having things like GC gives you more control and static languages allow compilers to make more optimizations. However, javascript engines are also already crazy smart and fast. So a naive automated convertion to C and then back to WASM is likely not going to result in noticeable speedups. You could manually rewrite Meteor from scratch, but then you no longer have a javascript framework. And I don't know about you, but I rather don't go back to writing C.

So WASM, at this point, only makes sense for some performance-critical stuff and bottlenecks (like sourcemaps) or for people who really hate JS, and Meteor devs aren't supposed to fall in that latter category.

[KoenLav]

@sebakerckhof I see, that's a shame, I guess it opens up avenues for further improvement, but for now not as significant as I imagined. Thanks for the insight!

[bouchepat]

Having the same issue on Windows 10 and meteor version 1.6.0.1. Works fine on Windows 7.

=> Started proxy.
[client]: HTML Files Compilation: 119.696ms  \
   Building for web.browser                  \
<--- Last few GCs --->

[9816:04337E38]   298690 ms: Mark-sweep 714.4 (805.7) -> 714.4 (797.7) MB, 1109.0 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1109 ms) last resort GC in old space requested
[9816:04337E38]   299798 ms: Mark-sweep 714.4 (797.7) -> 714.4 (797.7) MB, 1107.7 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 2751632D <JSObject>
    1: /* anonymous */ [C:\Users\bouch\AppData\Local\.meteor\packages\angular-compilers\0.2.9_2\plugin.Angular_Compilers.os\npm\node_modules\meteor\babel-compiler\node_modules\babel-traverse\lib\index.js:~94] [pc=4809DAE7](this=1F294765 <JSFunction traverse (sfi = 35AE1099)>,node=449A776D <Node map = 0737DA49>,opts=28043791 <Object map = 0F5B8FA9>,scope=3707FFBD <Scope map = 0F5B8EF9>,state=46B04101 <...

[KoenLav]

@hwillson any updates?

[spencern]

We're seeing this issue on the [email protected] release. Currently 1.6.1 with Babel 7 versions pinned to 7.0.0-beta.38 is working for us.

This thread/comment has more detail: https://github.com/reactioncommerce/reaction/pull/3615#issuecomment-365747880

[spencern]

Update to the above comment. Starting with TOOL_NODE_FLAGS="--max-old-space-size=4096" enables the app to build.

OSX Sierra | Core i7 | 16GB

[KoenLav]

Increasing the amount of memory available is the 'fix' for this issue for now (and works fine).

[fatbird]

Ran into this today with [email protected]. Interestingly, it was building fine (~1min) until I added the static-html package back in solely to pick up a single small html file. Setting TOOL_NODE_FLAGS as above fixed it, though the build now takes ~6 min.

[filipenevola]

@fatbird how big is your project? and how long take it to rebuild after a change?

See our problems here #9642 , maybe we can try to help each other.

We have to use TOOL_NODE_FLAGS to be able to build our project too but even without it we had problems with it taking to long too.

[fatbird]

@filipenevola it's about 50k LOC. After a change running in dev mode, it's usually quick to rebuild. It's only the production build (or running --production) that it takes 5-6 minutes. And the significant thing to me is that this only happened after I installed the static-html package for meteor--before that, it was taking only 3-4 minutes for a build, and didn't require the TOOL_NODE_FLAGS setting.

I'll look at the linked ticket and see what I can add.

[sebakerckhof]

In 1.6.2 my rebuild times seem to fluctuate mostly on the prepareProjectForBuild step.
The problem is that almost all time seems to be spend on other _buildLocalPackages. And the weird thing is that this fluctuates heavily between 5 and 100 seconds, even for changes on the same file.

I opened a feature request to make this more detailed. In the meantime, is there a guide somewhere on CPU benchmarking meteor tool using --inspect?

The actual Rebuild app step seems stable around 35 seconds in 1.6.2 for my project.

e.g.

| Top leaves:
| other _buildLocalPackages...............................82,657 ms (1)
| other IsopackCache Build local isopack...................3,089 ms (41)
| files.rm_recursive.......................................2,630 ms (6)
| files.readFile...........................................2,078 ms (7193)
| other optimistic lstatOrNull.............................1,070 ms (7125)
| files.writeFile............................................783 ms (8865)
| other Builder#copyDirectory................................694 ms (2)
| files.stat.................................................537 ms (18933)
| other optimistic readFile..................................518 ms (7034)
| other files.withCache......................................503 ms (12)
| other Isopack#saveToPath...................................375 ms (2)
| other optimistic lstat.....................................341 ms (7044)
| other optimistic statOrNull................................319 ms (6920)
| other compileUnibuild (barco:edu)..........................259 ms (3)
| other Builder#write........................................223 ms (2090)
| files.readdir..............................................215 ms (3043)
| files.lstat................................................165 ms (7047)
| other watch.isUpToDate.....................................157 ms (43)
| sha1.......................................................119 ms (2116)

I

[abernix]

@sebakerckhof Appreciate you trying to dig into it! You should be able to profile in Firefox/Chrome via the Node inspector API using:

TOOL_NODE_FLAGS=--inspect-brk meteor ...command as usual...

And of course, if you run meteor from a checkout, additional dimensions can be added to the METEOR_PROFILE output tree by introducing functions to Profile, à la:

originalFunctionName = Profile("GroupName", function () { /* the original function */ }})

[sebakerckhof]

Thanks @abernix .

I took 2 profiles during rebuilds (both attached) and it seems like all time is spend by the garbage collector. One time the prepareProjectForBuild _buildLocalPackages took 55 seconds, the other time prepareProjectForBuild _buildLocalPackages took 157 seconds.

Admittedly, I am running a little low on memory (got 8gb ram occupied and 4 out of 8gb ram swap used, because that's apparently what you use as a javascript developer running vs code, a couple of chrome tabs and meteor). But then I still can't explain why only the prepareProjectForBuild _buildLocalPackages suffers from this, while the other steps rebuild app are very consistent and reasonably fast
CPU-20180329T1.zip

Note that I'm already running Meteor with --max-old-space-size=4096 in the NODE_TOOL_FLAGS, otherwise it goes OOM

[veered]

@sebakerckhof Have you tried building on a machine with more RAM?

[sebakerckhof]

@veered , yeah, but if I run it with the same --max-old-space-size setting, it doesn't really matter.
However, on a 16gb machine I can increase the --max-old-space-size to 8192, which has a big impact.

So the _buildLocalPackages seem to use a lot of memory. When heap size reaches its limits a lot of time is spend by the garbage collector and reported under other _buildLocalPackages by meteor-tool profiling.

[benjamn]

@sebakerckhof Can you try running with TOOL_NODE_FLAGS=--no-expose-gc to disable the Meteor's requestGarbageCollection calls?

[georgeiou]

Having similar issues. Upgraded us from Meteor 1.5.0 to 1.6.0.1. Build times have shot up from 12 minutes to 25 minutes.

This is our production Meteor 1.6.0.1 build running on CircleCI 4gig VM:
Heres what I got from prepareProjectForBuild:
| Top leaves:
| meteorNpm.runNpmCommand................................148,546 ms (4)
| other _buildLocalPackages..............................144,883 ms (1)
| other meteorNpm.rebuildIfNonPortable....................36,212 ms (189)
| other optimistic lstat..................................27,741 ms (289863)
| files.writeFile.........................................23,299 ms (106793)
| files.readFile..........................................12,708 ms (118851)
| other optimistic statOrNull.............................11,756 ms (122889)
| other optimistic lstatOrNull............................10,667 ms (113262)
| other IsopackCache Build local isopack..................10,097 ms (10)
| files.stat...............................................9,996 ms (189427)
| other package-client _updateServerPackageData............9,898 ms (1)
| other optimistic readFile................................9,465 ms (117915)
| files.lstat..............................................9,442 ms (218987)
|
| (#1) Total: 465,114 ms (ProjectContext prepareProjectForBuild)

Adding TOOL_NODE_FLAGS=--max-old-space-size=4096 did resolve our OOM errors but sadly the production build times are just way too high.

[benjamn]

@georgeiou Yikes! That's not normal (even before the update). How big is your node_modules directory?

[georgeiou]

@benjamn
Spent some time bringing down the build times. Updated our minifier to the latest allowed, moved around some circleCI caches and now I'm clocking about 12-15minutes per build! Which has always been the normal range so I'm pleased to be back to normal. But I would love to figure out why the prepareProjectForBuild takes so long!

Here is the full put of prepareProjectForBuild
meteor1.6.0.1.txt

Our node_modules for production build is 15MB.

[filipenevola]

Hello, in the last 3 days I'm having problems with jobs on CircleCI too, they are running (build + deploy on Galaxy) for almost 40 minutes and sometimes being killed, probably running out of memory.

The time to build + deploy was around 28 minutes last week.

@georgeiou can you share the cache configuration that you are using in CircleCi?

[georgeiou]

@filipenevola I don't want to derail the purpose of this issue so visit the link below for what my set up is. I updated to [email protected], [email protected], [email protected] which helped with build times.

https://forums.meteor.com/t/what-is-the-simplest-setup-for-meteor-chimp-on-circle-ci/31627/28

[benjamn]

@filipenevola What version of Meteor are you currently using for those tests?

[filipenevola]

@benjamn we are using 1.6.1.

We already are at the latest version of the minifiers and already using cache in CircleCI too.

[georgeiou]

@filipenevola Mind posting your top leaves? METEOR_PROFILE=1 before your meteor build command on circleCI.

[filipenevola]

@georgeiou it's here https://gist.github.com/filipenevola/e749e892e847c0dbc5385cd7a7ffecd4

[sebakerckhof]

@benjamn adding TOOL_NODE_FLAGS=--no-expose-gc does seem to solve the huge amount of time spend in the other _buildLocalPackages step. Tested with 1.7-beta.24.

On the other hand, the time for 'rebuild app' seems to go up more with TOOL_NODE_FLAGS=--no-expose-gc when memory usage goes up.

Tried about 10 rebuilds each with increasing memory usage on 1.7-beta.24:

• With TOOL_NODE_FLAGS=--no-expose-gc:

  • ProjectContext prepareProjectForBuild step: 4 - 10 seconds
  • Rebuild App step: 8 - 68 seconds (as memory usage goes up)

• With TOOL_NODE_FLAGS=--expose-gc:

  • ProjectContext prepareProjectForBuild step: 15 - 250 seconds (as memory goes up)
  • Rebuild App step: 8-23 seconds

[veered]

Could you try running https://github.com/qualialabs/profile for a clean, local dev initial build (not production bundle build)? The package may work on a prod bundle build, but I haven't tried it yet.

It'll output a really nice CPU profile.

[sebakerckhof]

@veered Thanks, really useful!

Attached you can find 6 different performance profiles.

• no-expose-gc, low-memory, low-cpu
• no-expose-gc, high-memory, low-cpu
• no-expose-gc, high-memory, high-cpu
• expose-gc, low-memory, low-cpu
• expose-gc, high-memory, low-cpu => Here the garbage collector goes a little off track during the _buildLocalPackages step.
• expose-gc, high-memory, high-cpu => Here the garbage collector goes totally off track during the _buildLocalPackages step.

All of them ran with Meteor 1.7-beta.24 and --max-old-space-size=6192

• Low-cpu means only meteor was really doing work (and the usual OS stuff)
• High-cpu means I was using a software encoder to encode a video at the same time, having my CPU constantly above 90% (since I'm developing a webrtc application, this is my usual scenario).
• Low-memory means I had about 4/8 GB ram used and 2/8 GB swap used
• High-memory means I had about 7/8 GB ram used and 5/8 GB swap used

I didn't always restart meteor, so I only included the rebuild and server startup files.

profiles.zip

[veered]

Wow expose-gc, high-memory, high-cpu spends 85% of the reload time doing garbage collection... Given the difference in performance between the high memory & low memory profiles, it seems like the best immediate solution is to just make sure you have enough RAM available. Also, do you have a SSD?

Even in the profiles with plenty of memory, the rebuild still spends ~30% of time on GCs. V8 uses a generational garbage collection so it does minor GC's all the time, even if the process has a high memory limit. So the only way to bring that down is to allocate less memory during the rebuild.

You also seem to have a lot of files in your project. Is that mostly your code or is it possible that Meteor is reading files in node_modules/?

On a side note, most of the Babel compilation time is spent reading and parsing the Babel config file. It lookups the config file for every single JS file it compiles! I've noticed similar behavior in my application.

@benjamn you had a chance to look at these profiles? Really interesting!

[sebakerckhof]

Thanks for also taking a look @veered . Yeah, I got an SSD, but RAM is soldered on, so that's gonna be though to expand above my current 8GB. Anyway, running with --no-expose-gc kinda solves it at the moment.

There should be around 2000 files of the project itself. Most are quite small since we do 1 method / publication / component per file. The Babel config thing you see is interesting, because there does seem to be a cache for that in place ( https://github.com/meteor/meteor/blob/devel/packages/babel-compiler/babel-compiler.js#L37 )

[adamantivm]

Just to mention that I bumped into this issue on Linux with Meteor 1.6.1. Building OK for local but couldn't deploy to Galaxy or build with --production due to this crash.

Fortunately after updating to 1.6.1.1 the issue seems to have been alleviated enough that now I can build and deploy again.

[brianlukoff]

I'm having the same problem, but in reverse: trying to build Meteor 1.6.1.1 (in a Docker container using Meteor Launchpad), I get an out of memory error, and adding --no-expose-gc or --max-old-space-size=4096 does not help. However, the build works fine if I revert to 1.6.1.

[hwswin]

export TOOL_NODE_FLAGS="--max_old_space_size=4096" working for me. Thank you.

[sebakerckhof]

Somehow running my unit tests has become impossible. It takes 15 minutes (916 seconds) to start the tests. Of which 764 seconds are spend in babel compilation. Starting meteor normally is fine with --max-old-space-size=6192 --no-expose-gc

[bouchepat]

Still running into the same issue on Windows 10. Does not matter the meteor versions, tried : 1.6.0.1 and 1.6.1.1.

Tried all the below, works on Windows 7 but not on Windows 10

set export TOOL_NODE_FLAGS="--max_old_space_size=4096"
meteor run

set export TOOL_NODE_FLAGS="--max-old-space-size=6192 --no-expose-gc"
meteor run



md5-1a4e355e506ae3f7c1f18b12be112973



> [email protected] prod C:\Repositories\zeteor-web
> meteor --settings settings.json

[[[[[ C:\Repositories\zeteor-web ]]]]]

=> Started proxy.
   Building for web.browser                  \
<--- Last few GCs --->

[7952:03B8D5F8]   105075 ms: Mark-sweep 669.1 (737.5) -> 669.1 (738.0) MB, 1074.7 / 0.1 ms  allocation failure GC in old space requested
[7952:03B8D5F8]   106041 ms: Mark-sweep 669.1 (738.0) -> 669.1 (720.0) MB, 966.0 / 0.1 ms  last resort GC in old space requested
[7952:03B8D5F8]   106936 ms: Mark-sweep 669.1 (720.0) -> 669.1 (720.0) MB, 894.4 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 3B116169 <JSObject>
    0: builtin exit frame: slice(this=14F7B9CD <JSArray[558446]>)

    1: clone [C:\Users\bouch\AppData\Local\.meteor\packages\angular-compilers\0.2.9_2\plugin.Angular_Compilers.os\npm\node_modules\meteor\babel-compiler\node_modules\babylon\lib\index.js:~630] [pc=4CD9EA65](this=14F7B989 <State map = 18CCD505>,skipArrays=32C8417D <undefined>)
    2: arguments adaptor frame: 0->1
    3: parseMaybeAssign...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::Factory::NewUninitializedFixedArray
 3: v8::internal::Factory::NewJSArrayStorage
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! [email protected] prod: `meteor --settings settings.json`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the [email protected] prod script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\bouch\AppData\Roaming\npm-cache\_logs\2018-05-26T16_05_39_381Z-debug.log

Let me know if there is something else I should try.
Thx

[RobertLowe]

@benjamn

Also, just to be clear, build times of more than a few minutes (even ~10) are unacceptable, and should always be regarded as a bug

OTT: cough meteor node stubs, and dubious imports between client <-> server where they shouldn't be (ex: node modules, or even server code) can really mess with build times. There should probably be some baby-proofing for either.

---

To anyone with this issue:

Try:

1. Updating your app to from 1.6.0.1 to 1.6.1
2. Ensure you're local / system Meteor is up-to-date, or alternatively use 1.6.0.1 to build 1.6.0.1 you can use the release flag when installing meteor on your system(s): https://install.meteor.com/?release=1.6.0.1 if necessary but not advised

Cheers

[nathan-muir]

TLDR Build was _much_ faster with TOOL_NODE_FLAGS="--min_semi_space_size=8 --max_semi_space_size=256"

---

Performing an initial build (after a meteor reset) would take approx 131s in meteor 1.6.0.1

In meteor 1.7.0.1 it takes approx 390s to build the same app.

After profiling, I noticed that:

• About 45% (193s) of that time is spent in the garbage-collector.
• Adding --trace-gc showed Scavenge GC's starting at 0.1ms and ending up at 60ms per run. This was the major source of the cumulative delay in gc. (there's a few hundred scavenges between mark-sweeps)

Adding the options --min_semi_space_size=8 --max_semi_space_size=256 reduced build times to 200s. And reduced time spent in GC to 11% (or 28s).

I'm not an expert with node/v8 garbage collection, and couldn't find any convincing documentation. So, use those settings at your own risk.

[RobertLowe]

Semi-OT: @nathan-muir build times should not take that long, are you using meteor-node-stubs? Can you remove it? Is your package.json/package-lock.json bloated by something(s)? Can you trim there? Are you using lots of packages that use transpiling (ex: coffeescript, sass, jade, etc, etc)? Are your imports happening the correct environments (ex: not globally importing a package on both client and server, or importing something on the client thats intended for the server only, etc)

[nathan-muir]

Hey @RobertLowe 80% of that time is spent in Babel.compile. - https://gist.github.com/nathan-muir/c909cc7bf5b544ac7e475e02c3ccea0d

[RobertLowe]

@nathan-muir

| files.stat...............................................6,164 ms (35234)

suggests to me you have a large number of files in project, I'm not a 100% on this but I would highly suggest investigating

[nathan-muir]

@RobertLowe Interesting, the cause of the majority of those files.stat are from resolving imports (Within isobuilds Resolver._resolveNodeModule)

To be compliant when resolving import 'some-module', it needs to check the current directory for a ./node_modules/some-module dir, and again in every parent directory up to the root of the project.

These are the majority of those 40k file.stats -

• ./node_modules/meteor/${packagename} from import {Meteor} from 'meteor/{meteor-package}'
• ./node_modules/babel-runtime/... from es6 polyfills (helpers and corejs)
• ./node_modules/... from regular import 'some-node-module'
• ./babel-plugin-xx from a single .babelrc with {"plugins":["babel-plugin-idx"]} - changing it to {"plugins":["./node_modules/babel-plugin-idx"]} short-circuits this

In testing, I implemented a fast-path for resolving meteor/ imports, which reduced files.stat usage by 50%, and saved ~5s off the build time.

[adamgins]

Hi, can't get this to work (Meteor 1.6.1) , even with TOOL_NODE_FLAGS: --max-old-space-size=4096 on CircleCI. Any other workarounds?

[maxnowack]

@adamgins try TOOL_NODE_FLAGS="--max-old-space-size=3072". This works for me on CirleCI

[adamgins]

@maxnowack same issue.
BTW I can do a deploy to galaxy on AWS C9 with 2.5GB of ram.
I am about to try AWS Codedeploy, the have a 7GB machine
@benjamn @abernix I am using FROM abernix/meteord:node-8.11.2-onbuild wondering if you have any words of wisdom/magic settings, pls?

[adamgins]

I am getting the same issue when I run on AWS Codedeploy with a 7GB machine.
I got it to work by removing the standard minifiers and it works.... @abernix @hwillson

I saw a bunch of related issues
https://github.com/meteor/meteor/issues/9430
https://github.com/meteor/meteor/issues/10044
https://github.com/meteor/meteor/issues/10042

What's the right way forward pls?

[RobertLowe]

I found one possible root cause for this issue.

Meteor greedily includes assets. I had made a backup of my node_modules folder to _node_modules. Meteor was compiling all the assets therein. Meteor should be less greedy about it's root folder! Perhaps only a lazy scan for assets (js/css,etc) and then auto included folders client, server, imports, etc, and so include other folders, a .meteor/config/.meteor/includes could be set.

Edit: Essentially, this could happen with large assets (third party libs, etc outside compatibility) added to meteor haphazardly

[adamgins]

@RobertLowe I am using packaged like AWS which are comparatively large. This has always worked fine with previous Meteor versions.

@benjamn any ideas on how to resolve/work around please? Removing the minifiers works but then you lose all the benefits of minifying code, right?

[coagmano]

IIRC, I saw that some of the optimisations in 1.7.1 will help, as the compiler should then lazily compile assets instead of greedily

Maybe try the beta and see if it helps?

[RobertLowe]

@coagmano I'm not sure the root issue was solved, but max-old-space is set to 4GB by default not as of: https://github.com/meteor/meteor/commit/5c1ba3a85ec17141f02c61cea03044596c9b4a78

If you're builds are failing perhaps watch the memory of your app while it builds, maybe it's hitting the ceiling which may mean you have accidentally put more assets than meteor is ready to handle (in one case I lazily copy and pasted my node_modules folder).

[adamgins]

@RobertLowe how do I lazily copy and pasted my node_modules folder to try and solve this pls?

[RobertLowe]

@adamgins to clarify, that was the cause I had two node_modules folders (think folders with a TON of assets for meteor to process), the solution was tracking down bloat and removing it, keep things lean

[benjamn]

@RobertLowe The eager compilation of code (some of it in node_modules, but also application code) is a long-standing problem with the Meteor compiler plugin system, and something we're finally fixing (or at least drastically improving) in Meteor 1.7.1, as @coagmano noted: #9983

[RobertLowe]

Thanks @benjamn that's great work, I'm aware! Only keeping followers of this thread informed where to look for issues.

[Floriferous]

EDIT: moved comment to new issue #10218

[witcradg]

I'm seeing this same issue (memory allocation failure) on a DigitalOcean droplet running Ubuntu 18.04 and using Meteor 1.8. I am not able to run a build even with TOOL_NODE_FLAGS="--max-old-space-size=4096"
meteor build --server-only ~/meteorapps/myapp-built --architecture os.linux.x86_64
I'm getting a core dump in garbage collection.
After a several tries, it eventually ran successfully.

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory
1: node::Abort() [/home/dean/.meteor/packages/meteor-tool/.1.8.0.1gdnmok.4nus++os.linux.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]

[azrijamil]

I have this problem before, manage to get rid of it after simple a fix. What happen is, I accidentally run a npm link command in a meteor project instead of library project. I've been going a series of clearing, deleting, updating, downgrading the meteor and node itself, none of it help.

My meteor release version is 1.6.1.4 and using node version of 8.10.0 managed by nvm, hope this can help or give a clue to others if encounter similar scenarios or problem.

[macrozone]

we build the projects now on gitlab-ci and this used to work. Recently, we upgraded our runner to have more cpu and ram (from 1 GB to 2GB) but surprisingly, the runner gets now killed because it runs out of memory after meteor build needs more then GB.

The same app could be built successfully on a runner with 1GB, why does it no longer work on one with 2GB?

[macrozone]

any update on this? still happens a lot

using [email protected]

[macrozone]

Has anyone found a workaround?

we have currently this build command:

TOOL_NODE_FLAGS="--max_old_space_size=4096 --min_semi_space_size=8 --max_semi_space_size=256" meteor build $CI_PROJECT_DIR --architecture os.linux.x86_64 --allow-superuser --server-only --directory

i seem memory consumption goes very steeply to the 4Gigs and stays there, garbage collecting probably.

I already tried to see if i can remove some dependencies from the package.json, but its an older and bigger problem and i could not identify packages to remove.

EDIT: updated to meteor 1.8.2 does not seem to fix the issue.

[adamgins]

No. On aws c9 I had to go to a 8GB Machine. I had to move our Docker builds from continuous integration platform on to my local machine and se my docker memory to 8GB to get through the Docker Build. A real pain.

[benjamn]

Anybody interested in updating Meteor's source-map dependency to a version that uses Rust/WASM, per my previous comment? https://github.com/meteor/meteor/issues/9568#issuecomment-359806003

The current version is defined in this file, and you'll want to read up on building the dev bundle locally and running Meteor from a checkout: https://github.com/meteor/meteor/blob/release-1.9/DEVELOPMENT.md#running-from-a-git-checkout. Also, I would recommend working from the release-1.9 branch.

[sebakerckhof]

It's something I'd be wanting to do some time ago. But I remember this comment of @hwillson : https://github.com/meteor/meteor/issues/9568#issuecomment-359930688 . So the constructor became async, but it's created in a noYieldsAllowed block of loadIsoPackage. So likely not so trivial.

Edit: nevermind, turns out the noYieldsAllowed has been removed in the meanwhile ( https://github.com/meteor/meteor/commit/1cf59867b2f74f5be1a3e6f1eb13ea4ae10540db )

[macrozone]

@benjamn would it be possible first to disable source map generation? At least until the memory problem is fixed.

By the way, i played around with the --max_old_space_size setting and got some learnings:

• in my case we build on a gitlab-runner which is run in a kubernetes cluster. We set the memory limit to 4GB to these runners. Setting max_old_space_size to 4096 will use all the memory i that pod, even more, because of the os processes in that pod. That might lead to kubernetes killing that pod.
• therefore you have to set it lower e.g. to 3Gigs or 2gigs. Altough i still observed that the build will fail, altough not because of kubernetes killing that pod. The reason is still unknown to me
• Setting it lower does not mean that the build will be slower, alltough i could not make many tests on the gitlab runner, because it takes a lot of time to do these experiments. But i did it locally and i observed that the build is _faster_ with a limit of 2Gigs then setting the limit to 3 Gigs. This is unexpeced, because 2Gig should add more GC pressure / swaping or whatever.

[sebakerckhof]

I have a POC branch with source-map 0.7.3, using the WASM implementation.
Can you guys test with this branch: https://github.com/sebakerckhof/meteor/tree/chore/update-source-map

You will need to:

git clone --recursive https://github.com/sebakerckhof/meteor.git
cd meteor
git checkout chore/update-source-map
./scripts/generate-dev-bundle.sh
cd /path/to/your/meteor/project
/path/to/meteor/checkout/meteor build

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[macrozone]

the issue is still a problem

[sebakerckhof]

@macrozone Can you test with 1.9-rc.2 which includes https://github.com/meteor/meteor/pull/10798 ?

[opyh]

I still see a problem here - Tested with v1.10.2 and TOOL_NODE_FLAGS "--max-old-space-size=2048 --optimize_for_size --gc-interval=100":

It seems switching off the GC in source-maps generation (WASM, https://github.com/meteor/meteor/pull/10798#issuecomment-558784613) causes a spike at the end of the build that goes over the 2048M value you'd expect with this setting.

Apparent options for people stumbling upon this (correct me if I'm wrong):

• Increase VM size for builds to 3Gi+ (making memory dependent on your bundle size, seems odd)
• Fix https://github.com/nodejs/node/issues/29767 in Node / node-fibers so we can remove the --no-wasm-code-gc flag
• Find a different way to generate source maps in Meteor
• Add a (automatic?) switch between no / fast-method / slow-method source maps generation in meteor build so you can choose between more speed and more memory usage

[sebakerckhof]

@opyh This issue stems from a time when we were using the none WASM version. The slow / non-wasm version was using at least as much memory. The wasm comes from Rust code which does its own memory magement. GC isn't likely to help here.

So that leaves option 1 & 3.

[sebakerckhof]

Maybe we could use: https://github.com/parcel-bundler/source-map

[opyh]

In the meantime, how about displaying a helpful warning before source maps generation?

It’s a frustrating kind of error as it happens at the very end of the build process, and it is very likely to happen on typical CI setups. You get no sensible error message (or none at all depending on your Docker build tool). And when you stumble upon it, it’s unclear where to start searching.

Theoretically, the build process could even foresee the issue, and abort the build on start, before you wait 10 minutes for an error message to appear.

Suggestion:

``javascript console.log(
Your build machine has ${availableMemoryInMegabytes}M memory available for building.

If this process should abort with an out-of-memory (OOM) or “non-zero exit code 137”
error message, please increase the machine’s available memory.

See https://github.com/meteor/meteor/issues/9568 for details.
)

[opyh]

Is there a way to limit used memory in the Rust code? I think it's generally a good idea to have less non-wasm native dependencies to be more future-proof.

Maybe related:

https://github.com/mozilla/source-map/issues/412
https://github.com/mozilla/source-map/issues/342

[opyh]

Another idea: Are fibers absolutely necessary for this build step? Couldn't the build process generate them in separate child processes that don't use fibers, for example?

[opyh]

https://github.com/meteor/meteor/issues/11143 looks like this might be a bigger issue in the near future - if more node modules begin to use WASM, you won't be able to use some of them without OOM crashes in Meteor.