Standard: Memory error

[christianalfoni]

Hi there!

Monorepos are becoming more and more popular and it seems that standard gets into issues with multiple packages in one repo.

This is the error I get when I try to lint our monorepo:

<--- Last few GCs --->

   70345 ms: Mark-sweep 1345.7 (1435.2) -> 1345.7 (1435.2) MB, 1763.1 / 1.3 ms [allocation failure] [GC in old space requested].
   72108 ms: Mark-sweep 1345.7 (1435.2) -> 1345.7 (1435.2) MB, 1762.5 / 1.4 ms [allocation failure] [GC in old space requested].
   73943 ms: Mark-sweep 1345.7 (1435.2) -> 1345.7 (1404.2) MB, 1834.7 / 1.3 ms [last resort gc].
   75735 ms: Mark-sweep 1345.7 (1404.2) -> 1345.7 (1404.2) MB, 1792.3 / 1.4 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35f9d0cfb51 <JS Object>
    1: inflight [/Users/christianalfoni/Development/cerebral/node_modules/inflight/inflight.js:~7] [pc=0x3780a4557197] (this=0x35f9d0e6f19 <JS Global Object>,key=0x253882120d61 <String[166]\: lstat\x00/Users/christianalfoni/Development/cerebral/packages/cerebral-router/node_modules/@cerebral/monorepo/debugger/cerebral-debugger/node_modules/combined-stream/lib>,cb=0x253882122339 <JS Function lsta...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handle<v8::internal::String>, int, unsigned int) [/usr/local/bin/node]
 5: v8::internal::InternalizedStringKey::AsHandle(v8::internal::Isolate*) [/usr/local/bin/node]
 6: v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::HashTableKey*) [/usr/local/bin/node]
 7: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [/usr/local/bin/node]
 8: v8::internal::LookupIterator::LookupIterator(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::LookupIterator::Configuration) [/usr/local/bin/node]
 9: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [/usr/local/bin/node]
10: v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x3780a42092a7
13: 0x3780a4557197
npm info lifecycle @cerebral/[email protected]~postlint: @cerebral/[email protected]
[1]    16597 abort      npm run lint

Now, I believe this is related to traversing, because if I delete the node_modules folders of all the packages in the repo the linter works fine.

By default "standard" ignores node_modules folder and we even added an extra config to be certain, from our package.json:

{
  "standard": {
    "parser": "babel-eslint",
    "ignore": [
      "**/node_modules/",
      "**/build/"
    ]
  }
}

Could it be that ignore is only related to actually linting, but it still traverses all files?

I have tried everything... installing latest Node, Npm, delete repo from my machine and clone again. Also tried upping the memory of Node. It just does not work. Funny thing is that it works on some computers. I have a pretty powerful Macbook Pro from 2015.

Any help would be great as this is blocking contributions to our project currently. The repo is: https://github.com/cerebral/cerebral

Run:

npm install
then
npm run setup
and
npm run lint

[feross]

You have quite a unique situation. No other user has reported this error before. I took a look into what the glob package is doing and it's not filtering out node_modules right at the beginning of the search process, even though we passed that option.

I posted a comment here: https://github.com/isaacs/node-glob/issues/270#issuecomment-273949982

Right now, I'm treating this as a bug in glob.

[feross]

Ahh, nevermind. This is not a glob bug. Everything is working correctly. The default standard ignore is node_modules/** which only ignores the root node_modules folders. This is why it's doing so much work in your repo.

Your attempt to ignore all them by adding an explicit ignore of **/node_modules/ was the right idea, but you got the syntax wrong. It's supposed to be **/node_modules/**.

If you put that in, standard runs extremely quickly given the gargantuan size of your repo.

[christianalfoni]

@feross Ah man! Thanks a bunch :-D