Remark: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed

[AlexWayfer]

Subject of the issue

Error at simple remark run.

Your environment

• OS: Arch Linux
• Packages: "remark-cli": "^7.0.0"
• Env: node v11.9.0, npm 6.12.0, yarn 1.19.1

Steps to reproduce

> yarn run remark .

Expected behaviour

Result of linting (or nothing).

Actual behaviour

Pause about 2 minutes and:

> yarn run remark .
yarn run v1.19.1
$ /home/alex/Projects/foo/node_modules/.bin/remark .

<--- Last few GCs --->

[16692:0x3b3e640]    62016 ms: Mark-sweep 1326.5 (1441.8) -> 1314.4 (1441.8) MB, 494.6 / 2.3 ms  (average mu = 0.116, current mu = 0.039) allocation failure scavenge might not succeed
[16692:0x3b3e640]    62542 ms: Mark-sweep 1328.3 (1441.8) -> 1316.2 (1442.3) MB, 504.9 / 2.3 ms  (average mu = 0.079, current mu = 0.041) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x69bea9cfc5d]
    1: StubFrame [pc: 0x69bea9d0f81]
Security context: 0x0123ce91d949 <JSObject>
    2: resolve(aka resolve) [0x13031b81fcb9] [path.js:~1068] [pc=0x69bea9e0c8d](this=0x3f09ae0025b1 <undefined>)
    3: arguments adaptor frame: 2->0
    4: statAndIgnore(aka statAndIgnore) [0x14708d324701] [/home/alex/Projects/foo/node_modules/unified-engine/lib/finder.js:~219] [pc=0x69beaa2fd68](this=0x3f0...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191102.222326.16692.001.json
Node.js report completed
 1: 0x9518b0 node::Abort() [/home/alex/.nodenv/versions/11.9.0/bin/node]
 2: 0x9525f9 node::OnFatalError(char const*, char const*) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 3: 0xb28c2e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 4: 0xb28e64 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 5: 0xf25bd2  [/home/alex/.nodenv/versions/11.9.0/bin/node]
 6: 0xf25cd8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 7: 0xf323f8 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 8: 0xf32f0b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/11.9.0/bin/node]
 9: 0xf35c41 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/alex/.nodenv/versions/11.9.0/bin/node]
10: 0xeffe74 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/alex/.nodenv/versions/11.9.0/bin/node]
11: 0x11b582e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/alex/.nodenv/versions/11.9.0/bin/node]
12: 0x69bea9cfc5d 
error Command failed with signal "SIGABRT".

report.20191102.222326.16692.001.zip

[AlexWayfer]

I've updated Node.js to 13.0.1 and the same:

> yarn run remark .
yarn run v1.19.1
$ /home/alex/Projects/foo/node_modules/.bin/remark .

<--- Last few GCs --->

[18508:0x4ec40b0]   105757 ms: Mark-sweep 2033.9 (2060.4) -> 2027.1 (2075.6) MB, 1075.5 / 2.4 ms  (average mu = 0.135, current mu = 0.032) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x138df79]
Security context: 0x28bd56cc08d1 <JSObject>
    1: resolve(aka resolve) [0xd04754423a9] [path.js:~974] [pc=0x2e11c2061b2](this=0x238d6bbc04b1 <undefined>)
    2: arguments adaptor frame: 2->0
    3: statAndIgnore(aka statAndIgnore) [0x2e14be59cbe1] [/home/alex/Projects/foo/node_modules/unified-engine/lib/finder.js:222] [bytecode=0xa1178dd7b31 offset=64](this=0x238d6bbc04b1 <undefined>,0x1d59...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x9e8320 node::Abort() [/home/alex/.nodenv/versions/13.0.1/bin/node]
 2: 0x9e9506 node::OnFatalError(char const*, char const*) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 3: 0xb4333e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 4: 0xb436b9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 5: 0xcf0495  [/home/alex/.nodenv/versions/13.0.1/bin/node]
 6: 0xcf0b26 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 7: 0xcfd35a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 8: 0xcfe265 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 9: 0xcff8ff v8::internal::Heap::HandleGCRequest() [/home/alex/.nodenv/versions/13.0.1/bin/node]
10: 0xcae465 v8::internal::StackGuard::HandleInterrupts() [/home/alex/.nodenv/versions/13.0.1/bin/node]
11: 0x100b3a3 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/home/alex/.nodenv/versions/13.0.1/bin/node]
12: 0x138df79  [/home/alex/.nodenv/versions/13.0.1/bin/node]
error Command failed with signal "SIGABRT".

[ChristianMurphy]

@AlexWayfer sorry hear you're running into an issue.
A few question to help debug:

1. ~what operating system are you using?~ Arch Linux
2. what does the remark script in the package.json look like?
3. are you using a remark config? (If so what is it)
4. does runnin npx remark . work on your system?

[AlexWayfer]

what does the remark script in the package.json look like?

yarn run remark is like "run executable from local packages named remark". I don't have remark script in package.json, it's the executable from remark-cli.

are you using a remark config? (If so what is it)

It occurs without any config and with example config (from README) with remark-preset-lint-recommended in .remarkrc.

does runnin npx remark . work on your system?

No:

> npm remark .

Usage: npm <command>

where <command> is one of:
    access, adduser, audit, bin, bugs, c, cache, ci, cit,
    clean-install, clean-install-test, completion, config,
    create, ddp, dedupe, deprecate, dist-tag, docs, doctor,
    edit, explore, get, help, help-search, hook, i, init,
    install, install-ci-test, install-test, it, link, list, ln,
    login, logout, ls, org, outdated, owner, pack, ping, prefix,
    profile, prune, publish, rb, rebuild, repo, restart, root,
    run, run-script, s, se, search, set, shrinkwrap, star,
    stars, start, stop, t, team, test, token, tst, un,
    uninstall, unpublish, unstar, up, update, v, version, view,
    whoami

npm <command> -h  quick help on <command>
npm -l            display full usage info
npm help <term>   search for help on <term>
npm help npm      involved overview

Specify configs in the ini-formatted file:
    /home/alex/.npmrc
or on the command line via: npm <command> --key value
Config info can be viewed via: npm help config

[email protected] /home/alex/.nodenv/versions/13.0.1/lib/node_modules/npm

Did you mean this?
    restart

npm run remark . not too:

> npm run remark .
npm ERR! missing script: remark

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/alex/.npm/_logs/2019-11-02T19_59_20_437Z-debug.log

[ChristianMurphy]

@AlexWayfer I think the confusion here might be in how yarn run and npm run work.
docs: https://yarnpkg.com/lang/en/docs/cli/run and https://docs.npmjs.com/cli/run-script.html

Both will look in the scripts section of the package.json file
if you want to use yarn run remark add

"scripts": {
  "remark": "remark ."
}

to the _package.json_

Or to use remark without using a script in the package.json
use npx -p remark-cli remark .

Which can run the binary directly
docs: https://github.com/npm/npx#npx1----execute-npm-package-binaries

[ChristianMurphy]

does running npx remark . work on your system?

No:

npm remark .

npx and npm are two different scripts.
try npx remark .

[AlexWayfer]

@AlexWayfer I think the confusion here might be in how yarn run and npm run work.
docs: https://yarnpkg.com/lang/en/docs/cli/run and https://docs.npmjs.com/cli/run-script.html

I think you can find this interesting: https://yarnpkg.com/en/docs/cli/#locally-installed-clis

And the result of yarn remark . the same. Pay attention: not an error like "this command not found".

Also pay attention to Yarn logs:

$ /home/alex/Projects/foo/node_modules/.bin/remark .

So…

if you want to use yarn run remark add

OK, I've added lint to not be confused:

"scripts": {
  "lint": "remark ."
}

The same:

> yarn run lint
yarn run v1.19.1
$ remark .

<--- Last few GCs --->

[9799:0x5fbb0a0]   105031 ms: Mark-sweep 2035.5 (2063.9) -> 2027.6 (2064.6) MB, 1051.1 / 2.8 ms  (average mu = 0.137, current mu = 0.016) allocation failure scavenge might not succeed
[9799:0x5fbb0a0]   105930 ms: Mark-sweep 2036.2 (2064.6) -> 2028.7 (2077.1) MB, 886.4 / 3.1 ms  (average mu = 0.084, current mu = 0.013) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x138df79]
Security context: 0x0aa8326c08d1 <JSObject>
    1: directory(aka directory) [0x1d36cfc641b1] [/home/alex/Projects/foo/node_modules/unified-engine/lib/finder.js:~180] [pc=0x1db321ef831f](this=0x392ca67804b1 <undefined>,0x392ca67801b9 <null>,0x232b1c8aabd1 <JSArray[15]>)
    2: /* anonymous */ [0x9989bc285f1] [fs.js:~146] [pc=0x1db321ef363d](this=0x05d769a302d9 <FSReqCallback map = 0x35b164a19529>)
  ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191103.002656.9799.0.001.json
Node.js report completed
 1: 0x9e8320 node::Abort() [/home/alex/.nodenv/versions/13.0.1/bin/node]
 2: 0x9e9506 node::OnFatalError(char const*, char const*) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 3: 0xb4333e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 4: 0xb436b9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 5: 0xcf0495  [/home/alex/.nodenv/versions/13.0.1/bin/node]
 6: 0xcf0b26 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 7: 0xcfd35a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 8: 0xcfe265 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/13.0.1/bin/node]
 9: 0xd00d0c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/alex/.nodenv/versions/13.0.1/bin/node]
10: 0xcc75cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/alex/.nodenv/versions/13.0.1/bin/node]
11: 0x100b6de v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/alex/.nodenv/versions/13.0.1/bin/node]
12: 0x138df79  [/home/alex/.nodenv/versions/13.0.1/bin/node]
error Command failed with signal "SIGABRT".

try npx remark .

The same:

> npx remark .                                                                                                                                    03 Nov 00:28:25 

<--- Last few GCs --->

[10362:0x4ebc0f0]   109650 ms: Mark-sweep 2033.3 (2061.9) -> 2025.3 (2062.4) MB, 1031.2 / 2.6 ms  (average mu = 0.148, current mu = 0.032) allocation failure scavenge might not succeed
[10362:0x4ebc0f0]   110792 ms: Mark-sweep 2033.9 (2062.4) -> 2026.4 (2075.4) MB, 1126.3 / 2.8 ms  (average mu = 0.082, current mu = 0.014) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x138df79]
Security context: 0x3135cce008d1 <JSObject>
    1: relative(aka relative) [0x17cc2d380ec1] [path.js:~1053] [pc=0x1dfada792ba8](this=0x12d051cc04b1 <undefined>,0x38eb33ac0219 <String[30]: /home/alex/Projects/foo>,0x03c4b809ff01 <String[175]: /home/alex/Projects/foo/lib/spectator/lib/twitcr/lib/twitcr/lib/twitcr/lib/spectator/lib/ameba/lib/ameba/lib/ameba/src/ameba/rule/performance/first_last_a...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x9e8320 node::Abort() [node]
 2: 0x9e9506 node::OnFatalError(char const*, char const*) [node]
 3: 0xb4333e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb436b9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xcf0495  [node]
 6: 0xcf0b26 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xcfd35a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xcfe265 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xcff8ff v8::internal::Heap::HandleGCRequest() [node]
10: 0xcae465 v8::internal::StackGuard::HandleInterrupts() [node]
11: 0x100b3a3 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x138df79  [node]
fish: “npx remark .” terminated by signal SIGABRT (Abort)

Anything else?

ChristianMurphy added platform/linux type/question status/waiting and removed type/bug status/open labels 1 hour ago

I think that's the mistake.

I guess, you don't want to investigate and resolve this issue. I understand. But I don't know how to help. I open for any another debugging.

I see different files in logs, from node_modules and lib/ (Crystal shards). Maybe remark should lint only .md files.

[AlexWayfer]

> yarn remark **/*.md

...

node_modules/vfile/readme.md
              1:1  error    Error: Could not find module `preset-wooorm`
    at addModule (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:241:27)
    at use (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:202:7)
    at addEach (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:187:9)
    at addPreset (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:163:9)
    at merge (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:149:5)
    at Config.create (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/configuration.js:105:5)
    at done (/home/alex/Projects/who2follow/node_modules/unified-engine/lib/find-up.js:153:20)
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:61:3)

node_modules/wrapped/History.md
              5:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
              6:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             11:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             12:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             13:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             18:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             23:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint
             28:3  warning  Incorrect list-item indent: add 2 spaces              list-item-indent               remark-lint
             33:3  warning  Incorrect indentation before bullet: remove 1 space   list-item-bullet-indent        remark-lint
             33:4  warning  Incorrect list-item indent: add 1 space               list-item-indent               remark-lint
             38:3  warning  Incorrect indentation before bullet: remove 1 space   list-item-bullet-indent        remark-lint
             38:4  warning  Incorrect list-item indent: add 1 space               list-item-indent               remark-lint
             43:3  warning  Incorrect indentation before bullet: remove 2 spaces  list-item-bullet-indent        remark-lint

node_modules/wrapped/Readme.md: no issues found
node_modules/wrappy/README.md: no issues found
node_modules/x-is-string/README.md
              1:1  warning  Missing newline character at end of file              final-newline                  remark-lint
             44:3  warning  Incorrect indentation before bullet: remove 1 space   list-item-bullet-indent        remark-lint
             44:4  warning  Incorrect list-item indent: add 1 space               list-item-indent               remark-lint

node_modules/xtend/README.md: no issues found

1948 messages (✖ 52 errors, ⚠ 1896 warnings)
error Command failed with exit code 1.

That helped.

But there are too many files from node_modules/ anyway.

So… the remark . from README is pretty strange solution. I'm running it in a new project, really small, without another Node packages and with a few code:

> cloc --vcs=git                                                                                                                                  03 Nov 00:33:57 
      13 text files.
      13 unique files.                              
       5 files ignored.

github.com/AlDanial/cloc v 1.84  T=0.08 s (103.2 files/s, 1509.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
YAML                             2             11             21             36
Crystal                          4              6              0             23
JSON                             1              0              0             13
Markdown                         1              3              0              4
-------------------------------------------------------------------------------
SUM:                             8             20             21             76
-------------------------------------------------------------------------------

[AlexWayfer]

I see:

Searches for Markdown extensions

In https://github.com/remarkjs/remark/tree/master/packages/remark-cli

So, it looks like it does not. :upside_down_face:

[AlexWayfer]

Also I see:

If paths or globs to directories are given to the engine, they will be searched for matching files, but node_modules and hidden directories (those starting with a dot, ., such as .git) are normally not searched.

Here: https://github.com/unifiedjs/unified-engine/blob/master/doc/ignore.md#ignoring-1

So, something definitely goes wrong.

[ChristianMurphy]

I think that's the mistake

I guess, you don't want to investigate and resolve this issue. I understand. But I don't know how to help. I open for any another debugging.

I'm not sure if the issue is in remark-cli or upstream in dependencies.
I can add tag back if it makes you feel better.

---

@AlexWayfer from the logs you've shared it looks like remark is working in some folders but not in others.
Could you share some the markdown content from the folder that error?

---

But there are too many files from node_modules/ anyway.

So… the remark . from README is pretty strange solution. I'm running it in a new project, really small, without another Node packages and with a few code:

I see:

Searches for Markdown extensions

In https://github.com/remarkjs/remark/tree/master/packages/remark-cli

So, it looks like it does not. upside_down_face

I'm not sure what you're asking or saying here.
Could you clarify how it relates to the issue you are seeing?

---

If paths or globs to directories are given to the engine, they will be searched for matching files, but node_modules and hidden directories (those starting with a dot, ., such as .git) are normally not searched.

Here: https://github.com/unifiedjs/unified-engine/blob/master/doc/ignore.md#ignoring-1

was there a dot folder in the folder that was causing issue?
If so this _might_ be related to https://github.com/unifiedjs/unified-engine/issues/41

[ChristianMurphy]

/cc @remarkjs/maintainers any additional insights?

[AlexWayfer]

@AlexWayfer from the logs you've shared it looks like remark is working in some folders but not in others.
Could you share some the markdown content from the folder that error?

I didn't understand, sorry. About what directories are you talking?

My thought is that remark . trying to read/parse/lint ALL files: .md, .cr, .js, even in node_modules/ (this is was in logs) and maybe in .git (I'm not sure).

[ChristianMurphy]

remark lint is limited to looking at certain extensions when doing a directory based search.
https://github.com/remarkjs/remark/blob/0114ff967b913ad35fddfd45d44e2d1ed4d5e869/packages/remark-cli/cli.js#L23
https://github.com/sindresorhus/markdown-extensions/blob/9fa017bef29db93c459d7c29eb9064c2dbdf5f79/markdown-extensions.json#L2-L9
https://github.com/unifiedjs/unified-engine/blob/master/doc/options.md#optionsextensions

---

When doing a file based pattern it may look at files with different extensions.

[wooorm]

@AlexWayfer Can you share the project you are running remark from? And, can you run remark . as DEBUG="*" remark . and share the results?

[wooorm]

@AlexWayfer Ping!

[wooorm]

Closing this due to lack of information.

[AlexWayfer]

Closing this due to lack of information.

Sorry, I didn't see emails.

@AlexWayfer Can you share the project you are running remark from?

I don't think so, it's private, but I can try to create project for reproducing.

And, can you run remark . as DEBUG="*" remark . and share the results?

Yes, I can:

> env DEBUG="*" remark .
  unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms
  unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +2ms
  unified-engine:find-up No files found for `/home/alex/Projects/foo` +0ms

<--- Last few GCs --->

[22734:0x5158040]   132888 ms: Mark-sweep 2033.8 (2058.4) -> 2028.0 (2074.4) MB, 1034.3 / 5.3 ms  (average mu = 0.124, current mu = 0.026) allocation failure scavenge might not succeed
[22734:0x5158040]   133929 ms: Mark-sweep 2039.0 (2074.4) -> 2031.9 (2053.1) MB, 228.8 / 5.2 ms  (+ 754.1 ms in 175 steps since start of marking, biggest step 40.7 ms, walltime since start of marking 1041 ms) (average mu = 0.092, current mu = 0.056) final

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x138df79]
    1: StubFrame [pc: 0x1313869]
Security context: 0x03b9fe1808d1 <JSObject>
    2: concat(aka concat) [0x316d3a164db1] [/home/alex/.config/yarn/global/node_modules/unified-args/node_modules/unified-engine/lib/finder.js:~196] [pc=0x3afa74e5e7dc](this=0x295bbf3404b1 <undefined>,0x314bbfe5aa31 <String[26]: expectation_failed_spec.cr>)
    3: directory(aka directory) [0x316d3a164d31] [/home/alex/.config/yarn/...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191203.004919.22734.0.001.json
Node.js report completed
 1: 0x9e8320 node::Abort() [node]
 2: 0x9e9506 node::OnFatalError(char const*, char const*) [node]
 3: 0xb4333e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb436b9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xcf0495  [node]
 6: 0xcf0b26 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xcfd35a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xcfe265 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd00d0c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xcc75cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x100b6de v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x138df79  [node]
fish: “env DEBUG="*" remark .” terminated by signal SIGABRT (Abort)

@wooorm added invalid :no_good_woman:

"The lack of information" is not "invalid", I think.

[AlexWayfer]

I found that the lib/ directory, created by Crystal (like node_modules/) (and ignored by git by default) is the factor for this issue.

I'll publish a project with this directory content (open-sourced).

[AlexWayfer]

Here it is: https://github.com/AlexWayfer/test_remark

[AlexWayfer]

Now I ping you, @wooorm. :grimacing:

Sorry, but I don't want to open a new issue if here is existing one.

[ChristianMurphy]

@AlexWayfer I tested the example provided using:

$ node --version
v13.6.0

$ npm --version
6.13.4

$ remark --version
remark: 11.0.2, remark-cli: 7.0.1

$ DEBUG="*" remark .
  unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms
  unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +4ms
  unified-engine:find-up No files found for `/home/developer/Projects/test/test_remark` +0ms

$  DEBUG="*" remark "**/*.md"
  unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms
  unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +5ms
  unified-engine:find-up No files found for `/home/developer/Projects/test/test_remark/lib/ameba/README.md` +0ms
  unified-engine:file-set-pipeline:stdin Ignoring `streamIn` +0ms
  unified-engine:file-pipeline:read Reading `/home/developer/Projects/test/test_remark/lib/ameba/README.md` in `utf8` +0ms
  unified-engine:file-pipeline:read Reading `/home/developer/Projects/test/test_remark/lib/spectator/README.md` in `utf8` +1ms
  unified-engine:file-pipeline:read Reading `/home/developer/Projects/test/test_remark/lib/twitcr/README.md` in `utf8` +1ms
  unified-engine:file-pipeline:read Read `/home/developer/Projects/test/test_remark/lib/ameba/README.md` (error: null) +1ms
  unified-engine:file-pipeline:read Read `/home/developer/Projects/test/test_remark/lib/spectator/README.md` (error: null) +0ms
  unified-engine:file-pipeline:read Read `/home/developer/Projects/test/test_remark/lib/twitcr/README.md` (error: null) +0ms
  unified-engine:find-up No files found for `/home/developer/Projects/test/test_remark/lib/ameba/README.md` +8ms
  unified-engine:file-pipeline:configure Using settings `{}` +0ms
  unified-engine:file-pipeline:configure Using `0` plugins +0ms
  unified-engine:file-pipeline:parse Parsing `lib/twitcr/README.md` +0ms
  unified-engine:file-pipeline:parse Parsed document +13ms
  unified-engine:file-pipeline:transform Transforming document `lib/twitcr/README.md` +0ms
  unified-engine:file-pipeline:transform Transformed document (error: null) +0ms
  unified-engine:file-pipeline:queue Queueing `lib/twitcr/README.md` +0ms
  unified-engine:file-pipeline:queue Interupting flush: `lib/ameba/README.md` is not finished +0ms
  unified-engine:file-pipeline:queue Interupting flush: `lib/spectator/README.md` is not finished +0ms
  unified-engine:file-pipeline:queue `lib/twitcr/README.md` can be flushed +0ms
  unified-engine:file-pipeline:queue Not flushing: some files cannot be flushed +0ms
  unified-engine:file-pipeline:configure Using settings `{}` +15ms
  unified-engine:file-pipeline:configure Using `0` plugins +0ms
  unified-engine:file-pipeline:parse Parsing `lib/spectator/README.md` +2ms
  unified-engine:file-pipeline:parse Parsed document +43ms
  unified-engine:file-pipeline:transform Transforming document `lib/spectator/README.md` +44ms
  unified-engine:file-pipeline:transform Transformed document (error: null) +0ms
  unified-engine:file-pipeline:queue Queueing `lib/spectator/README.md` +44ms
  unified-engine:file-pipeline:queue Interupting flush: `lib/ameba/README.md` is not finished +0ms
  unified-engine:file-pipeline:queue `lib/spectator/README.md` can be flushed +0ms
  unified-engine:file-pipeline:queue `lib/twitcr/README.md` can be flushed +0ms
  unified-engine:file-pipeline:queue Not flushing: some files cannot be flushed +0ms
  unified-engine:file-pipeline:configure Using settings `{}` +43ms
  unified-engine:file-pipeline:configure Using `0` plugins +0ms
  unified-engine:file-pipeline:parse Parsing `lib/ameba/README.md` +1ms
  unified-engine:file-pipeline:parse Parsed document +9ms
  unified-engine:file-pipeline:transform Transforming document `lib/ameba/README.md` +10ms
  unified-engine:file-pipeline:transform Transformed document (error: null) +0ms
  unified-engine:file-pipeline:queue Queueing `lib/ameba/README.md` +10ms
  unified-engine:file-pipeline:queue `lib/ameba/README.md` can be flushed +1ms
  unified-engine:file-pipeline:queue `lib/spectator/README.md` can be flushed +0ms
  unified-engine:file-pipeline:queue `lib/twitcr/README.md` can be flushed +0ms
  unified-engine:file-pipeline:queue Flushing: all files can be flushed +0ms
  unified-engine:file-pipeline:stringify Not compiling document without output settings +0ms
  unified-engine:file-pipeline:copy Not copying +0ms
  unified-engine:file-pipeline:stdout Ignoring writing to `streamOut` +0ms
  unified-engine:file-pipeline:file-system Ignoring writing to file-system +0ms
  unified-engine:file-pipeline:stringify Not compiling document without output settings +1ms
  unified-engine:file-pipeline:copy Not copying +1ms
  unified-engine:file-pipeline:stdout Ignoring writing to `streamOut` +1ms
  unified-engine:file-pipeline:file-system Ignoring writing to file-system +0ms
  unified-engine:file-pipeline:stringify Not compiling document without output settings +0ms
  unified-engine:file-pipeline:copy Not copying +0ms
  unified-engine:file-pipeline:stdout Ignoring writing to `streamOut` +0ms
  unified-engine:file-pipeline:file-system Ignoring writing to file-system +0ms
lib/ameba/README.md: no issues found
lib/spectator/README.md: no issues found
lib/twitcr/README.md: no issues found

no issues.

Could you try updating to Node 12 or 13?
Node 11 is no longer officially supported by Node or by Unified/Remark.

[AlexWayfer]

Could you try updating to Node 12 or 13?
Node 11 is no longer officially supported by Node or by Unified/Remark.

Yes.

Still an issue on another computer:

> node -v
v13.6.0

> npm -v 
6.13.6

> remark --version
fish: Unknown command remark

> yarn global add remark
yarn global v1.21.1
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...

warning "[email protected]" has no binaries
Done in 2.18s.

> yarn global add remark-cli
yarn global v1.21.1
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
      - remark
Done in 2.68s.

> remark --version
remark: 11.0.2, remark-cli: 7.0.1

> env DEBUG="*" remark .
  unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms
  unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +2ms
  unified-engine:find-up No files found for `/home/alex/Projects/test_remark` +0ms

<--- Last few GCs --->

[26499:0x56110c335590]   140542 ms: Mark-sweep 2035.0 (2074.4) -> 2027.2 (2053.4) MB, 229.9 / 2.5 ms  (+ 764.3 ms in 202 steps since start of marking, biggest step 33.0 ms, walltime since start of marking 1067 ms) (average mu = 0.145, current mu = 0.096) [26499:0x56110c335590]   141533 ms: Mark-sweep 2032.4 (2053.4) -> 2024.5 (2065.2) MB, 979.3 / 2.4 ms  (average mu = 0.085, current mu = 0.012) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x56110b3ec779]
    1: StubFrame [pc: 0x56110b3ed728]
Security context: 0x0ec83e740921 <JSObject>
    2: normalizeString(aka normalizeString) [0x8e4f8e81999] [path.js:~52] [pc=0x3ac98604c11](this=0x2cf6f7fc04b9 <undefined>,0x025df9d335b9 <String[185]: /home/alex/Projects/test_remark/lib/spectator/lib/twitcr/lib/twitcr/lib/spectator/lib/twitcr/lib/spectator/lib/spectator/lib/spectator/lib/spectator/src/spectator/examp...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200113.195707.26499.0.001.json
Node.js report completed
 1: 0x56110aa87171 node::Abort() [node]
 2: 0x56110aa89765 node::OnFatalError(char const*, char const*) [node]
 3: 0x56110abfcf52 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0x56110abfd1b8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x56110ad87466  [node]
 6: 0x56110ad875a9  [node]
 7: 0x56110ad99d8d v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0x56110ad9aab8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0x56110ad9cfec v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0x56110ad9d054 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0x56110ad624fb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
12: 0x56110b0930d0 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
13: 0x56110b3ec779  [node]
fish: “env DEBUG="*" remark .” terminated by signal SIGABRT (Abort)

report.20200113.195707.26499.0.001.zip

[ChristianMurphy]

@AlexWayfer how did you install Node?
Did you get the build from https://nodejs.org/en/download/current directly, through nvm, or through another package manager/method?

[ChristianMurphy]

@AlexWayfer also see https://support.snyk.io/hc/en-us/articles/360002046418-JavaScript-heap-out-of-memory

[AlexWayfer]

@AlexWayfer how did you install Node?
Did you get the build from https://nodejs.org/en/download/current directly, through nvm, or through another package manager/method?

> which node
/home/alex/.nodenv/shims/node

> nodenv versions
* system (set by /home/alex/.nodenv/version)
  13.5.0

> yay -Qs node
local/apm 2.4.5-1
    Atom package manager
local/node-gyp 6.1.0-1
    Node.js native addon build tool
local/nodejs 13.6.0-1
    Evented I/O for V8 javascript

So, I'm using nodenv, but current Node version was system, it's nodejs package for Arch Linux.

I've installed 13.6.0 via nodenv and trying with it…

> nodenv install 13.6.0
Downloading node-v13.6.0-linux-x64.tar.gz...
-> https://nodejs.org/dist/v13.6.0/node-v13.6.0-linux-x64.tar.gz
Installing node-v13.6.0-linux-x64...
Installed node-v13.6.0-linux-x64 to /home/alex/.nodenv/versions/13.6.0

> nodenv local 13.6.0

> env DEBUG="*" remark .
  unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms
  unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +3ms
  unified-engine:find-up No files found for `/home/alex/Projects/test_remark` +0ms

<--- Last few GCs --->

[31548:0x4524cc0]   146265 ms: Mark-sweep 2036.2 (2076.4) -> 2029.8 (2054.2) MB, 254.9 / 2.6 ms  (+ 814.6 ms in 166 steps since start of marking, biggest step 30.9 ms, walltime since start of marking 1132 ms) (average mu = 0.120, current mu = 0.055) final[31548:0x4524cc0]   147356 ms: Mark-sweep 2034.1 (2054.2) -> 2027.6 (2064.4) MB, 1077.6 / 2.6 ms  (average mu = 0.069, current mu = 0.012) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x145c3d9]
    1: StubFrame [pc: 0x145d388]
Security context: 0x027f31240921 <JSObject>
    2: relative(aka relative) [0x299ebf6cadc9] [path.js:~1052] [pc=0x1c26efb4f8e0](this=0x398be89404b9 <undefined>,0x22fa51b70b99 <String[31]: /home/alex/Projects/test_remark>,0x1423a7230df9 <String[160]: /home/alex/Projects/test_remark/lib/spectator/lib/twitcr/lib/twitcr/lib/spectator/lib/twitcr/lib/twitcr/lib/twitcr/lib/twitcr/l...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200113.201746.31548.0.001.json
Node.js report completed
 1: 0xa9d200 node::Abort() [node]
 2: 0xa9f4c2 node::OnFatalError(char const*, char const*) [node]
 3: 0xc06ffe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xc07379 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xdb55f5  [node]
 6: 0xdb5c86 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xdc44f9 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xdc5335 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xdc7e0c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd8e9e4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10dbd3e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x145c3d9  [node]
fish: “env DEBUG="*" remark .” terminated by signal SIGABRT (Abort)

The same.

@AlexWayfer also see https://support.snyk.io/hc/en-us/articles/360002046418-JavaScript-heap-out-of-memory

OK, but I think it's bad work-around. Why remark is trying to lint/open/something else /home/alex/Projects/test_remark/lib/spectator/lib/twitcr/lib/twitcr/lib/spectator/lib/twitcr/lib/spectator/lib/spectator/lib/spectator/lib/spectator/src/spectator/examp... file at all? It's file from lib/, and it's not JS file.

So, we have unified-engine:find-up No files found for '/home/alex/Projects/test_remark' +0ms, then pause (what remark does after "No files found"?), then Node exception.

[ChristianMurphy]

Why remark is trying to lint/open/something else /home/alex/Projects/test_remark/lib/spectator/lib/twitcr/lib/twitcr/lib/spectator/lib/twitcr/lib/spectator/lib/spectator/lib/spectator/lib/spectator/src/spectator/examp... file at all? It's file from lib/, and it's not JS file.

Noting the repetition of lib over and over again, I suspect something isn't resolving correctly on your path.
Another possibility, is there may be a symlink causing infinite recursion somehow.

what remark [do] after "No files found"?

You can see what it does after here: https://github.com/unifiedjs/unified-engine/blob/2f48feaf3483edce7321edb57eed1c67216009a8/lib/find-up.js#L121

[vhf]

I was able to reproduce OP's issue.

@ChristianMurphy Good call regarding symlink loops:

$ cd /tmp

$ g clone AlexWayfer/test_remark

$ cd test_remark

$ npx -p remark-cli remark .
…
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
…

$ find . -follow -printf ""

find: File system loop detected; ‘./lib/twitcr/lib’ is part of the same file system loop as ‘./lib’.
find: File system loop detected; ‘./lib/spectator/lib’ is part of the same file system loop as ‘./lib’.
find: File system loop detected; ‘./lib/ameba/lib’ is part of the same file system loop as ‘./lib’.

$ ls lib/*/lib
9 Jan 13 18:36 lib/ameba/lib -> ../../lib/
9 Jan 13 18:36 lib/spectator/lib -> ../../lib/
9 Jan 13 18:36 lib/twitcr/lib -> ../../lib/

[AlexWayfer]

Noting the repetition of lib over and over again

It's single directory in my example repository.

I suspect something isn't resolving correctly on your path.

What? Which path? How it can be correctly resolved and incorrectly?

You can see what it does after here: https://github.com/unifiedjs/unified-engine/blob/2f48feaf3483edce7321edb57eed1c67216009a8/lib/find-up.js#L121

Thanks. I'm not expert in unifined-engine, but at first sight there is nothing strange or dangerous. It just should to cache nothing without files processing. I'm not sure where is cache storing…

[AlexWayfer]

I was able to reproduce OP's issue.

@ChristianMurphy Good call regarding symlink loops:

$ cd /tmp

$ g clone AlexWayfer/test_remark

$ cd test_remark

$ npx -p remark-cli remark .
…
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
…

$ find . -follow -printf ""

find: File system loop detected; ‘./lib/twitcr/lib’ is part of the same file system loop as ‘./lib’.
find: File system loop detected; ‘./lib/spectator/lib’ is part of the same file system loop as ‘./lib’.
find: File system loop detected; ‘./lib/ameba/lib’ is part of the same file system loop as ‘./lib’.

Oh, thank you! I got it.

Small explanation: lib/ in this repository is like node_modules/, but for Crystal Language.

And packages have sym-links to parent directory for communication with each another.

So, this case should be resolved (fixed, got around) in unified-engine?

[vhf]

So, this case should be resolved (fixed, got around) in unified-engine?

I'm not sure there is much to fix here. In your situation, ignoring lib/*/lib/ should solve your specific issue.

unified-engine seems to handle circular symlinks just fine:

/tmp $ mkdir foo

/tmp $ cd foo

/tmp/foo $ mkdir dir_a

/tmp/foo $ cd dir_a

/tmp/foo/dir_a $ ln -s ../dir_a dir_b

/tmp/foo/dir_a $ cd ..

/tmp/foo $ tree
.
└── dir_a
    └── dir_b -> ../dir_a

2 directories, 0 files

/tmp/foo $ npx -p remark-cli remark .
npx: installed 132 in 4.381s

/tmp/foo $ echo "# test" > dir_a/file.md

/tmp/foo $ npx -p remark-cli remark .
npx: installed 132 in 4.913s
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/dir_b/file.md: no issues found
dir_a/dir_b/file.md: no issues found
dir_a/file.md: no issues found

/tmp/foo $

In your situation, node becomes out of memory because backtracking into all of your files and directories and circular symlinks is just too many paths to keep track of.

[AlexWayfer]

But, do you know what's happening with lib/ after No files found? I can ignore this directory, but maybe there is an issue with behavior.

[vhf]

I don't think there is any issue. After No files found you end up here: https://github.com/unifiedjs/unified-engine/blob/2f48feaf3483edce7321edb57eed1c67216009a8/lib/find-up.js#L64-L69 , where the path to each folder where remark is looking for files gets added to cache. In any sensible setup it would reach the end of all paths and exit. In your setup, the circular symlinks generate so many paths that the cache gets too heavy for node to handle.

It's a configuration issue on your end (you should ignore the circular symlinks), there's not much that can be done on remark's end in my opinion. I mean, there's certainly room for optimization but there's hardly a way of preventing this OOM error from happening. We could probably get remark to visit 500000 paths before running out of memory instead of visiting "only" 100000.

[ChristianMurphy]

I agree with @vhf that the out of memory exception is probably out of scope of unified-engine.
Adding a conditional for the number of files in cache would add overhead, complexity, and could cause issues with projects which _do_ want to handle large numbers of files.

One idea to improve unified-engine could be to add a --[no-]follow-symlinks option, giving an automatic way to ignore symlinks from processing. :thought_balloon:

[AlexWayfer]

One idea to improve unified-engine could be to add a --[no-]follow-symlinks option, giving an automatic way to ignore symlinks from processing. thought_balloon

Symlinks can be nice, but recursive…

OK, I got your position. Thank you for help, I appreciate it.