Yarn: OOM on `yarn audit`

[taxonomic-blackfish]

Bug description

Due to https://www.npmjs.com/advisories/1488 and https://www.npmjs.com/advisories/1490, running yarn audit on a midsized project will OOM and crash.

Command

yarn audit --json

What is the current behavior?

Prior to those two advisories, yarn audit completed successfully.

What is the expected behavior?

yarn audit should not OOM because a commonly used package (that appears as a dep to many packages) has a vuln.

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x9ef190 node::Abort() [node]
 2: 0x9f13b2 node::OnFatalError(char const*, char const*) [node]
 3: 0xb5da9e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb5de19 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd0a765  [node]
 6: 0xd0adf6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd1760a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd18515 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd19c8c v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [node]
10: 0xd1b026 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0xce8c8c v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
12: 0x10a2305 v8::internal::IncrementalStringBuilder::Extend() [node]
13: 0xdff298 v8::internal::JsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [node]
14: 0xe03fe6 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
15: 0xe02917 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
16: 0xe04f06 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
17: 0xe02917 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
18: 0xe02558 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
19: 0xe02558 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
20: 0xe04f06 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
21: 0xe058b4 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
22: 0xc0f24b v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [node]
23: 0x13a72b9  [node]
29

Steps to Reproduce

Simply running yarn audit. I cannot provide a link to a public repo for this.

Environment

• Node Version: v12.16.1
• Yarn v1 Version: 1.22.4
• OS and version: Linux localhost 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

[bmwalters]

I'm experiencing this now due to https://github.com/lodash/lodash/issues/4744. This is especially painful since the workaround to #6669 is to run yarn audit --json and process the output.

This also seems to be related to / a dupe of #7404.

[taxonomic-blackfish]

Since this is happening again, I would like to suggest buffering output. The output OOMs because the path .data.advisory.findings[].paths gets as large as 20k elements, and each element is some 100+ characters. For example, yarn audit --json | grep '"auditAdvisory"' | jq -r .data will OOM after outputting the first line, and these are quite fast, streaming data processing utilities.

I would assume (I don't know) that using a size limited buffer of objects would be helpful. One iterator creates the JSON strings to be written and adds them to a size-limited FIFO queue. A writer pulls from the queue and writes them to the output (pipe, redirect, stdout, etc). This is more in line with out CLI utils generally work in *nix-land.