Node: unclear why using `for...in` causes out-of-memory error

[HP4k1h5]

• Version: v14.4.0
• Platform: Platform: Darwin 0p05 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64
• Subsystem: Memory 8 GB 2133 MHz LPDDR3

What steps will reproduce the bug?

1) run the following code with the uncommented for... in loop

let obj = {}
for (let i = 0; i < 1e8; i++) {
  obj[i] = i
}

for (o in obj) {
 // if you comment out this for loop, the code runs on my 8GB machine
 // but why is iterating over the loop adding to memory?
}

for (let i = 0; i < 1e8; i++) {
  let b = obj[i]
}

How often does it reproduce? Is there a required condition?

This can be so far 100% reliably reproduced. The code is fine without the for..in and breaks when it is included.

What is the expected behavior?

I wouldnt expect an iterator to occupy enough memory to cause oom

What do you see instead?

<--- Last few GCs --->

[80306:0x104182000] 34747 ms: Mark-sweep 2055.4 (2056.5) -> 2055.5 (2058.5) MB, 1355.7 / 0.0 ms (average mu = 0.091, current mu = 0
.004) allocation failure scavenge might not succeed
[80306:0x104182000] 36128 ms: Mark-sweep 2057.5 (2058.5) -> 2057.5 (2060.3) MB, 1374.2 / 0.0 ms (average mu = 0.049, current mu = 0
.005) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x100bc9ae7 node::Abort() (.cold.1) [/usr/local/bin/node]
2: 0x10008186d node::FatalError(char const, char const) [/usr/local/bin/node]
3: 0x1000819d6 node::OnFatalError(char const, char const) [/usr/local/bin/node]
4: 0x100184fe5 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [/usr/local/bin/node]
5: 0x100184f8f v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [/usr/local/bin/node]
6: 0x1002a5049 v8::internal::Heap::FatalProcessOutOfMemory(char const) [/usr/local/bin/node]
7: 0x1002a63ac v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
8: 0x1002a3d93 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0x1002a255e v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallback
Flags) [/usr/local/bin/node]
10: 0x1002aa520 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin
, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x1002aa576 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigi
n, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x10028658a v8::internal::FactoryBase::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bi
n/node]
13: 0x100289ca7 v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector const&, v8::internal::AllocationT
ype) [/usr/local/bin/node]
14: 0x100294a56 v8::internal::Factory::NumberToStringCacheSet(v8::internal::Handle, int, char const, bool) [/usr
/local/bin/node]
15: 0x100294dc4 v8::internal::Factory::SmiToString(v8::internal::Smi, bool) [/usr/local/bin/node]
16: 0x100397c49 v8::internal::(anonymous namespace)::ElementsAccessorBase ccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)3> >::DirectCollectElementIndicesImpl(v8::
internal::Isolate, v8::internal::Handle, v8::internal::Handle, v8::internal::Get
KeysConversion, v8::internal::PropertyFilter, v8::internal::Handle, unsigned int, unsigned int) [/usr/local/
bin/node]
17: 0x1003965e2 v8::internal::(anonymous namespace)::ElementsAccessorBase ccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)3> >::PrependElementIndices(v8::internal::
Handle, v8::internal::Handle, v8::internal::Handle, v8:
:internal::GetKeysConversion, v8::internal::PropertyFilter) [/usr/local/bin/node]
18: 0x10040415c v8::internal::FastKeyAccumulator::GetKeysFast(v8::internal::GetKeysConversion) [/usr/local/bin/node]
19: 0x100402638 v8::internal::FastKeyAccumulator::GetKeys(v8::internal::GetKeysConversion) [/usr/local/bin/node]
20: 0x1004e279c v8::internal::Runtime_ForInEnumerate(int, unsigned long, v8::internal::Isolate) [/usr/local/bin/node]
21: 0x100754239 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
22: 0x2ecd854430b3
[1] 80306 abort node test.js

Additional information

i may have a misunderstanding about all the background processes that are incurred when using for...in, though the documentation does not seem to indicate that it is creating a copy, or that memory concerns should be taken into account.

[HP4k1h5]

having tried to use Object.keys(), myself, and seen that go oom, i have a suspicion that either GetKeys or GetKeysFast is going to necessarily cause this issue. probably unavoidable then, im happy to close unless its possible to add to documentation somewhere that object iterators can have high memory costs associated.

[jasnell]

Without trying it out myself, I would say that it's because the for in uses the iterator pattern which, I believe, is allocating an object on every iteration, so you're allocating 1e8 additional objects in this loop.

[addaleax]

for in does not use iterators, and even if it did, the iterator objects could be GC’ed.

@HP4k1h5 You’re right, GetKeys() is the issue here, because it converts each of the object’s keys into a strings (which take up quite a bit more memory than integer indices), and tries to allocate all of them before performing the iteration steps.

[jasnell]

Ah right, for in, not for of.

[HP4k1h5]

thank you for the explanation. ill close, as i dont even know where/if node has documentation related to native js constructs.

[addaleax]

@HP4k1h5 We don’t have documentation related to JS language features, no. We usually link to npm, in this case that would be https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...in.