Node: Crash in GC Scavenge

[surajwy]

• Version:v6.11.0
• Platform:Amazon Linux 4.9.27-14.31.amzn1.x86_64

I am getting a crash in "std::_Rb_tree_rebalance_for_erase" during a GC scavenge call.
I do not have a reproducible case.

Stack

(lldb) v8 bt
 * SBThread: tid = 0x0000  * frame #0: 0x00007f9089d54e24 libstdc++.so.6`std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) + 388
    frame #1: 0x0000000000c89339 node`std::_Rb_tree<void*, std::pair<void* const, unsigned long>, std::_Select1st<std::pair<void* const, unsigned long> >, std::less<void*>, std::allocator<std::pair<void* 
const, unsigned long> > >::erase(void* const&) + 137
    frame #2: 0x0000000000c899bb node`v8::internal::ArrayBufferTracker::Promote(v8::internal::JSArrayBuffer*) + 251
    frame #3: 0x0000000000c9fe9e node`v8::internal::Heap::IteratePromotedObjectPointers(v8::internal::HeapObject*, unsigned char*, unsigned char*, bool, void (*)(v8::internal::HeapObject**, v8::internal::
HeapObject*)) + 302
    frame #4: 0x0000000000dbbd3b node`void v8::internal::BodyDescriptorBase::IterateBodyImpl<v8::internal::ObjectVisitor>(v8::internal::HeapObject*, int, int, v8::internal::ObjectVisitor*) + 251
    frame #5: 0x0000000000dbbe20 node`void v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::internal::HeapObject*, int, v8::internal::ObjectVisitor*>(v8::internal::InstanceType, 
v8::internal::HeapObject*, int, v8::internal::ObjectVisitor*) + 208
    frame #6: 0x0000000000c9638d node`v8::internal::Heap::DoScavenge(v8::internal::ObjectVisitor*, unsigned char*) + 269
    frame #7: 0x0000000000ca168a node`v8::internal::Heap::Scavenge() + 3050
    frame #8: 0x0000000000ca6d70 node`v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 336
    frame #9: 0x0000000000ca771a node`v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) + 330
    frame #10: 0x0000000000c5c67c node`v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) + 108
    frame #11: 0x0000000000eb7c91 node`v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) + 641
    frame #12: 0x00000730b7c079a7 <exit>

Register values:

(lldb) register read
General Purpose Registers:
       rax = 0x00000000056195f0
       rbx = 0x000000000661b450
       rcx = 0x0000000000000000
       rdx = 0x0000000000000001
       rdi = 0x0000000000000000
       rsi = 0x0000000005e9cb50
       rbp = 0x0000000000000000
       rsp = 0x00007ffd992060d8
        r8 = 0x0000000000000000
        r9 = 0x0000000004f4d0c0
       r10 = 0x0000101a937d4831
       r11 = 0x0000000003e1cc50
       r12 = 0x0000000000000000
       r13 = 0x0000000003e55630
       r14 = 0x0000000003e55628
       r15 = 0x0000000005d559e0
       rip = 0x00007f9089d54e24  libstdc++.so.6`std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) + 388
    rflags = 0x0000000000010246
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000

Attached is the disassemble taken using gdb.
gdb-disassemble.txt

[addaleax]

Can you provide a core dump in some way? Does this reproduce with the latest 6.x version (v6.12.0)? Does your application use any native modules (find node_modules -name '*.node')?

[surajwy]

I can't provide the core dump. Sorry about that.
Will try it with v6.12.0 and will let you know.
Yes, my app uses native modules.

[bnoordhuis]

The offending instruction is this:

   0x00007f9089d54e20 <+384>:   mov    0x18(%rbx),%rdi
=> 0x00007f9089d54e24 <+388>:   mov    (%rdi),%edx

With rbx = 0x000000000661b450 (which looks valid) and rdi = 0x0000000000000000 so it's a null pointer dereference from a field in the data structure that rbx points to, which is probably an rbtree node.

[hashseed]

@psmarshall may be familiar with this code, even though version 6 is a bit old.

[psmarshall]

I talked to some GC folks. Node 6.11.0 is using v8 5.1.281.102 which has a very outdated ArrayBufferTracker. There could definitely be concurrency issues, particularly if another thread calls v8::ArrayBuffer::Externalize() which leads to ArrayBufferTracker::Unregister() which modifies live_array_buffers_for_scavenge_ without taking the ArrayBufferTracker mutex.

The safest fix is probably to take the mutex (like is done on the first line of ArrayBufferTracker::Promote) each time any of the std::maps are touched, i.e. at the top of RegisterNew, Unregister, MarkLive, FreeDead and PrepareDiscoveryInNewSpace in ArrayBufferTracker.

[bnoordhuis]

Thanks, @psmarshall. We'll want to wait for confirmation that the crash isn't caused by a buggy add-on but that sounds like a good way forward.

Alternatively, @nodejs/lts - has upgrading V8 in v6.x been discussed? We should be able to move up a few minors with only, ah, minor tweaks for ABI and API compatibility. Once-in-a-lifetime offer: I volunteer to do the work.

[MylesBorins]

@bnoordhuis putting this on the LTS agenda

[gibfahn]

We should be able to move up a few minors with only, ah, minor tweaks for ABI and API compatibility. Once-in-a-lifetime offer: I volunteer to do the work.

+1 to upgrading if it's going to fix bugs

+1 to Ben doing the work (unless @targos insists on doing it 😁 )

[targos]

unless @targos insists on doing it 😁

[surajwy]

We'll want to wait for confirmation that the crash isn't caused by a buggy add-on

@bnoordhuis Please give me some steps to approve/discard this add-on suspicion.

[bnoordhuis]

@surajwy The most surefire way is to remove them from your application, if that's an option. Otherwise we'll have to wait until someone else reports/confirms the issue.

[surajwy]

I just got a crash on node v6.12.0

(lldb) v8 bt                                                                                                                                                                                                
 * SBThread: tid = 0x0000  * frame #0: 0x00007fad498c4e2a libstdc++.so.6`std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) + 394
    frame #1: 0x0000000000c8b5b9 node`std::_Rb_tree<void*, std::pair<void* const, unsigned long>, std::_Select1st<std::pair<void* const, unsigned long> >, std::less<void*>, std::allocator<std::pair<void* 
const, unsigned long> > >::erase(void* const&) + 137
    frame #2: 0x0000000000c8c3a6 node`v8::internal::ArrayBufferTracker::FreeDead(bool) + 102
    frame #3: 0x0000000000ca3a5c node`v8::internal::Heap::Scavenge() + 3388
    frame #4: 0x0000000000ca9030 node`v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 336
    frame #5: 0x0000000000ca99da node`v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) + 330
    frame #6: 0x0000000000c5e8fc node`v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) + 108
    frame #7: 0x0000000000eba441 node`v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) + 641
    frame #8: 0x00003d232e6079a7 <exit>

Below is the crash point from disassembly:

   0x00007fad498c4e28 <+392>:   je     0x7fad498c4e60 <_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+448>
=> 0x00007fad498c4e2a <+394>:   mov    0x10(%rdi),%rax

Values of registers:

General Purpose Registers:
       rax = 0x0000000003a8e7c0
       rbx = 0x0000000003d1c790
       rcx = 0x0000000000000000
       rdx = 0x0000000003c1c360
       rdi = 0x0000000000000000
       rsi = 0x00000000027f6670
       rbp = 0x0000000000000000
       rsp = 0x00007ffc6be307a8
        r8 = 0x0000000000000000
        r9 = 0x0000000003fa1290
       r10 = 0x00007fad491157b8  main_arena + 88
       r11 = 0x0000000003c44820
       r12 = 0x0000000000000000
       r13 = 0x00000000027f6670
       r14 = 0x00000000027f6668
       r15 = 0x000000000399d530
       rip = 0x00007fad498c4e2a  libstdc++.so.6`std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) + 394
    rflags = 0x0000000000010246
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000

[surajwy]

Hi All,

I've got another crash at node v6.12.0. This time I have removed all my add-ons expect microtime.
Microtime is pretty famous one. Now I think these crashes are not because of a buggy add-on.

(lldb) v8 bt
 * SBThread: tid = 0x0000  * frame #0: 0x00007f7d1c640d60 libstdc++.so.6`std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) + 192
    frame #1: 0x0000000000c8b5b9 node`std::_Rb_tree<void*, std::pair<void* const, unsigned long>, std::_Select1st<std::pair<void* const, unsigned long> >, std::less<void*>, std::allocator<std::pair<void* const, unsigned long> > >::erase(void* const&) + 137
    frame #2: 0x0000000000c8c3a6 node`v8::internal::ArrayBufferTracker::FreeDead(bool) + 102
    frame #3: 0x0000000000ca3a5c node`v8::internal::Heap::Scavenge() + 3388
    frame #4: 0x0000000000ca9030 node`v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) + 336
    frame #5: 0x0000000000ca99da node`v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) + 330
    frame #6: 0x0000000000c68065 node`v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::PretenureFlag, v8::internal::Handle<v8::internal::AllocationSite>) + 565
    frame #7: 0x0000000000c686a9 node`v8::internal::Factory::NewJSArrayWithElements(v8::internal::Handle<v8::internal::FixedArrayBase>, v8::internal::ElementsKind, int, v8::internal::PretenureFlag) + 105
    frame #8: 0x0000000000c4f898 node`v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedObjectElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)2> >::Splice(v8::internal::Handle<v8::internal::JSArray>, unsigned int, unsigned int, v8::internal::Arguments*, unsigned int) + 904
    frame #9: 0x00000000009eaf10 node`v8::internal::Builtin_Impl_ArraySplice(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)0>, v8::internal::Isolate*) + 976
    frame #10: 0x00000000009eb296 node`v8::internal::Builtin_ArraySplice(int, v8::internal::Object**, v8::internal::Isolate*) + 262
    frame #11: 0x00000067685092a7 <exit>

[targos]

@nodejs/lts have you discussed this issue?

[apapirovski]

I'm going to close this out given the lack of updates in about a year. I don't see this going anywhere given that 6.x is EOL relatively soon.