Arangodb: Removal of highly connected vertex crashes

[wizzl35]

my environment running ArangoDB

I'm using the latest ArangoDB of the respective release series:

• [ ] 2.8
• [X] 3.0
• [ ] self-compiled devel branch

On this operating system:

• [ ] DCOS on
  
  
  
  • [ ] AWS
  
  
  • [ ] Azure
  
  
  • [ ] own infrastructure
  
  
• [ ] Linux
  
  
  
  • [X] Debian .deb
  
  
  • [ ] Ubuntu .deb
  
  
  • [ ] SUSE .rpm
  
  
  • [ ] RedHat .rpm
  
  
  • [ ] Fedora .rpm
  
  
  • [ ] Gentoo
  
  
  • [ ] docker - official docker library
  
  
  • [ ] other:
  
  
• [ ] Windows, version:
• [ ] MacOS, version:

Foxx

Using the remove() function in require('@arangodb/general-graph') module to remove a highly connected vertex that has 300k plus edges causes Arango to crash without traceback. I believe it is a memory issues. Let me know if there is anymore information I can provide.

[jsteemann]

Reproduced in 3.0 by removing a vertex with 1M edges. V8 seems to run out of memory when executing the JavaScript code that collects the edges to be removed.
It happened once, but not every time I tried the removal:

<--- Last few GCs --->

  744506 ms: Scavenge 1257.1 (1416.2) -> 1257.1 (1427.2) MB, 26.2 / 0 ms (+ 0.0 ms in 72 steps since last GC) [allocation failure].
  744615 ms: Scavenge 1268.3 (1427.2) -> 1268.3 (1432.2) MB, 25.1 / 0 ms (+ 0.0 ms in 180 steps since last GC) [allocation failure].
  744680 ms: Scavenge 1272.8 (1432.2) -> 1272.8 (1434.2) MB, 30.9 / 0 ms (+ 0.0 ms in 72 steps since last GC) [allocation failure].


<--- JS stacktrace --->
Cannot get stack trace in GC.

#
# Fatal error in Scavenger: semi-space copy

# Allocation failed - process out of memory
#

Illegal instruction (core dumped)

[jsteemann]

It looks like V8 can run out of memory when the list of connected edges gets big. Whether or not it runs out of memory obviously depends on the state of its heap.

A mitigation may be to start ArangoDB with the option --javascript.v8-options="--max_old_space_size=4096" in order to increase the max heap size of V8. This allows V8 to consume more memory so out-of-memory situations become less likely.

AFAIK ArangoDB 3.1 will by default increase V8's max_old_space_size value on startup, at least compared to the default value in 3.0, so V8 running out of memory is less likely in 3.1 with default settings.

Additionally 3.1 will try to catch some of V8's out-of-memory errors (though they are not meant to be catchable). For example, for reproduction I tried setting the max_old_space_size to just 512 MB in order to make the operations fail more often. 3.1 then aborted the removal operation without crashing and printed:

2016-11-03T17:06:51 [30932] ERROR encountered out-of-memory error
2016-11-03T17:06:51 [30932] WARNING reached heap-size limit, interrupting V8 execution (heap size limit 603979776, used 442991576)
2016-11-03T17:06:51 [30932] WARNING reached heap-size limit, interrupting V8 execution (heap size limit 603979776, used 443008096)

(which should be much better than going down entirely)

[hkernbach]

ArangoDB 3.1 has been released. Please note that the download urls and repositories have changed slightly.

Download or see Release Notes for details.