Janusgraph: Memory Leak in JanusGraph Framework

[shubham190892]

I am using janusgraph for 6 months. Now getting OOM and after restarting the service It runs fine for 4-5 days. I created the heap dump and generated the analysis chart. I am not able to figure out what is the cause of OOM. Please find the screen shot and help.

[doofin]

can you provide more info ? like which backend,etc

[shubham190892]

I am using java 8 , including janusgraph 0.2.0 as dependency and 3 node cassandra cluster for backend storage. Please let me know If anything specific you want to know.

[doofin]

My setting is janusgraph 0.3.1 with cassandra ,have you tried the new version?

[makpcheng]

I am also getting a memory leak. When I analyze the heap dump, I see an abundance of CacheVertex objects not being cleaned up.

This can easily be reproduced by exporting a large database (exporting graphml). Memory heap continues to grow without being cleaned up after export is completed.

[marco-scoppetta]

Hey @makpcheng, we are having the same issue. When opening a graph and trying to export data from it, we can see the memory heap growing - eventually leading to OutOfMemory caused by GC not being able to free up any more memory.

Have you managed to find a workaround for this? Any hints?

[porunov]

I am not sure but this could be connected to #1401 which is fixed in 0.3.2 version of JanusGraph. Are you able to reproduce this issue with the 0.3.2 version?

[marco-scoppetta]

@porunov I just tried, it looks a bit better, but still leaking. I will try to dig a bit deeper, please let me know if anyone has any more advice.

[marco-scoppetta]

I find this comment to be potentially interesting:

//TODO: release non crucial data structures to preserve memory?

on this line when closing a Transaction:

https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardJanusGraphTx.java#L1421

I will keep searching, but as always if anyone has any idea please let me know

[makpcheng]

@marco-scoppetta I'm sorry we never did have a work around for it, other than to not use JanusGraph. We were just prototyping with JanusGraph as an investigation to see how scalability and performance is, and unfortunately it fell short of our expectations.

[marco-scoppetta]

@porunov a couple of points:

• regarding this comment: //TODO: release non crucial data structures to preserve memory?, I have tried closing all the open data structures in releaseTransaction() and that does not seem to make a difference, so I think the todo is not needed

• apparently Janus starts leaking when using graph.tx() rather than graph.newTransaction(),
  I see in the code that there is some magic going on with tx() method (it always returns the same GraphTransaction object which in turn automagically calls startNewTx() every time any operation tries to execute queries on the graph). Handling transaction through explicitly requested new transactions (graph.newTransaction()) seems to solve the leak.

I think documentation should be a bit more clear on what are the differences of the 2 approaches, I see that the main explanation is:

Transactions are started automatically with the first operation executed against the graph. One does NOT have to start a transaction manually. 
The method newTransaction is used to start multi-threaded transactions only.

So it seems that someone interested in ThreadBound transaction should always rely on Janus automatic handling of transactions. But again, it seems to be leaking for us.
Our current workaround for threadBound transactions is

[...]
StandardJanusGraph graph = (StandardJanusGraph) graphBuilder.open();
JanusGraphTransaction tx = graph.newThreadBoundTransaction();
[...]
tx.close();

With this approach we can keep the graph open for a very long time, opening and closing transactions and avoiding OOM exceptions (memory shows no sign of leaking whatsoever).

Maybe I am missing something, but I thought I'd share this for someone looking for similar solutions

[porunov]

@marco-scoppetta Thank you for sharing it!

graph.tx() returns current thread local transaction and graph.newTransaction() creates a transaction which isn't related to the current thread.

I didn't research the core of this opened issue. I think there could be a memory leak issue if we work with transactions and then don't close them explicitly.
For example, there could be a non obvious issue with Tomcat servlet container or other similar containers which are using cached thread pool executor in case when we don't close transactions at the end. In cached thread pool stale threads are removed from the pool and new threads are created when no threads are available. So, in case when we have a user API endpoint something like GET: myserver.com/users/123 which uses JanusGraph to retrieve user information like .V().hasLabel("user").has("user_id", id).next() and we don't close the transaction in the end it means that we are creating a new thread local transaction which won't be closed even after the thread is removed from the pool. I think it could lead to small memory leak (but I am not sure because I didn't test this scenario). I don't think that this scenario is related for this issue but it might help someone.

Do I understand it correctly that there is a memory leak after closing thread local transaction with graph.tx().close() in the thread where it was opened?
Do you have the same memory leak when you execute graph.tx().commit() or graph.tx().rollback() for the current thread?

You are right about documentation. It should definitely be improved.

[marco-scoppetta]

@porunov

I just created a very simple snippet that is highlighting the issue we're having.
First let me say that in a single threaded environment, opening and closing threadBound transactions
does not seem to create any issue. (i.e. when I am about to close the graph, regardless of the number of opened and closed transactions, I see that in memory there is only 1 StandardJanusGraphTx)

The problem seems to appear when using a multithreaded approach:
In this picture we are:

• spawning 4 threads
  
  
  
  • in each thread we request a new Transaction with graph.tx()
  
  
  • in each thread we close the tx with tx.commit()
  
  

Setting a breakpoint right before closing the graph, once all the threads are done, shows how in memory we still have 4 instances on StandardJanusGraphTx in the main thread:

When trying the same snippet with graph.newTransaction() (or also graph.newThreadBoundTransaction() if you cast the graph to a StandardJanusGraph) you can see how, before closing the graph we have 0 instances of StandardJanusGraphTx:

Note that the only difference in code between the two pics is line 44.

I hope this helps, please let me know if I am doing something wrong here

[porunov]

@marco-scoppetta Thank you for this investigation!
Looks like a bug for me. We should dig into this issue

[porunov]

@marco-scoppetta Could you also try to execute any traversal before the commit in the first case?
Like: graph.traversal().V().hasNext(); and then graph.tx().commit().
Is the result the same?

[marco-scoppetta]

@porunov I have tried with graph.traversal.V().hasNext(), same result.

I think the issue here is quite simple actually, sorry for the confusion but apparently this is a leak that happens only in particular cases, namely when you don't kill all the spawned threads.

graph.tx() relies on the txs field inside JanusGraphBlueprintsGraph, and txs is a ThreadLocal variable, so each of my 4 threads will initialise a new txs, but when we close the transaction we don't set txs.set(null). So for as long as we will have the threads alive, txs will be still retaining a transaction in each thread.

Killing the threads solves the issue (note that I moved up executor.shutdownNow(), before the graph.close()):

I guess in Janus we could force txs.set(null) or txs.remove() we the user closes the tx, so that we don't have ask the users to always make sure all their threads are dead? Or maybe it's good that the user goes out of memory if he/she forgets to kill all the threads?

Either way it could maybe be useful to have a warning in the documentation.

Hope this solves the issue for someone else!

[mad]

Related issue https://github.com/JanusGraph/janusgraph/issues/1341

Currently exists two leaks in JanusGraphBlueprintsGraph.txs and GraphTransaction.close() do not invoke super AbstractTransaction.close() that clear threalocals

[MikeThomsen]

@porunov I think I too can confirm that this is happening. I was sending a large volume of scripts like this:

//Some traversal code
graph.tx().commit()
graph.tx().close()

Even with JanusGraph having 2GB of RAM allocated, it was consistently experiencing OutOfMemoryError issues.

Changed it like this, and it appears to be able to chug through the same submissions:

def tx = graph.newTransaction()
def t = tx.traversal()
//traversal code
tx.commit()
tx.close()

I was hammering it in all cases with Apache NiFi (we just added support for Janus :-D ) using three threads on a custom record processor (input was 10k records in JSON format x 3 threads). After about 60k-90k records in the first scenario, the leak happened.

[MikeThomsen]

Forgot to mention: HBase 2.0.5 + ElasticSearch 6.7.0 as the data stores for JanusGraph 0.3.2.

[porunov]

@MikeThomsen @marco-scoppetta @shubham190892 could you please check if the fix #1653 helps with your issues?

For simplicity you can clone the fork https://github.com/mad/janusgraph
and make:

git checkout 1341-fix-potential-thread-local-leaks
mvn clean install -DskipTests=true
rm janusgraph-codepipelines-ci/target/*.jar
cp janusgraph*/target/*.jar /your/project/libs/

If you are using Gradle you can include you jars from libs folder like:

compile fileTree(dir: 'libs', include: ['*.jar'])

Also, not sure about current release but in older releases those dependencies were also necessary:

compile group: 'io.vavr', name: 'vavr', version: '0.9.2'
compile group: 'com.github.stephenc.high-scale-lib', name: 'high-scale-lib', version: '1.1.1'

Thank you!

[munprathap]

Hi Team, I'm running a java app which runs a janusgraph server and connects to a cassandra db as it's backend storage.
we started seeing java heap issue and GC causing OOM exception.
we analyzed the dumps and found that cache vertex objects was filling up the memory causing OOM exception to occur and it was not getting cleaned up.
To fix OOM issue many people suggested that consider adding frequent commits.We tried that and it was working fine for some days and again we started seeing this issue.
Can someone please guide us what workaround we can do to fix this issue and it is impacting us a lot since we are in production now.
Any help to fix this issue is greatly appreciated.

[li-boxuan]

My hypothesis is, a lot of people are seeing "CacheVertex" objects not cleaned up because there are transaction leaks, plus the transactions contain cached vertices. With https://github.com/JanusGraph/janusgraph/pull/2413 and https://github.com/JanusGraph/janusgraph/pull/2457, we explicitly clean the transaction level cache, which should reduce the chance of getting OOM even if there are transaction leaks. We also have https://github.com/JanusGraph/janusgraph/pull/2472 which aims to fix transaction leaks when using thread-bound transactions (e.g. graph.tx().commit()).

Users might still see memory leak if they close the graph before they call graph.tx().commit()/graph.tx().rollback()/graph.tx().close() unless they are using manual transaction only (i.e. graph.newTransaction()).