Redisgraph: Memory leaks when running queries with aliases

[mdecuir]

Running on redisgraph:edge, I can consistently produce a memory leak by running queries that contain aliases. The rate of memory leak seems to increase with the number of distinct match clauses (or possibly just number of distinct aliases) per request.

The simplest query pattern that produces this behavior for me is something like the following:

MATCH (n0:node) WHERE n0.name = 'node0'
RETURN n0

When I run this same query 100k times when the node type I am looking for only has ~128 nodes, I see an increase in memory usage by ~5MB. When I run a query with an additional match clause and return the result of the additional match clause, memory usage increases by ~10MB.

The use case that led me to find this problem is trying to create edges between many nodes that already exist. In doing so, I found that a memory leak was occurring when looking up 2 nodes and creating an edge between them 1 at a time. When I bundled together multiple edge creations into a single request I found that execution time was much faster but the memory leak observed was more severe. Further, I found that when the memory leak got so severe that redisgraph was using up all of the memory on my workstation, the docker container would crash without writing a crash dump to the logs.

This issue seems very closely related to https://github.com/RedisGraph/RedisGraph/issues/456 and may actually be the same issue. It appears that the issue was supposedly made irrelevant by https://github.com/RedisGraph/RedisGraph/issues/454 but I am still observing very similar symptoms running on edge from this morning.

[mdecuir]

Some additional information I have found since the initial report.

In addition to memory leakage growing faster with more match clauses in a query, it also seems to grow with the number of queries sent on a given redis connection.

By closing and reopening the redis connection after some number of requests, I can greatly decrease the rate of memory leakage at the cost of slower performance. I don't have full numbers on how things change, but by making only 100 requests over the life of a single connection rather than 50-100k, the amount of memory leakage is reduced from around 10GB to around 200MB.

If it would be useful, I can provide some more numbers as I explore this behavior in more detail.

[jeffreylovitz]

Hi @mdecuir,

Thanks for your report! When I test the query MATCH (n0:node) WHERE n0.name = 'node0' RETURN n0, I see the server's memory consumption rise by ~5MB, like you reported. This is (I believe) due to our use of jemalloc for our primary allocator, which keeps some memory in reserve to reduce the number of system calls for memory. The 5MB number stays fairly stable between 100k executions and 1M+.

More complex query patterns (multiple MATCH patterns or long traversal chains), however, definitely have indications of a memory leak! We're going to dig into this shortly, and hopefully start pushing fixes this week or next.

The per-connection leakage you describe is new to me. Are you using a specific client or interface to issue multiple queries over the same connection? It would be very helpful if you can give some steps to reproduce for this!

[mdecuir]

I am still trying to explore what I am seeing regarding connection based.

I am using the python Redis client.

My testing so far seems to suggest that the main thing I could get benefit from is just explicitly closing the connection I was sending commands on rather than depending on python garbage collection to dispose of the connection. When I explicitly open a connection for any sized block of work and then explicitly close that connection, the amount of memory usage does not grow nearly as fast.

The work being done is a multiple phase process where each phase has its own redis connection. I am not 100% certain that the objects doing the work are being promptly disposed of when a phase is complete. When I use the object's connection for each phase of the work, I see a much larger increase in memory usage than when I create a brand new connection when I start executing a phase and explicitly close that connection when the work ends. The odd bit is that memory usage doesn't go down in the former case when the process completes and all of the objects should have gone away.

[mdecuir]

I need to dig further on the connection cleanup stuff because I am unable to reproduce the same behavior about explicitly cleaning up connections resulting in better memory usage that I observed on Friday. For now, it probably should be considered a red herring.

[jeffreylovitz]

Thanks, @mdecuir! For the moment, I'm going to label this issue and add it to our short-term roadmap (next few weeks), since the memory leaks on more complex patterns is indisputable.

We'll also try to reproduce the connection bloat based on the steps you described - please update here if you find anything further!

[mdecuir]

I have a few additional observations from my exploration of this yesterday.

When took a ~300MB graph and ran a complex query with 3 match clauses against it repeatedly, I saw an increase in memory usage by about 300MB, but that increased memory usage remained stable as I sent the same query more times. This makes me think that the uncapped memory leak is either related to sending many different queries with multiple match clauses or it is related to sending queries with multiple match clauses combined with create statements.

The next test I want to run is cycling through several different queries against the same graph and see if the memory increase is larger than using a single test query and whether or not it is uncapped.

[dillonhows]

I am also facing the same memory leak issue, to store a large amount of real time data, almost query per microsecond like this:
MATCH ()-[t:TO]->() WHERE t.typeA = 'A' AND t.typeB = 'B' SET t.t1 = 1 SET t.t2 = 2 SET t.t3 = 3 SET t.t4 = 4 SET t.t5 = 5 SET t.t6 = 6 SET t.t7 = 7 SET t.t8 = 8
I saw an increase in memory usage by about 20MB+ per minute. The query is getting slower by using redisgraph:edge container.

[jeffreylovitz]

@mdecuir @dillonhows We've merged a series of a leak fixes to the master branch and the redisgraph:edge image today! I believe that all the queries we've discussed here should run leak-free now.

Please let us know if you notice any further leaks!

[schwab]

Thanks so much for the effort @jeffreylovitz! You guys are really nailing the bug fixes this year. It helps keep us on track knowing the product is continually improving. We've recently complete work to to expand our use of redis graph to replace a Mariadb and CassandraDb backend. Knowing you guys are all over the bugs so quickly has helped us sell it to leadership as a good path forward.

[dillonhows]

@jeffreylovitz We are still seeing the memory leak in redisgraph:edge. Init new RedisGraph database, no data, using the redisgraph.js javascript client like this:
setInterval(() => { graph.query("MATCH ()-[t:TO]->() WHERE t.typeA = 'A' AND t.typeB = 'B' SET t.t1 = 1 SET t.t2 = 2 SET t.t3 = 3 SET t.t4 = 4 SET t.t5 = 5 SET t.t6 = 6 SET t.t7 = 7 SET t.t8 = 8").then(res => { console.log(res.hasNext()); }).catch(err => { console.error(err); }); }, 1);
You can test the same code in your database.

[emin156]

Are here any new insides ? We experience similar things so far.

[jeffreylovitz]

@emin156 This is a fairly old thread; all the leaks reported here have been resolved. If you're encountering leaks now, please open a new issue with details and, if possible, steps to reproduce!