Elasticsearch-js: Memory Leak

[NassiHarel]

Node: 5.6.0
elasticsearch.js: 12.1.3
elasticsearch: 5.0

var client = new elasticsearch.Client({
  host: 'localhost:9200',
  apiVersion: '5.0',
  sniffOnStart: false,
  connectionClass: 'http'
});

When we are using bulk API, we are facing serious memory leak, the app memory reached up to 5.0G.

client.bulk([params, [callback]]) // Just a bulk of 10 records per second.

The app has only this code, so it's definitely the source.
am I the only one who has experienced it?

[cblanc]

If forcing garbage collection causes your memory consumption to go down, then it's probably related to #310.

However there are workarounds to the RSS leak that I should note here. These are a combination of the following settings (try one then the other/both).

- Set the node param --max-old-space-size to a value suitable for your environment (especially for cloud based instances)
- Set the node param --expose-gc and within your code on a regular schedule manually garbage collect global.gc(); (stick it in a setTimeout function at start up of your app)

If not, it could be a new problem

[NassiHarel]

Yes, I already watched this..
I don't like this solution.
Thanks

[NassiHarel]

There must be a built in solution...

[NassiHarel]

Switching to bluebird...

[bcoe]

I'm seeing similar issues to this on Node 6, with v12 of the library, the queries being run can be found here:

https://github.com/npms-io/queries/blob/master/lib/search.js

As was suggested in other threads, I've downgraded to v3 of the elasticsearch client library. Here's the behavior before and after:

To reproduce simply preform a large # of queries with the query library linked above, the link seems to be immediately apparent.

[spalger]

For the last couple days I have been profiling and working to better understand what is contributing to esjs memory usage. This research has lead me to these conclusions:

• forever-agent (87fe5d4a83359c95182ed1d5bde4c327ff31c5bd), lodash 4 (https://github.com/elastic/elasticsearch-js/pull/501), and the promise shim (317c76f5d6859ec79bd5f9867b70cf656d6d4f04) all seem to be contributing considerably to the amount of memory-thrash observed (especially lodash 4 in 0.x versions of node)
• only loading the necessary api also lowers esjs memory usage (fa4c08748b63efc51e421b066958721deb6d2c18)
• memory usage on every version of esjs seems dramatically improved in node 6

I think I finally have a test stood up that I can let run for a couple days, so I will report back soon with results from that.

Here are the preliminary results:

You can see v3 on node 0.12 is a steady red line in the midst of the noise, but it still sits near the top of the graph around 300mb of RSS (I'm running these containers with a 512MB memory limit). At the bottom of the graph you can see several green lines representing master running on different versions of node, along with v8 and v12 running on node 6.

If you would like to checkout the results in Kibana please ping me on irc.

You can find the code I'm using to run these tests at https://github.com/spalger/esjs-mem-tests. If you would like to suggest better methods of doing this testing, I'd also love your feedback.

[spalger]

I increased the memory limit to my test containers to 3GB and let them run for 12 hours. These are my new results:

Because of the new memory limit the top of the graph as risen, but it's still not quite to 2GB, so the garbage collector is clearly doing it's job and memory is not technically leaking. It's pretty clear though that newer versions of node shouldn't have the same problem and master is consistently doing a good job, so I'm going to push out 13.0 today with the changes from master.

[spalger]

I just published version 13.0.0-beta1 and would appreciate feedback from anyone who is willing to give it a shot

[spalger]

Alright, I just revisited those metrics I was quoting earlier and it looks like the average is pretty misleading. The processes at the top of the chart above were actually running out of memory and dying, so definitely seems like a memory leak. Thankfully, master (which is now 13.0.0-beta1) is not in that group.

[tienvx]

I tested on 13.0.0-beta2, and it works. No more memory leak while I'm using it with rabbitmq to index ~200.000 records from mysql to elasticsearch.