Elasticsearch-js: socket hang up error on high volume transactions..

[sahas-]

Hello team,

Version: 12.1.1

I commented this on issue #226 too.. wondering if we can get more guidance on this error 'socket hang up'. My usage is not very high volume, about ~500 requests/sec.. my elasticsearch cluster seems to be healthy ( ie. <10% CPU usage, disk i/o 128KB/s etc), however, on the client side, I keep getting socket hang up on both _search & _update calls.
Provided the log in #226 .
Below is one more specifically to _update call

Elasticsearch ERROR: 2017-03-30T01:43:15Z
Error: Request error, retrying
POST http://${myelasticsearch.domain.com}/elasticsearch/${index}/${type}/${doc}/_update => socket hang up
at Log.error (/Users/sahaswaranamam/../node_modules/elasticsearch/src/lib/log.js:225:56)
at checkRespForFailure (/Users/sahaswaranamam/../node_modules/elasticsearch/src/lib/transport.js:240:18)
at HttpConnector. (/Users/sahaswaranamam/../node_modules/elasticsearch/src/lib/connectors/http.js:162:7)
at ClientRequest.wrapper (/Users/sahaswaranamam/../node_modules/lodash/lodash.js:4968:19)
at emitOne (events.js:78:13)
at ClientRequest.emit (events.js:170:7)
at Socket.socketOnEnd (_http_client.js:295:9)
at emitNone (events.js:73:20)
at Socket.emit (events.js:167:7)
at endReadableNT (_stream_readable.js:906:12)

Appreciate any guidance to deal with this.
Thanks in advance.

[spalger]

Can you try out v13.0.0-beta2? See https://github.com/elastic/elasticsearch-js/issues/476

[kareljuricka]

Same problem here. What I am trying to do is reindex large data.

Everytime after 30s I got Error: Request Timeout after 30000ms error.

If I raise up requestTimeout connection parametr, instead of request timeout error I got after 30s:

Error: Request error, retrying POST http://localhost:9200/_reindex => socket hang up

When I run reindex with CURL direct request to elasticsearch (without elasticsearch-js), it works all right.

[godber]

We have been seeing this issue too. We experience conditions that can result in bulk requests taking a very long time to run (45 min), and we see these socket errors on these bulk requests. I can reproduce this scenario in our environment and have been experimenting for the last week or so.

At this point, I have longer bulk requests in a test client working properly if I extend the requestTimeout to a really long time and set keepAlive: false.

...
var client = new elasticsearch.Client({
    host: eshost,
    requestTimeout: 1000 * 60 * 60,
    keepAlive: false,
    log: 'debug',
});
...

Right now I am looking into the source of the trouble being the default for the timeout option on the agentkeepalive package, which according to its docs is:

timeout: {Number} Sets the working socket to timeout after timeout milliseconds of inactivity on the working socket. Default is freeSocketKeepAliveTimeout * 2

I am not sure of this at this point. But I am sure that disabling keepAlive results in a bulk insert that completes correctly without error.

[godber]

I think it's this timeout. My client will work properly without error even if I leave keepAlive enabled and keep my long requestTimeout, but then also modify makeAgentConfig in lib/connectors/http.js so that it sets timeout to something large, somewhat like this:

HttpConnector.prototype.makeAgentConfig = function (config) {
  var agentConfig = {
    keepAlive: config.keepAlive,
    keepAliveMsecs: config.keepAliveInterval,
    maxSockets: config.maxSockets,
    maxFreeSockets: config.keepAliveMaxFreeSockets,
    freeSocketKeepAliveTimeout: config.keepAliveFreeSocketTimeout,
    timeout: 1000 * 60 * 60
  };

  if (this.useSsl) {
    _.merge(agentConfig, this.host.ssl);
  }

  return agentConfig;
};

Perhaps the timeout value used by agentkeepalive should be the same as requestTimeout ... those are conceptually the same thing right?

[spalger]

Interesting point @godber, I'm hesitant to base the timeout in agentkeepalive on the requestTimeout config because that value can technically be set per request...

I wonder what the drawbacks of disabling agentkeepalive's timeout and instead relying on our own timeouts would be...

[godber]

@spalger I don't have a good understanding of the overall architecture of elasticsearch-js, but couldn't you do it the other way around? That is, just rely on the agentkeepalive timeout and handle the timeout error if it throws it, using that to trigger your retries? Are there other reasons you're forced to handle timeouts on your own?

I guess you could add another configuration parameter that sets the agentkeepalive timeout, but it feels like they are serving the same role.

[hathija]

Hello,

i'm getting the same issue as @sahas- while migration ES (2.4 to 5.4) and nodejs pckage (12.0 to 13.0).
How did you resolve? Thanks

[tiste]

Hello,

Same as @hathija, we've upgraded from 2.4 to 5.4, lib as well, and while doing bulk requests (around 100mb), the socket hang up.
Will try to put keepAlive to false.

Thanks for your help.

[helmutkian]

Experiencing the same issue. Anyone have a work around?

[calvinwiebe]

@spalger I thought I'd take a closer look into this. Since there have been enough reports in this issue, and #508, to suggest that this is a real problem. It hasn't gotten much attention, and has been open for 3 months :(.

Regarding the timeouts, as far as I can see, elasticsearch-js has agentkeepalive: ^2.2.0 in its package.json. In that version, the option for the _free socket timeout_ is keepAliveTimeout. However, elasticsearch-js is passing in freeSocketKeepAliveTimeout. That latter option name is what is used in agentkeepalive >= 3.x. Therefore, as it stands, agentkeepalive will be locked to its default timeouts for both the _free_ and _active_ sockets (15s and 30s, respectively). I hacked up my locally installed agentkeepalive to log out the option values to make sure I wasn't jumping to conclusions and being blind, and what I described seems to be the case. (I'm guessing this might have happened because the initial addition of the agentkeepalive happened 8 months ago, but the adding of the options inside elasticsearch-js happened 3 months ago, which was after the option/documentation rename in agentkeepalive).

OPTIONS GIVEN BY elasticsearch
{ keepAlive: true,
  keepAliveMsecs: 1000,
  maxSockets: Infinity,
  maxFreeSockets: 256,
  freeSocketKeepAliveTimeout: 60000 }
OPTIONS DEFAULTED BY agentkeepalive
{ keepAlive: true,
  keepAliveMsecs: 1000,
  maxSockets: Infinity,
  maxFreeSockets: 256,
  freeSocketKeepAliveTimeout: 60000,
  keepAliveTimeout: 15000,
  timeout: 30000 }

It makes sense that keepAlive: false is working for those above, since we bypass the agentkeepalive library completely. We then loose the performance benefits it should provide, though.

If this is fixed, then at least the _active socket_ timeout will be double the freeSocketKeepAliveTimeout, so users of this library can control that somewhat. However, I think it would be a good idea to expose a activeSocketKeepAliveTimeout, as @godber suggested above, to override the agentkeepalive timeout option. Especially since the default of net.Socket is to have _no_ timeout.

If this is indeed the problem, I could work on a PR if your schedule is packed. However, this is a pretty fundamental change and would prefer the owner to take a crack at it :).

EDIT: I also just realized that the original issue posted was for version 12.1.1, which was before the addition of agentkeepalive. So it may be that the issue the rest of us are experiencing with 13.x is unrelated to that of the original issue.

[imyoungyang]

Hi

I encountered the same error message but in the "cluster" mode. I used version 13.2.0 client to connect elastic 5.5.0. But interested thing is that I did not encountered this issue if I connected to single node.

Elasticsearch ERROR: 2017-07-14T11:19:09Z Error: Request error, retrying POST http://localhost:9200/products/goodsListInCate/_bulk => socket hang up at Log.error (/Users/youngbe/Dev/ehs-new-gitlab/iceCream/nodeTools/node_modules/elasticsearch/src/lib/log.js:225:56) at checkRespForFailure (/Users/youngbe/Dev/ehs-new-gitlab/iceCream/nodeTools/node_modules/elasticsearch/src/lib/transport.js:258:18) at HttpConnector.<anonymous> (/Users/youngbe/Dev/ehs-new-gitlab/iceCream/nodeTools/node_modules/elasticsearch/src/lib/connectors/http.js:157:7) at ClientRequest.bound (/Users/youngbe/Dev/ehs-new-gitlab/iceCream/nodeTools/node_modules/lodash/dist/lodash.js:729:21) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at Socket.socketCloseListener (_http_client.js:285:9) at emitOne (events.js:101:20) at Socket.emit (events.js:188:7) at TCP._handle.close [as _onclose] (net.js:501:12)

[goravsingal]

I found that, if I use "http-aws-es" lib to connect via aws auth, it breaks.
If I left my ES instance on aws open to public (just for testing), and leave "http-aws-es" code unused. It works fine.

I ended up using: "aws-es" npm lib.

[rhughes89]

This problem seems to be haunting me as well. No matter what settings I change, it will "seemingly" fixes the problem but it always comes back. I tried even minimizing the size of the bulk operation but nothing seems to be a permanent fix.

It's getting to the point where if it fails to bulk index into elastic search, I catch the error and add each one one by one. That I have never had the problem on indexing a single record.

[mfeltscher]

I encountered the same issue using 13.2.0 and Elasticsearch 5.5.1. From what I can see I think @calvinwiebe's assumption is correct and this should be fixed (as described in #582). For me this issue is 13.x related since I never encountered it using 12.x.

[ynux]

Same issue, with elasticsearch 5.6.0 in elastic cloud, pushing data with nodejs v8.2.1 and elasticsearch client 13.0.0-rc2

This happened a lot (only 33 out of 80 bulk requests persisted) when running from a MacBook over a private Wifi. No issues when running from a linux box with a more stable connection.

[k0t0fey]

Same issue with elasticsearch 5.6.2 and elasticsearch client 13.3.1

[acarstoiu]

This issue was reported 7 months ago and it still wastes people's time. There are in fact two bugs:

• the blocking one, identified by @godber and caused by a narrow-sighted decision in agentkeepalive to set a timeout on sockets, if none was specified; but in the case of elasticsearch-js the requestTimeout parameter must always prevail
• the less severe one, discovered by @calvinwiebe, which is a name confusion for the parameter that sets the time for which free sockets are kept around

All you need to do is this:
```lang=patch
--- src/lib/connectors/http.js 2017-08-09 06:11:25.000000000 +0000
+++ src/lib/connectors/http.js 2017-10-27 09:04:53.756157518 +0000
@@ -93,7 +93,8 @@
keepAliveMsecs: config.keepAliveInterval,
maxSockets: config.maxSockets,
maxFreeSockets: config.keepAliveMaxFreeSockets,
- freeSocketKeepAliveTimeout: config.keepAliveFreeSocketTimeout
+ keepAliveTimeout: config.keepAliveFreeSocketTimeout,
+ timeout: 0
};

if (this.useSsl) {
```
Consider it a pull request.

[executives]

I've been running into the same issue.

This thread mentions using the beta version of elasticsearch.js

npm install elasticsearch@beta

Using the above command, I was able to bulk add 100k records with no errors.

I recommend giving this a try.

[hangang128]

Got the same issue. The patch and @beta didn't work. It turned out that our app is submitting ~200 request at the same time(looping in node.js :( ). I have to stagger the requests with q.delay(variable time) to avoid time out.

[spalger]

Fixed in #600

[Faizanq]

Hi @godber I tried your solution but still am facing timeout issue

[senthilbalajiganesan]

you can set requestTimeout to Infinity while creating connection instance