Elasticsearch-js: Bulk indexing errrors

[coldlamper]

We are doing bulk indexing about every 5 minutes. Previously using elasticsearch-js 8.2.0 and Elasticsearch 2.4 everything worked fine.
The output log we create looks like similar to this

2017-02-24 11:15:16: Index data ...
2017-02-24 11:15:16:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:21:     indexed 7500 2017-02-24 11:15:21: documents: 5273.976ms
2017-02-24 11:15:21:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:26:     indexed 7500 2017-02-24 11:15:26: documents: 4030.945ms
2017-02-24 11:15:26:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:29:     indexed 7500 2017-02-24 11:15:29: documents: 3200.260ms
2017-02-24 11:15:29:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:32:     indexed 7500 2017-02-24 11:15:32: documents: 2794.364ms
2017-02-24 11:15:32:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:34:     indexed 7500 2017-02-24 11:15:34: documents: 2760.464ms
2017-02-24 11:15:34:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:37:     indexed 7500 2017-02-24 11:15:37: documents: 2216.049ms
2017-02-24 11:15:37:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:39:     indexed 7500 2017-02-24 11:15:39: documents: 2716.405ms
2017-02-24 11:15:39:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:42:     indexed 7500 2017-02-24 11:15:42: documents: 2426.167ms
2017-02-24 11:15:42:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:44:     indexed 7500 2017-02-24 11:15:44: documents: 2372.236ms
2017-02-24 11:15:44:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:47:     indexed 7500 2017-02-24 11:15:47: documents: 2548.165ms
2017-02-24 11:15:47:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:49:     indexed 7500 2017-02-24 11:15:49: documents: 2566.625ms
2017-02-24 11:15:49:   bulk request of 714 on promokits_1487952787856:promokit...
2017-02-24 11:15:49:     indexed 714 2017-02-24 11:15:49: documents: 323.010ms

After upgrading to elasticsearch-js 12.1.1 and Elasticsearch 5.2.0 we get random errors a few times a day.

2017-02-24 06:54:44: Index data ...
2017-02-24 06:54:44:   bulk request of 7500 on promokits_1487937219324:promokit...
2017-02-24 06:54:45: Elasticsearch ERROR: 2017-02-24T11:54:45Z
Error: Request error, retrying
  POST https://host1:9280/_bulk  => socket hang up
      at Log.error (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/transport.js:240:18)
      at HttpConnector.<anonymous> (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/connectors/http.js:162:7)
      at ClientRequest.wrapper (/etc/gigsalad_crons/es-indexer/node_modules/lodash/lodash.js:4968:19)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at TLSSocket.socketCloseListener (_http_client.js:284:9)
      at emitOne (events.js:101:20)
      at TLSSocket.emit (events.js:188:7)
      at TCP._handle.close [as _onclose] (net.js:492:12) 
2017-02-24 06:54:48:     indexed 7500 2017-02-24 06:54:48: documents: 4253.678ms
2017-02-24 06:54:48:   bulk request of 7500 on promokits_1487937219324:promokit...
...

or sometimes the error will be

Error: Request error, retrying
  POST https://host1:9280/_bulk  => write EPIPE
      at Log.error (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/transport.js:240:18)
      at HttpConnector.<anonymous> (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/connectors/http.js:162:7)
      at ClientRequest.wrapper (/etc/gigsalad_crons/es-indexer/node_modules/lodash/lodash.js:4968:19)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at TLSSocket.socketErrorListener (_http_client.js:308:9)
      at emitOne (events.js:96:13)
      at TLSSocket.emit (events.js:188:7)
      at onwriteError (_stream_writable.js:319:10)
      at onwrite (_stream_writable.js:337:5)
      at WritableState.onwrite (_stream_writable.js:89:5)
      at fireErrorCallbacks (net.js:462:13)
      at TLSSocket.Socket._destroy (net.js:503:3)
      at WriteWrap.afterWrite (net.js:795:10)

These errors always occur on the first bulk index request and the retry is always successful so everything does get indexed fine.

I would like to know why this error is occurring and how to prevent it.

[spalger]

Hmm, are you sure that elasticsearch is being deployed exactly the same way as before? Did you upgrade node.js versions? Perhaps you could try npm install elasticsearch@beta?

[coldlamper]

Using node 6.3.1, same as before. I'm connecting through an Nginx proxy if that makes a difference, but was during that before also.

[spalger]

Can you give the beta a shot?

[spalger]

It uses a persistent socket library that other members of the community have had better luck with

[calvinwiebe]

@coldlamper Did you ever find a resolution to this? I'm having the same issue with 13.x. On the same test that throws socket hang up with 13.x, it succeeds without a problem with 8.2.0.

[calvinwiebe]

@spalger any guidance on this? Thanks!

[hathija]

I also have the similar issue as @calvinwiebe.
We are migrating Elastic 2.4 to 5.4 and the module didn't work with 13.X version.

Thank you for your help

[ynux]

similar issue with hosted elasticsearch 5.6.0 in elastic cloud, pushing data with nodejs v8.2.1 and elasticsearch client 13.0.0-rc2, also in combination with socket hang up

[executives]

@spalger The beta version via npm install elasticsearch@beta fixed this issue for me. Thanks!

[baronworks]

similar issues elasticsearch 5.6.4, elasticsearch-js 14.0.0 (as well as 13.3.1) and nodejs v8.9.1
everything works fine with 13.0.0-beta2
client config: {
apiVersion: '5.x',
hosts: ['localhost:9200'],
log: ['error', 'warning'],
sniffOnStart: false
}
Working on upgrading from elasticsearch 2.4.3 with elasticsearch-js 11.0.1 and v6.11.1 where it was also working.

[tam5]

I too was having this problem with elasticsearch 14.0.0.

I solved it by simply chunking my bulk requests into smaller pieces. Truth is, the Elasticsearch docs recommend not trying to bulk too much at once anyway, so we should probably be splitting up the requests anyway, making this a none issue.

[delvedor]

Hi! I'm closing this due to inactivity, if it is still an issue, please feel free to reopen! :)