Elasticsearch-js: Cannot read property 'resurrectTimeout' of undefined

[kulakowka]

Greetings!

I found an error in the code:

https://github.com/elastic/elasticsearch-js/blob/1b638d69925d64a7402617265e739dc85b316ca1/lib/pool/ConnectionPool.js#L107

    const connection = this.connections.find(c => c.id === this.dead[0])

    if ((opts.now || Date.now()) < connection.resurrectTimeout) {
      debug('Nothing to resurrect')
      callback(null, null)
      return
    }

If the connection is not found. Then the code will throw an error further: Cannot read property 'resurrectTimeout' of undefined

I just got a lot of such errors in production.

[delvedor]

Hello! Thank you for reporting this!
It looks like the issue you have encountered is hiding another bug.
That code should always have a "valid" dead connection because few lines above, we are checking that the dead array contains something.
So very likely, the dead array is not cleaned up properly.

Which version of the client and Elasticsearch are you using?
Where are you running Elasticsearch?
Can you paste here the options you are using for configuring the client?

Thanks!

[kulakowka]

Client config:

elastic:
  client:
    maxRetries: 6
    requestTimeout: 10000
    sniffOnConnectionFault: true
    sniffOnStart: true
    nodes:
      - http://***.***.**.**:9201
      - http://***.***.**.**:9201
      - http://***.***.**.**:9201
    agent:
      keepAlive: false
      keepAliveMsecs: 1000
      maxSockets: 256
      maxFreeSockets: 256

Client version:

{
  "@elastic/elasticsearch": "^7.6.1"
}

Elastic version:

{
  "name" : "mvs-elastic-01-mvs",
  "cluster_name" : "mvs",
  "cluster_uuid" : "-q1EWefBQa2auCjGwMLnfw",
  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
    "build_date" : "2019-10-28T20:40:44.881551Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

We also observe periodically errors: There are no living connections

Perhaps the reason is this?

[delvedor]

Thanks! How are you running Elasticsearch?
Is there a proxy between the client and Elasticsearch?

If Elasticsearch and the client live on two different networks, then the sniffing might not work, as the IP addresses returned by Elasticsearcfh would be useless for the client. If this is the case, then configuring network.publish_host should help.

We also observe periodically errors: There are no living connections
Perhaps the reason is this?

It should not, that error just means that all the connections registered in the client are marked as dead.

Can you add a listener to the response event and log the errors?
So we can figure out why connections are being marked as dead:

client.on('response', (err, meta) => {
  if (err) {
    // log err.name and err.meta
  }
})

---

Regarding the original issue, I think that those lines cause the bug:
https://github.com/elastic/elasticsearch-js/blob/1b638d69925d64a7402617265e739dc85b316ca1/lib/pool/ConnectionPool.js#L216-L220

Can you extend the connection pool and override that function? So we can confirm if I'm right:

const { Client, ConnectionPool } = require('@elastic/elasticsearch')

class CustomConnectionPool extends ConnectionPool {
  update (connections) {
    super.update(connections)
    this.dead = []
    return this
  }
}

const client = new Client({
  nodes: [...],
  ConnectionPool: CustomConnectionPool
})

[kulakowka]

Thank you very much for your help!

I add CustomConnectionPool to my application. And I collect logs. I'll be back as soon as this happens again.

It should not, that error just means that all the connections registered in the client are marked as dead.

Yes. Restarting the application helps. Errors stop for a while.

[kulakowka]

There were errors again last night: There are no living connections

I could not find this in the application logs. But we record all errors in the APM.
I added a custom ConnectionPool. But it didn't seem to work.

const { Client, ConnectionPool } = require("@elastic/elasticsearch");

class CustomConnectionPool extends ConnectionPool {
  update(connections) {
    super.update(connections);
    this.dead = [];
    return this;
  }
}

const client = new Client({
  ...config.get("elastic.client"),
  ConnectionPool: CustomConnectionPool,
});

client.on("response", (err, meta) => {
  if (err) {
    console.log("ELASTIC RESPONSE ERROR", { err, meta });
    // log err.name and err.meta
  }
});

[delvedor]

The way you are logging errors is not the same as I have suggested, using console.log in that way could hide some information.
Try this:

client.on('response', (err, meta) => {
  if (err) {
    console.log(err.name, JSON.stringify(err.meta, null, 2))
  }
})

I could not find this in the application logs

I'm not sure I understand this comment. The response event is emitted in case of any error happening in the client or in case of any successful response.

Can you answer these questions? :)

How are you running Elasticsearch?
Is there a proxy between the client and Elasticsearch?

[kulakowka]

How are you running Elasticsearch?
Is there a proxy between the client and Elasticsearch?

Elastic in a cluster of 3 nodes without any proxies

[delvedor]

After adding the CustomConnectionPool did you see other Cannot read property 'resurrectTimeout' of undefined errors?
I was able to reproduce the issue with a very unstable cluster, basically, the error happens when running a sniff and updating the connection pool at the same time, leaving the dead list in a dirty state. If I use the CustomConnectionPool the issue disappears.

Regarding the NoLivingConnectionsError, it can happen if the cluster you are using is unstable or under heavy load (502/3/4 errors), if these errors happen in a short period of time, the ConnectionPool will mark them as dead, and when the very last connection is marked as dead, then you start getting the NoLivingConnectionsError, leaving you with no other solution than restarting the application.
I'll work on a fix for both issues, regarding the NoLivingConnectionsError it will only mitigate your problem, I fear that there might be some bad configuration or other issues with your cluster that are showing up during time.

[delvedor]

Hello! I've opened https://github.com/elastic/elasticsearch-js/pull/1127 with a few changes that will solve the Cannot read property 'resurrectTimeout' of undefined error.
Furthermore, I did some change in the connection pool that will help in case of high-frequency failures like it seems you are getting.

[delvedor]

I've released this fix in v7.7.0-rc.1, you can install it with

 npm install @elastic/elasticsearch@next