Elasticsearch-js: Unable to revive connection

Created on 9 Sep 2014  Â·  62Comments  Â·  Source: elastic/elasticsearch-js

Hello,

I keep getting this error, and it never recovers. I'm using Elasticsearch 1.3.2 and elasticsearch-js 2.4.0 (node client)

Elasticsearch WARNING: 2014-09-09T08:09:05Z
  Unable to revive connection: https://elasticsearch.****.net/

Elasticsearch WARNING: 2014-09-09T08:09:05Z
  No living connections

Here is my configuration

{
    apiVersion:'1.3',
    sniffOnConnectionFault:true,
    maxRetries:5,
    keepAlive:true,
    connectionClass:'http'
}
stale

Most helpful comment

My problem has been solved.

code

let client = new elasticsearch.Client({
    host: 'hostname.domain:9200',
    maxSockets: 20,
    keepAlive: false,  
    requestTimeout: 300000,
    sniffInterval: false
});
try {
    await client.search({
        index: 'index',
        body: {"query": {"match_all": {}}},
    });
} catch (error) {

} finally {
    client.close();
}

version

node: v8.12.0

elasticsearch.js: 15.1.1

elasticsearch: 6.4.2

All 62 comments

I am also getting this same error. We have tried setting keepAlive to false and the app actually fails faster. I can't find anything in the other issues that fixes this problem.

elasticsearch: 1.5.14
forever-agent: 0.5.2
lodash-node: 2.4.1

node: 0.10.26

I have seen this timeout error just before all of the no living connections errors, but not all the time.

Elasticsearch ERROR: 2014-09-10T19:52:11Z
  Error: Request error, retrying -- connect ETIMEDOUT
      at Log.error (/Users/.../node_modules/elasticsearch/src/lib/log.js:213:60)
      at checkRespForFailure (/Users/.../elasticsearch/src/lib/transport.js:185:18)
      at HttpConnector.<anonymous> (/Users/.../node_modules/elasticsearch/src/lib/connectors/http.js:150:7)
      at ClientRequest.bound (/Users/.../node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)
      at ClientRequest.EventEmitter.emit (events.js:95:17)
      at Socket.socketErrorListener (http.js:1547:9)
      at Socket.EventEmitter.emit (events.js:95:17)
      at net.js:440:14
      at process._tickCallback (node.js:415:13)

@DavidTanner Are you using a similar configuration?

The only configuration I have set is keepAlive, host, and port.

here is a bit more information about the error :

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  No living connections

Trace: { [Error: No Living connections] message: 'No Living connections' }
    at /[...]/transfertES.js:60:22
    at respond (/[...]/node_modules/elasticsearch/src/lib/transport.js:252:9)
    at sendReqWithConnection (/[...]/node_modules/elasticsearch/src/lib/transport.js:172:7)
    at next (/[...]/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
    at process._tickCallback (node.js:419:13)

the issue comes from the server.

I've increased the deadTimeout option, and the maxRetries and I haven't seen the error since

I can understand that something happens on the server side to cause the initial error, but is there nothing the library dies to try to reconnect?

With your declaration of it being a server issue, is there any hint you can give us as to what is causing the issue?

we have a river plugin on ES (RabbitMQ), and every time I got the Unable to revive connection error, the river timed out too.

Increasing deadTimeout and maxRetries 'solved' it. I have to investigate if it comes from Elasticsearch or from our physical server itself

I was wrong, I just ran into the issue again...

@DavidTanner do you use a frontend with your ES cluster such as Nginx or HAProxy, or do you directly connect to your node(s) ?

Our instance of ES is not behind any proxies.

I ran into a similar issue with the python library...

The issue seems similar to this one : http://elasticsearch-users.115913.n3.nabble.com/Increasing-CLOSE-WAIT-connections-and-HTTP-current-open-metric-td4019752.html

I was reindexing 1 billion documents from one cluster to another, using scan & scroll API. The destination cluster was overflowed by bulk indexing requests (around 90 request/sec with 1k documents in each), and was unable to respond after a few minutes.

I haven't try to connect to the cluster using the Java API yet, I've decided to use the RabbitMQ river instead, and I've no more problems.

Getting this exact issue on our end as well. What is the fix or work around?

Here's our info:
node -v
v0.10.33

Elasticsearch:
1.4.1

Client library:
2.4.2

Connection:

var client = new ES.Client({
    host: 'localhost:9200',
    log: 'trace'
});

Another interesting side affect. Whenever this error is thrown in nodejs and we restart node, the error is still there. Restarting elasticsearch via esrestart will usually fix the issue. However, in order to get nodejs to communicate with elastic search again, we have to manually open up an HTTP connection to http://server:9200. Very strange.

@bugs181 Does it occur when you are performing a lot of requests on the cluster too, or does it occur randomly ? And are you using a proxy in front of ES ?

@juliendangers Technically, yes. We hit this barrier whenever we hit the server just a few times. However, it's been noted to be random. One or two requests is sometimes enough.

Behind the scenes, I'm not sure how it works, on our nodejs server we only call Client() once in our module. We are attempting to let the elasticsearch-js module handle the connections for us but it doesn't seem to be working out that way.

I suppose the nodejs COULD be considered the proxy. We are not using any sort of other proxy.
Client JS -> elasticsearch-js (nodejs) -> Elasticsearch

The connection code: (At the top of the module)

var client = new ES.Client({
    host: 'localhost:9200',
    log: 'trace',
    keepAlive: false
});

Multiple queries, inside various functions:

function readEntity(filter, callback) {
        client.getSource({
            index: 'entities',
            type: filter.type,
            id: filter.id,
            //_sourceInclude: entityAllowedFields(filter.fields) // What fields to include.
        }, function(error, response) {
            if (callback)  callback(error, response);
        });
}

Hopefully this helps you help us. At this point, ES is unusable to us.

P.S. We have tried all kinds of various ES.Client() parameters/flags and nothing seems to fix this issue.

I also get this issue and tracked it down:
On connection loss, the client performs HEAD requests using the internal ping() function. The default timeout for this request is set to 100 milliseconds. One can override the timeout when calling ping manually, but the retry logic just goes with the defaults and there is no way to set a different timeout.

In my case this is really problematic since I'm sending reports from Asia to Europe. The ping time is always > 100, so connections never get revived.

Please consider increasing this very aggressiv default and also allow users to configure the timeout. Personally, I would have expected the timeout to be the same as the requestTimeout.

@mantoni Wow, that does seem to be the case. It was never my intention to prevent setting the default ping timeout. I'll get a fix released soon. Track #185 for updates

@mantoni as a temporary workaround you can do the following. This is not ideal, as it modifies the default for all instances of the client but will do the trick.

Client.apis[config.apiVersion].ping.spec.requestTimeout = customDefaultMs;

Yes, thank you @mantoni!

Ok, the workaround gets me one step further, but I still have connection issues. The elasticsearch instance is not available for a few minutes due to a reboot, and reconnects don't work after that. What I see in the logs is this:

Error: Request Timeout after 30000ms
Error: Request Timeout after 30000ms
Error: Request Timeout after 30000ms
Error received Request error, retrying -- socket hang up
Unable to revive connection: http://.....:9210/
No living connections
Unable to revive connection: http://.....:9210/
No living connections
Unable to revive connection: http://.....:9210/
No living connections

And then it continues to log the last two lines on each use of the bulk upload function, although the server is up and fine again.

@mantoni I hope this isn't a bug in the client, but to be sure you could shut-down and rebuild the client when you have a connection failure and you have reason to believe the server is back up – Just call .close() on the old client and recreate it.

If that fixes it then it is definitely a bug and I will invest as much time as required to fix it.

Sorry, I just left for holidays, so I can't run this test. However, I simply creat one instance of the client and use the bulk upload every 15 seconds.

You can try and reproduce with this library: https://github.com/mantoni/measured-elasticsearch.js

:+1: for a fix.
Getting same error's here....
indexing 6 million docs..(actually one after the other)
by row 16275 error appears. (first Elasticsearch ERROR: 2015-03-03T17:18:37Z - Error: Request error, retrying -- connect EADDRNOTAVAIL
)

Hi, have almost similar issue. In my case I listen to stream of information which comes from Firebase and was pushing it straight to ES via constructed client:

var client = new elasticsearch.Client({
    host: config.elasticsearch.host,
    log: config.elasticsearch.logLevel
});

But I used create action quite aggressively and for each new document I would call it, so in my case initially there were 815 documents and for each I would end up calling:

ids.forEach(function(itemId,idx){
    client.create({
        index: indexName,
        type: type,
        id: itemId,
        body: itemBody,
        ignore: [409]
    }).then(function(resp){
        // noop
    }, function(rejection){
        console.log(('>>> ' + (new Date()).toUTCString()));
        console.log('' + JSON.stringify(rejection));
        console.log('<<<');
    });
});

While testing on local elasticsearch instance I had no issues, but when I switched to remote ES on found.no and used https with basic auth I did immediatelly get

Error: Request error, retrying -- connect EADDRNOTAVAIL
Followed by
Unable to revive connection: ...

My solution was to switch to bulk indexing and problems went away.

var ops = [];
ids.forEach(function(itemId,idx){
    var operation = {
        create: { _index: indexName, _type: type, _id: itemId }
    };
    var operationDoc = itemBody;
    ops.push(operation);
    ops.push(operationDoc);
});

client.bulk({
    body: ops,
    ignore: [409]
}).then(function(resp){
    // noop
    console.log('bulk upload done, operation count', ops.length);
}, function(rejection){
    console.log(('>>> ' + (new Date()).toUTCString()));
    console.log('' + JSON.stringify(rejection));
    console.log('<<<');
});

Hi ivarprudnikov, do you have idea about if the "Unable to revive connection" a js client problem or ES service problem?
I'm set up Kibana work with ES + Shield, and Kibana server cannot connect with ES because of the connection error. curl and browser works well with the https connection, and openssl verified the certificate with https connection, so it looks server side is pretty good. Kibana is js client, I'm wondering if this is a client side issue.

@RongQiao For me I have added a check for any time this error shows up. I recreate the client and try again. I think that is the expected solution from reading above. So this did end up being a client error.

+1 having the same issue during importing million rows

I have the same issue with node v0.10.33 and https connection to Elasticsearch 1.5.2.

var client = new elasticsearch.Client({
  host: 'https://[user]:[passwd]@localhost:9200',
  log: 'trace',
  connectionClass: customHttpConnector
});

It works with node v.0.12

Doing around 300 groovy script updates/sec and having the error within seconds. Tried setting concurrency, and all proposed solutions in here, but nothing helped. Also still happening after a quick restart of the nodejs script (while the "old" updates were still executing i guess). After a few secs it revived, but shortly after the issue is there again (renewed load).

Maybe it has something to do with ES blocking due to a lot of groovy updates, and the driver believing the connection was lost.

Experienced before that this issue was not there with nodejs 0.12, but after upgrading to v5.3.0 it is.

the same problem here. ES 1.7 (latest), nodejs 0.12.13, 4.4.2, 5.10.1
No living connections. Server restart doesn't help. Nothing discussed here actually help.

I have the same issue.
We established a new ES instance like 3 weeks ago.
It hasn't been used for quite a while. We used it just this last Tuesday but now we keep getting the error in question. We ran a big index several days ago and now we cannot even get new Client() in JS.

Any prediction if this gets fixed soon? Been around for a while.

We are running our app on two different Test Servers.
Now everything is fine on TestServer 1 however TestServer 2 breaks throwing this error.
We are facing this error recently otherwise the app was working fine on both the server.
debug: Error: No Living connections at sendReqWithConnection (/Users/kadhiresan/sites/tw-api/node_modules/elasticsearch/src/lib/transport.js:207:15) at next (/Users/kadhiresan/sites/tw-api/node_modules/elasticsearch/src/lib/connection_pool.js:213:7) at nextTickCallbackWith0Args (node.js:433:9) at process._tickDomainCallback (node.js:403:13) { [Error: No Living connections] message: 'No Living connections' } null vendor detail

We solved this problem for good with the custom http agent "workaround". You need to install the agentkeepalive module, but hey, if it solves the issue... Example code (you don't need to specify all options; it's just a snippet of what we use. We're using Elasticsearch 2.3, node 5.3). See also #196.

var elasticsearch = require('elasticsearch');
var AgentKeepAlive = require('agentkeepalive');
var client = new elasticsearch.Client({
        hosts: ['localhost:9200'],
        maxRetries: 10,
        keepAlive: true,
        maxSockets: 10,
        minSockets: 10,
        createNodeAgent: function (connection, config) {
          return new AgentKeepAlive(connection.makeAgentConfig(config));
        }
      }
);

@breakbild I'm still receiving this error message:

Error: Request complete with error
GET http://...:9200/_nodes/_all/clear => connect EMFILE ...:9200 - Local (undefined:undefined)

Followed by

Elasticsearch WARNING:
 Unable to revive connection: http://...

I've also tried to upgrade to v12 rc4, without success.

Is there any other strategy guys?

correct the server host
not like http://192.168.1.111 。if kibana and elasticsearch in the same box
try http://localhost:9200

How can i fix it ? . Plz help me :(

Elasticsearch INFO: 2017-01-11T03:19:19Z
Adding connection to http://localhost:9200/

Elasticsearch DEBUG: 2017-01-11T03:19:19Z
starting request {
"method": "HEAD",
"requestTimeout": 5000,
"castExists": true,
"path": "/",
"query": {}
}

Elasticsearch DEBUG: 2017-01-11T03:19:19Z
starting request {
"method": "PUT",
"path": "/chesshub",
"query": {}
}

(node:7088) DeprecationWarning: Mongoose: mpromise (mongoose's default promise l
ibrary) is deprecated, plug in your own promise library instead: http://mongoose
js.com/docs/promises.html
Elasticsearch TRACE: 2017-01-11T03:19:20Z
-> HEAD http://localhost:9200/

<- 0

Elasticsearch ERROR: 2017-01-11T03:19:20Z
Error: Request error, retrying
HEAD http://localhost:9200/ => connect ECONNREFUSED 127.0.0.1:9200
at Log.error (C:\Users\MyPC\Desktop\gamehub.io-master\node_modules\elastic
search\src\lib\log.js:225:56)
at checkRespForFailure (C:\Users\MyPC\Desktop\gamehub.io-master\node_modul
es\elasticsearch\src\lib\transport.js:240:18)
at HttpConnector. (C:\Users\MyPC\Desktop\gamehub.io-master\node
_modules\elasticsearch\src\lib\connectors\http.js:162:7)
at ClientRequest.wrapper (C:\Users\MyPC\Desktop\gamehub.io-master\node_mod
ules\elasticsearch\node_modules\lodash\lodash.js:4968:19)
at emitOne (events.js:96:13)
at ClientRequest.emit (events.js:188:7)
at Socket.socketErrorListener (_http_client.js:309:9)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at emitErrorNT (net.js:1281:8)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)

Elasticsearch TRACE: 2017-01-11T03:19:20Z
-> PUT http://localhost:9200/chesshub

<- 0

Elasticsearch ERROR: 2017-01-11T03:19:20Z
Error: Request error, retrying
PUT http://localhost:9200/chesshub => connect ECONNREFUSED 127.0.0.1:9200
at Log.error (C:\Users\MyPC\Desktop\gamehub.io-master\node_modules\elastic
search\src\lib\log.js:225:56)
at checkRespForFailure (C:\Users\MyPC\Desktop\gamehub.io-master\node_modul
es\elasticsearch\src\lib\transport.js:240:18)
at HttpConnector. (C:\Users\MyPC\Desktop\gamehub.io-master\node
_modules\elasticsearch\src\lib\connectors\http.js:162:7)
at ClientRequest.wrapper (C:\Users\MyPC\Desktop\gamehub.io-master\node_mod
ules\elasticsearch\node_modules\lodash\lodash.js:4968:19)
at emitOne (events.js:96:13)
at ClientRequest.emit (events.js:188:7)
at Socket.socketErrorListener (_http_client.js:309:9)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at emitErrorNT (net.js:1281:8)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)

Elasticsearch TRACE: 2017-01-11T03:19:21Z
-> HEAD http://localhost:9200/

<- 0

Elasticsearch WARNING: 2017-01-11T03:19:21Z
Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-01-11T03:19:21Z
No living connections

elasticsearch is down!
Elasticsearch TRACE: 2017-01-11T03:19:21Z
-> HEAD http://localhost:9200/

<- 0

Elasticsearch WARNING: 2017-01-11T03:19:21Z
Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-01-11T03:19:21Z
No living connections

Elasticsearch DEBUG: 2017-01-11T03:19:21Z
starting request {
"method": "POST",
"path": "/chesshub/game/1/_create",
"body": {
"white": "Foo",
"black": "Anonymous",
"content": "1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6 4. O-O Bc5 5. c3 O-O 6. d4 exd4

  1. cxd4 Bb4",
    "result": "1-0"
    },
    "query": {}
    }

Elasticsearch TRACE: 2017-01-11T03:19:22Z
-> HEAD http://localhost:9200/

<- 0

Elasticsearch WARNING: 2017-01-11T03:19:22Z
Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-01-11T03:19:22Z
No living connections

Elasticsearch DEBUG: 2017-01-11T03:19:23Z
starting request {
"method": "POST",
"path": "/chesshub/game/2/_create",
"body": {
"white": "Anonymous",
"black": "Bar",
"content": "1. e4 c6 2. e5 d5 3. exd6 exd6 4. Nf3 Bg4 5. d4 Nf6 6. Bg5 Be7
",
"result": "1-0"
},
"query": {}
}

Elasticsearch TRACE: 2017-01-11T03:19:24Z
-> HEAD http://localhost:9200/

<- 0

Elasticsearch WARNING: 2017-01-11T03:19:24Z
Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2017-01-11T03:19:24Z
No living connections

@filipedeschamps

Were you able to solve the issue? I also got the same error where it mentions

Elasticsearch ERROR: 2017-05-05T19:05:37Z Error: Request error, retrying GET http://<HOST>:9200/<index>/<type>/<doc_ID> => connect EMFILE <IP>:9200 - Local (undefined:undefined)

Followed by

Elasticsearch WARNING: 2017-05-05T19:05:37Z Unable to revive connection: http://HOST:9200/

which is followed by

Elasticsearch WARNING: 2017-05-05T19:05:37Z No living connections

Since it mentions EMFILE which means that process is trying to open too many files/sockets and OS is denying that, so I suspect that to solve we might need to implement some sort of connection pooling.

However as per @spalger 's comment on #90 i.e.

elasticsearch.js has a pretty good connection pool built in

So I am not sure if that would solve the issue. For now I have tried to close the connection after I am done using it OR before creating more es-client instances(using es-js library).

Thanks,

@luthraG thanks for calling me back to this issue.

The answer is yes and it was a bug in our code that used all file descriptors of the OS. The script kept opening tmp files without closing their buffers, until OS ran out of file descriptors.

Check this stats in your OS to see if it's related.

Best regards.

Thanks @filipedeschamps for prompt reply. Let me monitor the file handles and see if error is caused by it.

Issues with Connecting ElasticSearch Server from Node Server:-

I have hosted elastic search on aws ec2 t2 micro.
Node and mongodb hosted on separately two aws ec2 t2 small instance.
The problem is node is able to connect to mongo and elastic search instance but Its not able to fetch data from elasticsearch.
Using sense chorme browser plugin I am able to see my indices on ec2 t2 micro.But, when i search from node i get below given error message:

I am getting “error: error in searching patientsError: No Living connections”.

ElasticSearch version: 2.4.4
Node version: 4.4.5

Update: My problem was resolved thanks to @cyrilcyril70 on this thread.

I have the same problem
I am using a docker compose (YAML here)

#docker-compose up
...
elasticsearch    | [2017-07-22T17:08:31,022][INFO ][o.e.t.TransportService   ] [kj5tP_v] publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch    | [2017-07-22T17:08:31,174][INFO ][o.e.b.BootstrapChecks    ] [kj5tP_v] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch    | ERROR: [1] bootstrap checks failed
elasticsearch    | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
elasticsearch    | [2017-07-22T17:08:31,310][INFO ][o.e.n.Node               ] [kj5tP_v] stopping ...
elasticsearch    | [2017-07-22T17:08:31,683][INFO ][o.e.n.Node               ] [kj5tP_v] stopped
elasticsearch    | [2017-07-22T17:08:31,685][INFO ][o.e.n.Node               ] [kj5tP_v] closing ...
elasticsearch    | [2017-07-22T17:08:31,887][INFO ][o.e.n.Node               ] [kj5tP_v] closed
elasticsearch    | [2017-07-22T17:08:31,896][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
elasticsearch exited with code 78
kibana           | {"type":"log","@timestamp":"2017-07-22T17:08:23Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}
kibana           | {"type":"log","@timestamp":"2017-07-22T17:12:34Z","tags":["info","optimize"],"pid":1,"message":"Optimization of bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page complete in 250.91 seconds"}
kibana           | {"type":"log","@timestamp":"2017-07-22T17:12:34Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana           | {"type":"log","@timestamp":"2017-07-22T17:12:35Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana           | {"type":"log","@timestamp":"2017-07-22T17:12:35Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana           | {"type":"log","@timestamp":"2017-07-22T17:12:35Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
...
(continues)

```

systeminfo | findstr /B /C:"OS Name" /C:"OS Version"

OS Name: Microsoft Windows 10 Home
OS Version: 10.0.15063 N/A Build 15063

docker-machine version

docker-machine.exe version 0.12.2, build 9371605

docker version

Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:30:30 2017
OS/Arch: windows/amd64

Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:51:55 2017
OS/Arch: linux/amd64
Experimental: false```

For dev purposes only, mine was resolved with http.cors.enabled and http.cors.allow-origin

I was facing the same problem. For me issue was with setting auth configuration. Following configuration is worked for me.

var client = new elasticsearch.Client( {  
  host: [
    {
      host: '127.0.0.1',
      auth: 'username:password',
      protocol: 'http',
      port: 9200
    }
  ]
});

UPDATED

For me the error occurs when using the Scroll API with a query size of 1. My goal was to create an rxjs stream of the elastic query. Code is the following

// elasticClient initialized only with { host: 'hostname.domain:9200' } 
import elasticClient from '../elasticHelper';
import * as Rx from 'rxjs';
import * as _ from 'lodash';

function scrollElastic(scroll, scrollId) {
    return Rx.Observable
        .fromPromise(elasticClient.scroll({scroll, scrollId}))
        .map(({_scroll_id, hits: {hits}}) =>
            ({scrollId: _scroll_id, data: hits.map(x => x._source)[0]}));
}

function searchElastic({query, sort}) {
    const body = {
        size: 1,
        query,
        _source: { excludes: ['editable'] },
        sort
    };
    // keep the search results "scrollable" for 30 secs
    const scroll = '30s';

    return Rx.Observable
        .fromPromise(elasticClient.search({ index: 'data', body, scroll }))
        .map(({_scroll_id, hits: {hits}}) =>
            ({scrollId: _scroll_id, data: hits.map(x => x._source)[0]}))
        // invokes scrollElastic recursively
        .expand(({scrollId}) => scrollElastic(scroll, scrollId))
        // stop resursion when there are no more data
        .takeWhile(res => res.data != null)
        .map(x => x.data);
}

function updateAncestors(ancestors, e, format) {
    if(format === 'csv' || format === 'json') {
        const idx = ancestors.indexOf(e.id);
        if(idx > -1) {
            ancestors[idx] = _.pick(e, ['type', 'name']);
        }
    }
}

function getEntryQueryStream(entriesQuery, query) {
    const {parentSearchFilter, filter, format} = query;

    return searchElastic(entriesQuery)
        .concatMap(entry => {
            const ancestors = entry.ancestors || [];

            // if no parents => doesn't match
            if(!ancestors.length) {
                return Rx.Observable.empty();
            }

            const parentsQuery = getElasticQuery(parentSearchFilter, filter);
            parentsQuery.query.filtered.filter.bool.must.push({
                terms: {
                    id: ancestors
                }
            });

            // fetch parent entries
            return searchElastic(parentsQuery)
                .do(e => updateAncestors(ancestors, e, format))
                .count()
                .concatMap(count => {
                    // no parents match query
                    if(!count) {
                        return Rx.Observable.empty();
                    }

                    // fetch all other ancestors that weren't part of the query results
                    // and are still a string (id)
                    const restAncestorsToFetch = ancestors.filter(x => _.isString(x));
                    return fetchAncestors(restAncestorsToFetch, ancestors, format)
                        .concatMap(() => Rx.Observable.just(entry));
                });
        });
}

function executeQuery(query, res) {
    try {
        const entriesQuery = {
            query: {
                filtered: {
                    filter: {
                        bool: {
                            must: [{
                                range: {
                                    creationTimestamp: {
                                        // some dates passed here
                                        gte: startdate.toISOString(),
                                        lte: enddate.toISOString()
                                    }
                                }
                            }, {
                                query: {
                                    query_string: {
                                        query: "+type:*bla* +name:*blabla*"
                                    }
                                }
                            }]
                        }
                    }
                }
            },
            sort: [{creationTimestamp: {order: 'asc'}, id: {order: 'asc'}}]
        };

        getEntryQueryStream(entriesQuery, query)
            .concatMap(e => // do non elastic stuff)
            .map(e => // do stuff)
            .subscribe(
                x => res.write(x);
                err => {
                    console.error(err);
                    res.status(500).json(err);
                },
                () => res.end()
            );

        HandlerFactory.buildHandler(query, enrichedEntries, res);
    } catch(e) {
        logger.error(e);
        res.status(500).json(e);
    }
}

and the error I get is:

Elasticsearch ERROR: 2018-06-20T12:01:13Z
  Error: Request error, retrying -- connect EADDRNOTAVAIL X.X.X.X:9200 - Local (X.X.X.X:0)
      at Log.error (.../node_modules/elasticsearch/src/lib/log.js:218:56)
      at checkRespForFailure (.../node_modules/elasticsearch/src/lib/transport.js:211:18)
      at HttpConnector.<anonymous> (.../node_modules/elasticsearch/src/lib/connectors/http.js:153:7)
      at ClientRequest.wrapper (.../node_modules/lodash/index.js:3095:19)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at Socket.socketErrorListener (_http_client.js:309:9)
      at emitOne (events.js:96:13)
      at Socket.emit (events.js:188:7)
      at emitErrorNT (net.js:1277:8)
      at _combinedTickCallback (internal/process/next_tick.js:80:11)
      at process._tickDomainCallback (internal/process/next_tick.js:128:9)

Elasticsearch WARNING: 2018-06-20T12:01:13Z
  Unable to revive connection: http://.....:9200/

Elasticsearch WARNING: 2018-06-20T12:01:13Z
  No living connections

ES v1.7 and driver v8

env: OpenSUSE 12.3

node: v6.11.1

Any ideas?

Hi XeniaSiskaki ,
Have you checked with

  1. normal js connection i.e. without rx
  2. by adding proper auth for rx

@deepaksomase I'll try number 1. and let you know! What do you mean though by

proper auth for rx?

@deepaksomase It does seem to work without Rxjs. What could that mean?

@XeniaSiskaki , not sure. Is it showing some data and then throwing an error.

I have tried your above-provided code (with my elasticsearch connection initialization helper class), at my end and observed it working fine. I have verified above code with the version of node=v8.9.4 and ES version 6.2.4

Is it showing some data and then throwing an error.

Exactly! It processes correctly some data and then at some point the driver just reports "No live connections". I'm not sure how to proceed...

How many records (count) do you have in your index for which you are doing search/ scroll?

status3

Seems to be an issue with the opening of a number of socket connections simultaneously. tried

  1. update log setting (set it to trace), while configuring elasticsearch. And check the logs, it may give more info about an error
  2. As per the elasticsearch client version modify the configuration for timeout, hostnames ( modify this as IP address ), requestTimeout, deadTimeout, maxRetries and check that still issue exist
  3. try using agentkeepalive with http
  4. configure/update net.ipv4.tcp_tw_reuse value.

I'm not sure what point 3. means but the other points didn't help at all

@XeniaSiskaki , I have tried to reproduce this with a huge amount of data at my end but not able to reproduce. Can you please provide http/consumer code which consuming searchElastic function ( i have tried consuming this on the console directly)

@deepaksomase You are right, there are more elastic stuff happening in the rest of the code. I updated my original code with all elastic calls.

@deepaksomase It also seems to work fine if I delay the scrollElastic calls by a few milliseconds, but this way feels very hackish. I would prefer if I could somehow fix this in the elastic driver

EDIT
I doesn't work with a delay if multiple users request the same API endpoint.

Any news/ideas on how to fix this?

Following the Quick Start on elastic.co I came onto the error Invalid protocol "file", expected one of http, https that I solved as @deepaksomase described in this comment

My problem has been solved.

code

let client = new elasticsearch.Client({
    host: 'hostname.domain:9200',
    maxSockets: 20,
    keepAlive: false,  
    requestTimeout: 300000,
    sniffInterval: false
});
try {
    await client.search({
        index: 'index',
        body: {"query": {"match_all": {}}},
    });
} catch (error) {

} finally {
    client.close();
}

version

node: v8.12.0

elasticsearch.js: 15.1.1

elasticsearch: 6.4.2

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy

We just encountered this problem and the reason was due to Elastic Search running on Node 8 which has a bug with SSL encryption

https://github.com/nodejs/node/issues/19359

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy

Was this page helpful?
0 / 5 - 0 ratings