Elasticsearch-js: Sniff test missing the 'http' property required by the connectionPool.nodesToHost.

[TheChrisOneil]

Sniff test returns results.body.nodes missing the 'http' property required by the connectionPool.nodesToHost.

results.body.nodes is missing the property http
{
'fQOZTtH-T02SgbQWbiEL7w': {
name: '600909800c2ae15a84cbdbef4598a58a',
version: '7.4.2',
build_flavor: 'oss',
build_type: 'tar',
build_hash: 'unknown',
roles: [ 'ingest', 'master', 'data' ]
}
}
To Reproduce
Define Client configuration
const elasticSearchConnection = new Client({
node: 'https://vpc-es-institutional-investor-r5ck3aevprngvntulspwqae5ha.us-west-2.es.amazonaws.com',
maxRetries: 5,
requestTimeout: 60000,
sniffOnStart: true
});

Deploy to AWS running nodejs v Node.js version: 12.16.1
and using "@elastic/elasticsearch": "^7.6.1",

npm start
During the initialization the following error is observed.

Paste your code here:
Below is my debug code identifying the initialization the phase that results in the sniff test
/var/log/nodejs/nodejs.log
==== [ConnectionPool.js] update ====
==== [BaseConnection.js] Updating the connection pool ====
==== [Transport.js] sniff ====
==== [ConnectionPool.js] getConnection ====
==== [ConnectionPool.js] resurrect ====
==== [ConnectionPool.js] resurrect: Nothing to resurrect ====
====[index.js] New Client transport established ====
====[index.js] New Client api established ====

handlerId: Index_bulkActions handlerCache[handlerId]: [Function: bulkActions] handlerId:
==== [ConnectionPool.js] markAlive - Updating the connection pool ====
==== [Transport.js] Sniffing ended successfully ====
sniff nodes results
{
'fQOZTtH-T02SgbQWbiEL7w': {
name: '600909800c2ae15a84cbdbef4598a58a',
version: '7.4.2',
build_flavor: 'oss',
build_type: 'tar',
build_hash: 'unknown',
roles: [ 'ingest', 'master', 'data' ]
}
}
==== [BaseConnection.js] nodesToHost ====
node[name]=
node[name]=600909800c2ae15a84cbdbef4598a58a
node[version]=
node[version]=7.4.2
node[build_flavor]=
node[build_flavor]=oss
node[build_type]=
node[build_type]=tar
node[build_hash]=
node[build_hash]=unknown
node[roles]=
[ 'ingest', 'master', 'data' ]
/var/app/current/middleware/utility/@elastic/elasticsearch/lib/pool/BaseConnectionPool.js:209
var address = node.http.publish_address
^

TypeError: Cannot read property 'publish_address' of undefined
at ConnectionPool.nodesToHost (/var/app/current/middleware/utility/@elastic/elasticsearch/lib/pool/BaseConnectionPool.js:209:31)
at /var/app/current/middleware/utility/@elastic/elasticsearch/lib/Transport.js:370:41
at f (/var/app/current/node_modules/once/once.js:25:25)
at IncomingMessage. (/var/app/current/middleware/utility/@elastic/elasticsearch/lib/Transport.js:305:11)
at IncomingMessage.emit (events.js:323:22)
at endReadableNT (_stream_readable.js:1204:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)## Expected behavior

A clear and concise description of what you expected to happen.
The hostToNodes should handle the lack of a'http' property ' and perhaps default to the users entered configure node url.

Your Environment
AWS running nodejs v Node.js version: 12.16.1
and using "@elastic/elasticsearch": "^7.6.1",

[TheChrisOneil]

Some updates:
The elasticsearch version is 7.4 running in a VPN

Here is the complete response from the endpoint _nodes/_all/http.

{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "955902261784:es-institutional-investor",
"nodes": {
"fQOZTtH-T02SgbQWbiEL7w": {
"name": "600909800c2ae15a84cbdbef4598a58a",
"version": "7.4.2",
"build_flavor": "oss",
"build_type": "tar",
"build_hash": "unknown",
"roles": [
"ingest",
"master",
"data"
]
}
}
}

[TheChrisOneil]

Here is the complete response from the _nodes/_all/http endpoint

{
body: {
_nodes: { total: 1, successful: 1, failed: 0 },
cluster_name: '955902261784:es-institutional-investor',
nodes: { 'fQOZTtH-T02SgbQWbiEL7w': [Object] }
},
statusCode: 200,
headers: {
date: 'Wed, 29 Apr 2020 18:31:15 GMT',
'content-type': 'application/json; charset=UTF-8',
'content-length': '299',
connection: 'keep-alive',
'access-control-allow-origin': '*'
},
warnings: null,
meta: {
context: null,
request: { params: [Object], options: [Object], id: 1 },
name: 'elasticsearch-js',
connection: {
url: 'http://vpc-es-institutional-investor-r5ck3aevprngvntulspwqae5ha.us-west-2.es.amazonaws.com/',
id: 'http://vpc-es-institutional-investor-r5ck3aevprngvntulspwqae5ha.us-west-2.es.amazonaws.com/',
headers: {},
deadCount: 0,
resurrectTimeout: 0,
_openRequests: 0,
status: 'alive',
roles: [Object]
},
attempts: 0,
aborted: false
}
}

Its seems that fix could be:

IS
const hosts = this.connectionPool.nodesToHost(result.body.nodes, protocol)

Could Be:
const hosts = this.connectionPool.nodesToHost(result, protocol)

Then in
nodesToHost (results, protocol) {
var nodes = results.body.nodes;
const ids = Object.keys(nodes)
const hosts = []
.
.
.
IS:
address = node.http.publish_address

Could Be:
if (node.hasOwnProperty('http')){
address = node.http.publish_address
else {
address = result.meta.connection.url
}

[delvedor]

Hello! Unfortunately, I wasn't able to find documentation about this, but very likely aws puts a load balancer between you and the Elasticsearch cluster. This means that the nodes live in a different network, making the sniffing result useless as the addresses would not be reachable.
I would recommend not to use sniffing at all in such case.

I'll write a fix in the client to throw a more meaningful error, thanks!

[TheChrisOneil]

Thanks for responding. You have labeled my issue as a question, but I believe after my explanation I adjure you to consider it a bug.

The sniff test assumes the elasticsearch endpoint _nodes/_all/http provides 11 properties for each node. The AWS installation of elasticsearch only provides 6 properties; Furthermore, the 'http' property is omitted. This omission causes an _exception in @elastics/elasticsearch code and consequently the server fails to start_.

Typical Elasticsearch Installation
Observe the typical 11 properties provided per node in a typical elasticsearch installation

1. "name": "Chriss-MacBook-Pro-2.local",
2. "transport_address": "127.0.0.1:9300",
3. "host": "127.0.0.1",
4. "ip": "127.0.0.1",
5. "version": "7.6.1",
6. "build_flavor": "default",
7. "build_type": "tar",
8. "build_hash": "aa751e09be0a5072e8570670309b1f12348f023b",
9. "roles": ["master", "ingest", "data", "ml" ],
10. "attributes": { "ml.machine_memory": "...}
11. "http" : { bound_address, publish_address, max_content_length_in_bytes}

AWS Elasticserach Installation
Now observe the typical 6 properties provided per node on an AWS installation

1. "name": "600909800c2ae15a84cbdbef4598a58a",
2. "version": "7.4.2",
3. "build_flavor": "oss",
4. "build_type": "tar",
5. "build_hash": "unknown",
6. "roles": []

Missing are the properties: host, ip, transport_address, attributes, and http.

[rveitch]

Verified, I am experiencing this issue as well with AWS ES.

Update
I worked around this issue by setting sniffOnStart to false.