Elasticsearch: DNS round robin assisted discovery

[jimmyjones2]

If all nodes in a cluster are added to a round robin DNS record and the following changes are made:

• discovery.zen.ping.unicast.hosts has the DNS RR name and Zen tries connecting to all the IPs
• clients initially connect to the cluster via the DNS RR name

Why? Modern HTTP clients will get a random node in the cluster, but if it fails they'll automatically try the rest until one works [1], when it can then discover the rest of the cluster in the normal way. Currently AFAIK to guarantee a client will always work on startup (ie. before it has a chance to discover all the other nodes in the cluster) and for unicast zen to always work, it has to be supplied with a _current_ list of at least half (?) of the master-eligible nodes.

This would make the cluster rely on DNS to operate, although I expect most people refer to their nodes via DNS anyway, so doesn't add any further dependency for people who'd want to take advantage of this.

[1] http://blog.engelke.com/2011/06/07/web-resilience-with-round-robin-dns/

[ianwestcott]

I just attempted to build a cluster using round robin DNS, and was surprised to find that it doesn't seem to work. Am I missing something, or is this not a supported discovery method? If not, what is the current limitation on supporting round robin DNS discovery?

[JesperTerkelsen]

:+1: Would be really nice to have this.

[clintongormley]

I _think_ this should now work with 2.0. Want to try it out and let us know?

[jimmyjones2]

TL;DR confirmed working with 2.1. Should this now be documented? I think for some use cases this is a great feature!

sudo docker run -i -t elasticsearch

Now add a load of docker IPs as A records for es.mydomain, making sure one matches the container above. Check:

nslookup es.mydomain
Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
Name:   es.mydomain
Address: 172.17.0.18
Name:   es.mydomain
Address: 172.17.0.16
Name:   es.mydomain
Address: 172.17.0.15
Name:   es.mydomain
Address: 172.17.0.19
Name:   es.mydomain
Address: 172.17.0.17

Start up another ES node:

sudo docker run -i -t elasticsearch -Des.discovery.zen.ping.unicast.hosts="es.mydomain"

It'll give "No route to host" errors for all but one of the IPs associated with es.mydomain and join the other container in a cluster.

[jwaldrip]

works like a charm

[JarenGlover]

@jwaldrip @ianwestcott it sounds like the TLDR; i can now bootstrap a elasticsearch master cluster using DNS? (ex. consul DNS)

[geekofalltrades]

It looks like this was documented at one point. Is it still? I can't find this mentioned in the Elasticsearch docs on elastic.co.

UPDATE: A-ha: found it still in the network settings documentation, where the change linked here originally put it. But it's not documented under Discovery; seems it merits a mention there?