Elasticsearch: cluster node not joining

[arslan70]

Elasticsearch version: 5.0.0

Plugins installed: [none]

JVM version: 1.8.0_25

OS version: Ubuntu 14.04.5 LTS

Out of the box extract from tar.gz. Trying to setup 2 nodes in a cluster but cant:

Steps to reproduce:

1. Setup Node - 1

node.name: SANNNNN-1

network.host: 10.3.185.250

discovery.zen.ping.unicast.hosts: ["10.3.185.251:9300"]

1. Setup Node 2

node.name: SAN-2

network.host: 10.3.185.251

discovery.zen.ping.unicast.hosts: ["10.3.185.251:9300"]

1. Start Node 2( Node is up and running)

   1. Start Node 1 and getting this exception
      ([INFO ][o.e.d.z.ZenDiscovery ] [SANNNNN-1] failed to send join request to master [{SAN-2}{DxExoYHHTu2-rFvuvQSuEg}{OfYBe97HQCmcha63CFiYlQ}{10.3.185.251}{10.3.185.251:9300}], reason [RemoteTransportException[[SAN-2][10.3.185.251:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {SANNNNN-1}{DxExoYHHTu2-rFvuvQSuEg}{hP4gLRugRgWzSuNnEhGHSw}{10.3.185.250}{10.3.185.250:9300}, found existing node {SAN-2}{DxExoYHHTu2-rFvuvQSuEg}{OfYBe97HQCmcha63CFiYlQ}{10.3.185.251}{10.3.185.251:9300} with the same id but is a different node instance]; ])

Both the nodes are getting the same ID, atleast the first part {DxExoYHHTu2-rFvuvQSuEg}
{SAN-2}{DxExoYHHTu2-rFvuvQSuEg}{OfYBe97HQCmcha63CFiYlQ}
{SANNNNN-1}{DxExoYHHTu2-rFvuvQSuEg}{hP4gLRugRgWzSuNnEhGHSw}

[bleskes]

@arslan70 the first part of the node id is persisted in the data folder of the node and is now stable across restarts (so you can identify the node). I presume from your description that this error doesn't go away after a couple of seconds. Did you copy the data folder from 1 to 2 when you were setting things up? are you using some kind of a container image that may have a non-empty data folder baked into it?

[arslan70]

Yes, the entire elasticsearch folder(extracted) was copied from one node via scp.

[bleskes]

@arslan70 that explains it. I suggest you start with an empty data folder and let ES do the copying if needed. I'm going to close this now as it seems all works as designed.

[arslan70]

Thank you @bleskes . Deleted the data folder on one node and the cluster is up :)

[celesteking]

This stuff should be documented.
I cloned existing elastic VM and now I need to change the unique node hash / UUID / ID, whatever it is in order to let it join the cluster.

[zacbayhan]

Sorry to bring up an old topic but which data folder did you delete? The once located in the logstash directory?

[JosiahAllen]

Zac, I believe they mean the /var/lib/elasticsearch/nodes directory

[zacbayhan]

Thanks JosiahAllen, took me a bit but figured out which one they were talking and was able to get the cluster to boot. onward to the next problem!

[rana-prashant-208]

Sorry to open an old topic but is there a way to remove or modify the Node hash / UUID / ID on startup?

[rana-prashant-208]

@celesteking can you tell me how you changed the node ID

[bleskes]

@PRASHANT9897941533 I'm not sure why you would like to do that (that may influence the answer). As note above, ES will start if you delete the data folder (and new data will be put on it by the cluster). Can you open a topic on the forum so we can see what's the best approach for you? we reserve github for issue and features.

[rana-prashant-208]

https://discuss.elastic.co/t/cluster-node-not-joining/136072 added.
Thanks

[ziyunhx]

@PRASHANT9897941533 Thanks, if the data is large enough. the scp method will save you a lot of time.

[lepe]

Zac, I believe they mean the /var/lib/elasticsearch/nodes directory

I installed elasticsearch in 3 LXD containers (created one and copied over to other 2). I had this issue, but removing: /var/lib/elasticsearch/_default/ directory and restarting ES fixed it.