Elasticsearch: Tribe Node

[kimchy]

Note, this feature is experimental.

The tribes feature allowed to create a tribe node that can act as a federated client across multiple clusters.

The tribe node configuration looks something like this:

tribe.t1.cluster.name: cluster1
tribe.t2.cluster.name: cluster2

The configuration above configure connections to 2 clusters, named t1, t2. It creates a "node" client to each (so by default, above, multicast discovery is used). The settings for each node client is extracted from the tribe.[tribe_name] prefix.

The way the tribe node works is by merging the cluster state from each cluster, and creating a merged view of all clusters. This means all operations work the same, distributed search, suggest, percolation, indexing, ... .

The merged view drops conflicted indices and picks one of them if there are 2 indices with the same name across multiple clusters.

By default, read and write operations are allowed. Master level read operations (cluster state for example), require setting the local flag to true (since there is no elected master). Master level write operations are not allowed (create index, ...).

The tribe node can be configured to block write operations tribe.blocks.write to true, and metadata operations by setting tribe.blocks.metadata to true.

[gseng]

Just to confirm, is this how it's supposed to work?

Steps

• We have 2 clusters, boston_cluster and seattle_cluster.
• We leave their setting as is.
• We create a new tribe node with the following setting:

tribe.t1.cluster.name: boston_cluster
tribe.t2.cluster.name: seattle_cluster

• When the tribe node starts, it will connect to both clusters and aggregate their information.
• We can query this tribe node to retrieve results across both clusters.
  
  
  
  • For example, if boston_cluster had an index docs-bos and seattle_cluster had an index docs-se, a <tribe_node>/_all/_search would return documents from both indices.
  
  

Questions

I tried the above steps but wasn't able to get the aggregated results. I think I'm missing something.

• What cluster should the tribe node have? It joins the elasticsearch by default.
• Are the node.master and node.data settings important for the tribe node?

[kimchy]

yea, this is how ti will work. You need to make sure that those internal clients opened against boston and seattle can actually connect. Make sure they were started each with the respective cluster name, and by default, it will do multicast discovery, so potentially you need to configure unicast discovery? Something like:

tribe.t1.cluster.name: boston_cluster
tribe.t1.discovery.zen.ping.unicast.hosts: ["host1", "host2"]

tribe.t2.cluster.name: seattle_cluster
tribe.t2.discovery.zen.ping.unicast.hosts: ["host3", "host4"]

UPDATE: to answer your other questions, the cluster name or the nodes.master, node.data are not important for the tribe node.

[jensihnow]

Hi,
what about Tribe Nodes on AWS?

How to configure three cluster in different regions (US,EU,AP) for tribe nodes?
In this scenario we would like to use the AWS EC2 discovery, but it looks like this doesn't work with Tribe nodes.

Please enhance the tribe nodes docs. We know it's brand new, but it's too less detailed.

Thanks

[Kamapcuc]

Is it possible to have the same data in all clusters?
For example:
We have the some servers in data centers in several countries to decrease ping and latency for clients. For some reasons, sometimes there is no connection between those data centers. So we want tribe to merge data in clusters after connection restore.

[rparkhunovskyi]

Would you please update the docs with a wider life-applicable examples? What is there now is quite unclear and non-obvious how to set the tribe node in AWS at all. Thank you.

[NehaSood]

Do tribe nodes support AWS EC2 discovery? I wouldn't want to hard-code IP addresses to support unicast discovery.

[jianchen2580]

Do tribe nodes support AWS EC2 discovery? +1

[bleskes]

Do tribe nodes support AWS EC2 discovery?

They should yes.

[dshweta]

I am still unable to find any documentation/instructions for how to set the tribe node in AWS. We are trying to have unified kibana dashboard for multiple clusters in different regions in AWS. Any pointers/input would be helpful

[jianchen2580]

@dshweta, which discovery in your clusters, EC2 discovery or Zen discovery?

[dshweta]

@jianchen2580, I am new to AWS and not sure about it. The AWS Elastic Search clusters are created using the AWS console in different regions. Tried to find the cluster discovery but no luck.

[jianchen2580]

@dshweta -
For Zen discovery -
make sure your tribe node can reach clusters (9200/tcp, 9300-9310/tcp)

```
tribe:
eu:
cluster.name: eu-cluster
discovery.zen.ping.unicast.hosts: ["tribe-node-ip", "eu-cluster-node1-ip", "eu-cluster-node2-ip", ...]
network.bind_host: 0.0.0.0
network.publish_host: tribe_ip
us:
cluster.name: us-cluster
discovery.zen.ping.unicast.hosts: ["tribe-node-ip", "us-cluster-node1-ip", "us-cluster-node2-ip", ...]
network.bind_host: 0.0.0.0
network.publish_host: tribe_ip

[bleskes]

@dshweta AWS ES as a service doesn't support transport level access to the cluster (see here ) and thus also doesn't support tribe nodes. This is different for Elasticsearch that is hosted on your own instances on EC2.

[dshweta]

@bleskes Thanks for your response.

[lucyl]

Does tribe node use client clusters' coordinating nodes, or it acts as a coordinating node itself?

[amazinganshul]

Hi,
What are the scaling concerns related to a tribe node? We have 2 nodes added to the tribe node environment but I don't see them connecting to each. They independently connect to the federated clusters but not to themselves. How will they sync the state among themselves?

[jianchen2580]

You only need one tribe node, and this tribe node will join two clusters and become the norm node of the cluster.

[amazinganshul]

Hi, won't one tribe node become bottle neck? We have a use case where this tribe node will be used both for reads and writes. Also our write tps will be around 1500 writes per second for both clusters via tribe nodes and read tps of around 200 tps from kibana via tribe node.