Elasticsearch: Kibana + 2 data nodes = uneven search loads?

[markharwood]

Perhaps a common scenario here where in a 2 data node system one data node gets hammered while the other is idle. The reason seems to lie in default Kibana and search routing config.

• Kibana looks to use a preference key for routing searches based on a session ID. It also uses _msearch for dashboards to bulk up requests.
• All primaries were on one data node and the replicas on the other (not uncommon after a restart)

When elasticsearch gets a request to route on a preference setting that is a session ID it looks to select a choice of primary vs replica by hashing the preference string only. If each shard of an index presents their list of primaries and replicas in the same order (and I haven't confirmed this is the case!) then this routing algo will pick the same node for all searches given the same session key which is what the user was seeing.

If we hashed the preference key AND the shard number we would randomize the choice of primary vs replica and hence node choice for each shard. This would spread load more evenly.

[javanna]

Isn't the whole point of the preference to query the same shard copy all the time when the value is the same?

[markharwood]

Yes. This change proposes that for each shardID we deterministically pick the same replica given the same session key _but not using the same policy across all shardIDs_ which leads to uneven loads.

[markharwood]

We discussed this on FixItFriday and decided to adopt the proposed change of including shard_id in the hash of the preference key but with appropriate check for backwards-compatibility.

[s1monw]

@markharwood I am +1 on this but we need to make sure that we preserve BWC. It should be rather simple here like in OperationRouting you can do this:

private ShardIterator preferenceActiveShardIterator(IndexShardRoutingTable indexShard, String localNodeId, DiscoveryNodes nodes, @Nullable String preference) {
  // ...
  if (nodes.getMinNodeVersion().onOrAfter(Version.V_6_0_0_alpha1_UNRELEASED)) {
    // use new method
  } else {
    // use old method
  }
}

[ruria]

Sorry, but I don´t get the point. Why is so important to preserve BWC? "eventually" primary and replica can diverge, so hitting same shards in a "real time" scenario along the (say) session is fine. But if you update your cluster, restart nodes, and so on... to install this "patch" What´s the problem to hit another shards? They should be synced and almost sure your session is changed.