Druid: Support for etcd and consul

[F21]

It would be awesome if druid could support etcd and consul in addition to Zookeeper. This allows people who already have etcd (perhaps by running something like coreos) to not have to deploy a zookeeper cluster.

[fjy]

@F21 one of our goals is to move Druid off of Zookeeper entirely and instead have some of the leader election stuff be done in the coordinator.

[F21]

That would be really awesome! Less third-party moving parts. Do you have a roadmap or estimate of when that might happen?

[xvrl]

Closing this since there currently are no plans to support additional dependencies, only removing them.

[leventov]

I actually think that this issue should be reopened. What if we remove support for ZooKeeper coordination and replace it completely with Consul? We could also piggy-back it's key-value store for indexer locks.

[sixtus]

I am all for it! We are running consul in production for a good year now, it sneaked in "well, let's deploy and see how it goes" to "consul keeps the lights on". It's orders of magnitude better than zookeeper.

[andriyfomenko]

it would be great to abstract zookeeper/consul/etcd out and allow for the externally supplied dependency setting[s]

for example, in our case, we have kubernetes-native application and we would consider getting druid hosted in the same cluster where the application lives, etcd usage looks to be logical in this scenario, but having no choice makes it hard unnecessarily

[jbguerraz]

Same here, would be happy to replace zookeeper with etcd :)

[himanshug]

actually most of the work to abstract zookeeper is already done. see https://groups.google.com/forum/#!searchin/druid-development/zookeeper/druid-development/tWnwPyL0Vk4/2uLwqgQiAAAJ

I could probably write an etcd extension as reference for using something other than zookeeper .