Pulsar: Don't depend on ZooKeeper for coordination

[thedrow]

Expected behavior

Pulsar should be able to function without ZooKeeper or provide other backends for Etcd and Consul

Actual behavior

ZooKeeper is required for Pulsar's operation.
One of the major hurdles of using Kafka in production is ZooKeeper. The component is a source for many operational problems and thus not suitable for use in small startups/shops.
If this project wants to succeed it should provide us with a way to replace this dependency with another one or implement all coordination on it's own (as Mesos is working on now)

[dave2wave]

Are you in an environment that is virtualized? When ZK gets confused are you having migrations? I've seen a case where Solr 5.5 clusters which also use ZK got confused in those scenarios especially when logging was configured off host. The solution required 5 ZK nodes to 3 Solr nodes. along with not using mounted disks for Jetty logs in front of the Solr nodes.

Also I think that the ZK dependency in Pulsar is due to the use of BookKeeper...

[youurayy]

Bookkeeper seems to be Etcd ready: https://github.com/apache/bookkeeper/issues/1639
Kafka also already supports etcd.

Is it time for Pulsar to be reassessed for adding etcd support?

[mingfang]

I'm now running without zookeeper by using zetcd in front of etcd

[atombender]

I'd like this, too. We use Etcd.

How solid is zetcd? It's not been updated since January, the last pre-release was 16 months ago, and the last release was 2 years ago. Readme doesn't have any information on whether it's production-ready. Not too promising.

[adamfeldman]

@mingfang thank you for sharing that you're using zetcd! How has your experience been?

(pardon me for potentially hijacking this issue -- it is very difficult to find information on zetcd in production...)

[mingfang]

Unfortunately zetcd didn't work very well so I stopped using it and reverted back to using zookeeper.

The problem was it intercepts the four letter health check command (srvr) so there was no way of knowing if the backend fails. Also the project seems abandoned.

[IsNull]

When I stumbled upon Plusar I died twice. First I read the description and, coming from a Kafka / Kubernetes background, I thought man these guys really tick everything I could wish for. After that, I clicked the Docs page and when I discovered those nasty ZK boxes in the diagram the dream died as fast as it was emerging.

I can understand the argument of not repeating yourself and it may look like a good idea to have that complicated quorum stuff sorted out with a separate microservice (ZK). However, If you produce the situation that the state of multiple services are interconnected and can desynchronize, its an operational nightmare. At least in a containerised and managed environment like Kubernetes.

Kafka has (finally) accepted this reality and is moving away from an external solution to embed quorum (KIP-500). By having the Quorum State and Broker Data state in the same location, desynchronisation are no longer possible and operations are very much simplified.

So my two cents would be, to not make ZK replaceable by something other, but instead remove the need for an external solution altogether.

[kjaggann]

My understanding zookeeper has an issue in terms of "quorum" and "split Brain"
my finding correct me if I am wrong lets take we have 5 servers. The voting to elect the leader is based on the majority of the servers. the number of servers must accept specific server to become the leader, the other becomes the followers
So there is this bad thing that may happen called "split brain". A split brain is simply this, as far as I understand: The cluster of 5 servers splits into two parts, or let's call it "server teams", with maybe one part of 2 and the other of 3 servers. This is really a bad situation as if both "server teams" must execute a specific order how would you decide which team should be preferred? So it is really important to know what "server team" is still relevant and which one can/should be ignored.

When one server in {3,4,5} realizes the leader is gone, it triggers an election. Once a new leader is elected, it starts serving client requests normally. The other partitioned cluster {1,2} won't serve any client request and will be stuck in the election until the partition is resolved. So it waits till the issue resolved and the cluster runs normal again.

Currently, pulsar uses Apache zookeeper to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of pulsar itself, in a separate Zookeeper cluster. so is there a plan to break this dependency and bring metadata management into Pulsar itself.