Akka.net: Implement cluster discovery

[Horusiath]

Right now Akka.Cluster doesn't expose any common way to implement cluster discovery mechanism - everything is either config- on JoinSeed-driven. I think, we could think about some cluster node discovery plugin, which would expose some API to be implemented by specific implementations i.e. based on ZooKeeper, ASF or even some distributed datastore.

Any thoughts @akkadotnet/developers? Should we talk about it with Typesafe?

[Haukinger]

Any news here? It would be great if we could configure some udp-broadcast-plugin and make instances of our app discover each other and start/join a cluster without seed nodes.

[JTrotta]

I realized it by myself. But it would be wonderful to have this feature native.

[Aaronontheweb]

Why is this necessary? What's wrong with having seed nodes?

[JTrotta]

Because in addition to the cluster node machines, you need to have seeds nodes machine active. Moreover if you implement the seed node inside the cluster nodes you need to have that machine active before others. In my opinion, the best cluster is composed by parithetic nodes.

[Haukinger]

I think it strongly depends on the usage scenario. If you run your cluster in a data center or amazon cloud or something like that, you have no problem setting up seed nodes and preconfiguring everything.

But if you have an app running on multiple machines on a lan (office/campus) and have the cluster as secondary feature built into the app (think ncrunch) to distribute work to other machines running the app (but less/no cpu load right now), then the user does not want to take care of cluster configuration. He just wants to start the app and see it utilize all available machines.

[JTrotta]

@Aaronontheweb it seems you are not convinced, or do not agree at all ?

[Aaronontheweb]

Not convinced either way. It's something I'd probably want to kick the tires on myself before I formed a stronger opinion about it. So I'll leave the current status as "thinking it over" :p

[mithril52]

Scenario in which I would like the option of using another cluster bootstrap mechanism:

I am putting together an Apache Mesos cluster in which I will be running one or more dynamic Akka.Net clusters that get spun off as containers. Mesos uses Zookeeper for master election purposes. It would be great if I could use the same Zookeeper for Akka.Net cluster bootstrapping as well so I do not need a secondary form of service discovery that needs to be managed alongside Zookeeper. Static seed nodes will also not work in this instance as the containers can get placed on any machine in the cluster.

[JTrotta]

@mithril52 uses Zookeeper, somebody else uses Etcd. It would be very nice to integrate those cluster stores to discover an Akka cluster.

[Horusiath]

@mithril52 @JTrotta I've created a project for a generic service discovery mechanism, called Akka.Cluster.Discovery. So far it only implements Consul-based service discovery (also allows consul to act as cluster health check service). Unfortunately I haven't got time to implement other providers.

[JTrotta]

@Horusiath Great news. I could add Etcd and UDP features, if you like.

[Horusiath]

@JTrotta sure, contributions are always welcome ;)

[mithril52]

@Horusiath that sounds great. I’ll take a look and see if I might be able to contribute with adding ZooKeeper to the mix