Akka.net: Problem when two well-know Cluster Seed nodes starts at same time

[oberdannunes]

I'll try to describer better the issue I'm getting here. Scenario:

Seed node A:

remote {
    helios.tcp {
        port = 4051
    hostname = "192.168.3.37"
    }
}
cluster {
    seed-nodes = [ "akka.tcp://[email protected]:4051", "akka.tcp://[email protected]:4051" ]
}

Seed node B:

remote {
    helios.tcp {
        port = 4051
        hostname = "192.168.3.35"
    }
}
cluster {
    seed-nodes = [ "akka.tcp://[email protected]:4051", "akka.tcp://[email protected]:4051" ]
}

It seems that, when I start the node B before the node A starts InitJoin messages, both nodes start as role leaders, like two distinct clusters. And there is no INFO messages in each one about the other node, neither the InitJoin messages. I've seen some DEBUG messages, but I couldn't determine yet when it stops to look at the other seed.

Is this a expected behavior or can be something wrong?

Thanks!
Oberdan

[Aaronontheweb]

@oberdannunes your problem is that your seed node configurations both have each seed list itself as the first seed node. During cluster startup when the cluster first forms, Akka.Cluster requires that the first seed node listed in the seed node list must be up in order for the cluster to form in order to avoid this very issue. All you need to do to fix it is to have the seed node list be identical on all nodes. Right now the order is different on both of your seeds.

Sorry for the confusion here; we haven't documented this very well. Opened up an issue on our docs website for addressing this.

[oberdannunes]

@Aaronontheweb thank you very much! Your answer really helped me! I made the changes and now the cluster is initializing like expected.

Best regards,
Oberdan

[aventurella]

So then there is really no point in listing additional seed nodes for the first/primary/seed-seed seed node. Since that node has to be up before starting the other seeds. If that primary node goes down, you couldn't simply restart it as it would come up, but it would join itself, and the rest of the cluster would not see it.

So effectively all the seed nodes in the cluster probably should have a single seed node? Any services that exist in the cluster however should list all the different seed node possibilities, so they can try to find someone to let them into the cluster?

[Aaronontheweb]

So effectively all the seed nodes in the cluster probably should have a single seed node? Any services that exist in the cluster however should list all the different seed node possibilities, so they can try to find someone to let them into the cluster?

No. The _very first time the cluster forms_ the first seed node in the seed node list _must be up_ in order for the cluster to form, in order to avoid the split brain scenario I described above. After that the order doesn't matter at all.

Best rules of thumb for this, IMHO, is to eliminate moving parts from your seed node strategy:

1. Use something like Lighthouse and just park seed nodes on static addresses and don't touch them during deployments.
2. If you do need to upgrade Lighthouse, and this should be done rarely (only when you need to upgrade serialization or Akka.NET itself to a new major / minor version, really) - upgrade the seed nodes one at a time. Don't do it all at once or they'll form a new cluster.