Orleans: Add OnClusterConnectionRecovered event in the ClusterClientBuilder

[bobanco]

Hi Guys,

i want to propose adding OnClusterConnectionRecovered event on the cluster client builder so it will be much more easier for us who are using streams in the client side to gracefully resume the subscriptions
when cluster client lose the connection to the cluster and will successfully recover the connection.

Because currently if cluster client lose the connection and when it will get reconnected all of the pervious streams subscriptions are not automaticly resumed, and the only option for now is to handle OnClusterConnectionLost event and manually retry untill cllient gets reconnected and resume the subscriptions so we can't relay on the builtin auto-reconnect feature of the cluster client itself. If we introduce the proposed event then we can gracefully just resume/subscribe to the event streams which we were subscribed too.

this is also related to #3484 and #2226

[gabikliot]

Seems like it would have been much better if we automatically resubscribed the client streams and made client subscriptions be virtual, unaware of the cluster liveness, the way it was envisioned from the beginning.

[amccool]

I thought I could use ClientBuilder.AddClusterConnectionLostHandler from https://github.com/dotnet/orleans/issues/2226#issuecomment-310899796 as a way to resubscribe my client streams. However this isnt working for me when I always have at least one silo up.

consider Silos A,B,C all running. Client is subscribed to a stream X.
I lose connection(silo dies) to A and B. Silo C remains up and functional
Silo A is restarted and rejoins
Then Silo C dies.
Once Silo C dies - Client no longer receives published messages from stream X.

In this case I the ClusterConnectionLost is never fired, because numberOfConnectedGateways never goes to 0

internal void OnGatewayConnectionClosed()
        {
            if (Interlocked.Decrement(ref numberOfConnectedGateways) == 0)
            {
                this.connectionStatusListener.NotifyClusterConnectionLost();
            }
        }

I would love to see @gabikliot vision for virtual stream reconnection. But for the short term (I need 1.5.x right now.

I would be happy enough to get an event if the numberOfConnectedGateways changed (added, removed, whatever), but that appears to need a change to the internal classes and I dont to see a way to augment any of the existing services to tie into numberOfConnectedGateways.

Could a PR on 1.5.x deliver OnConnectedGatewaysChange (not sure its possible to get OnClusterConnectionRecovered) without forcing a problematic/misnamed interface especially now that 2.0 is out?

[mindlink]

This would be fixed if GatewayCountChangedHandler worked, since then you could resubscribe whenever the count is decremented #6098