Orleans: Question: fail proof observer handling.

[Xanatus]

I'm kinda new to Orleans and was wondering if it's possible to make either observers or stream observers persistent and fail proof across silos.

Consider this:

You have a client and 2 silos.

The client subscribed to a grain on one of those silos to receive update events.
Now the silo that stores the activation of the grain which stores the subscriber list crashes and the second silo may take over and create a new activation.

The new activation doesn't know the previous subscribers and the subscribers don't know they got unsubscribed and therefore the connection is lost without notice.

Is there any way to verify on the client whether a subscription got lost so it can be re-requested?
Or maybe is it possible to somehow serialize the ObserverSubscriptionManager to a persistent storage and revive that list on the new activation?

I tried both approaches with both SubscriptionManager and Streams but can't get either to work.

Maybe I'm missing something here and there is a much more elegant solution.
Any help would be appreciated.

[sergeybykov]

There's a fundamental gap here. Clients aren't virtual, unlike grains, and can disconnect and disappear forever at any moment. Hence, holding and persisting observer references for clients works only while the client is actually there. ObserverSubscriptionManager is just a tiny helper class. A list of observer references can be stored and serialized in many different ways.

The pattern that we've recommended for handling client observers is periodic re-subscription/re-confirmation that the client is still there and wants to stay subscribed.

Client keeps making a subscribe call periodically (every minute or whatever interval makes sense from the app perspective). The grain updates a timestamp associated with that client's subscription every time is receives a subscribe call from it. Periodically (every minute, every 5 seconds or whatever interval - this part is cheap) the grain scans its list of subscribed clients and removes those that haven't confirmed their subscriptions in the last N re-subscription intervals.

Essentially, this gives you an eventually consistent list of subscriptions where the chosen time intervals define staleness of the list.

[ReubenBond]

Here's a helper class which assists in implementing that pattern. You can use it instead of ObserverSubscriptionManager. It supports associating some additional data with your subscribers.

https://gist.github.com/ReubenBond/0bb430a52eedce50e6976f2f89154f5a