Orleans: Deactivate grains upon write conflict

Created on 10 Feb 2016 · 6Comments · Source: dotnet/orleans

Currently Orleans does not deactivate grains when a call to WriteStateAsync fails with a write conflict. In such cases, a grain may fail indefinitely and exceptions would need to be handled by user code.

The correct behavior in this case (in my opinion) is to force a grain deactivation so that any transient state is cleared and any persistent state is read from storage on the receipt of the next message.

I believe we should either throw a well-known exception from storage providers in this case and have Orleans handle the exception with a call to DeactivateOnIdle before rethrowing, or require storage providers to handle this case themselves. I prefer the former.

enhancement

Source

ReubenBond

Most helpful comment

I think as @gabikliot said, it only should deactivate if the user lets the exception escape and not handle it since in many valid cases you might be able to tolerate some write failures or want to reread state and try to write again (i.e. if it is a conflict, you might want to handle it yourself).

ashkan-saeedi-mazdeh on 29 Nov 2016

👍2

All 6 comments

@reubenbond, I agree.
I clearly remember we discussed that in the past, either privately with Sergey and Jason, or on GH.
I think the design was to throw a known Orleans specific conflict exception from the storage provider, the grain can catch and recover if it wants, but if not and the exception escapes, the runtime deactivates the grain and returns the exception to the caller.

gabikliot on 10 Feb 2016

👍1

Hi, I want to add a comment regarding possible consequences with de-activating a grain due to state conflict. With Azure worker role deployment, conflict may be due to swap-upgrade where production & staging slot work at the same time against same storage account. It may be important for the grain to remain active because it runs a low-frequency timer that is critical and shouldn't wait for re-activation. I thought we can add a configurable flag whether to de-activate grains on state conflict, that is by default false in order to keep backwards comparability for Orleans users relying on current behaviour. The idea @gabikliot suggested, with Orleans handling the conflict if the grain does not, seems like a better solution than configuration, since existing applications recovering from state conflicts are probably handling the conflict exceptions already.