Orleans: Question: Can response timeout be defined per grain type/per grain method?

[Eldar1205]

In my cloud service there's a grain that serves a request that may take a while, up to few minutes (creating service bus namespace and waiting for activation) while other grains serve requests very fast. The global configuration ResponseTimeout allows setting a single timeout for all grain requests, and I'd like some grain requests to specify shorter timeout than others. Is this somehow possible today with Orleans out of the box? If not, is there a way to cancel a grain request given the client decides to give up on it, as a way to implement custom timeout that is shorter than the global one?

[sergeybykov]

It is possible but tedious to set response timeout for individual calls via the global static GrainClient.SetResponseTimeout by setting it before a call and immediately resetting back right after it. https://github.com/dotnet/orleans/pull/1599 is trying to add support for cancellation tokens.

I general, we recommend the pattern of making a short call to _begin_ a long running operation with a separate completion notification. Although there is nothing wrong with long running calls if you don't have a large number of them (millions) executing in parallel.

[Eldar1205]

1. Can you please specify a way to provide separate completion notification? Are you suggesting something inspired by HTTP Accepted for initial request and have the client poll the grain to see if the request completed? If so, is this advised for the grain to run the long running operation using default task scheduler?
2. If the grain request is sent from a reminder and it's timed out, the reminder traces an error that it couldn't deliver reminder. Should this be ignored since the grain still executes the "ReceiveReminder" code and the reminder will tick again after its period at any case?

[sergeybykov]

First of all, no long running operations should execute on the grain threads. They should be "outsourced" to the thread pool as described in http://dotnet.github.io/orleans/Advanced-Concepts/External-Tasks-and-Grains.

There are several options on how to report completion: polling, observer notification, streams. In this pattern grain acts like a control state machine that manages state of the long running operation that executes on the threadpool or even as a separate process. Since the grain is never blocked, it is available for status query calls and reminders, can check and report progress on timer or callbacks form the executing operation or cancel the running operation.

Yes, if a reminder is scheduled to fire periodically, it will tick after the next period.

[gabikliot]

I would like to upvote the option of submitting long running task, and then reporting status via stream back to the caller, with an option to cancel via the upcoming cancellation support.

[eckerdj7]

First of all, no long running operations should execute on the grain threads. They should be "outsourced" to the thread pool as described in http://dotnet.github.io/orleans/Advanced-Concepts/External-Tasks-and-Grains.

There are several options on how to report completion: polling, observer notification, streams. In this pattern grain acts like a control state machine that manages state of the long running operation that executes on the threadpool or even as a separate process. Since the grain is never blocked, it is available for status query calls and reminders, can check and report progress on timer or callbacks form the executing operation or cancel the running operation.

@sergeybykov Can you link to some examples of how this is done? I can't seem to find any examples of this in the documentation or anywhere else. Also, your link doesn't work. :stuck_out_tongue_closed_eyes: You are missing the "Documentation" part of the path.

[sergeybykov]

Sorry, we moved the doc pages around since then. Here's the current link - http://dotnet.github.io/orleans/Documentation/Advanced-Concepts/External-Tasks-and-Grains.html.

[zevanreg]

The

Sorry, we moved the doc pages around since then. Here's the current link - http://dotnet.github.io/orleans/Documentation/Advanced-Concepts/External-Tasks-and-Grains.html.

The current link is http://dotnet.github.io/orleans/Documentation/grains/external_tasks_and_grains.html :)