Azure-functions-durable-extension: Looking for clarification on the use of Await in Orchestration functions

[ssanderlin]

First, let me start by saying that I'm familiar with the code constraints article and have reviewed an issue in azure-docs requesting clarification on this topic.

My question is somewhat different. If I move the invocation of CallActivityAsync into a method in the base class of the Orchestrator class, is invoking that method using await a problem?

For example:
```c#
public abstract class OrchestratorBase
{
protected async Task DoTheThing(DurableOrchestrationContextBase context, int value)
{
var result = await context.CallActivityAsync("TheActivity", value).ConfigureAwait(true);
return result;
}
}

public class Orchestrator : OrchestratorBase
{
public async Task RunOrchestrator([OrchestrationTrigger] DurableOrchestrationContextBase context)
{
var result = await DoTheThing(context, 1).ConfigureAwait(true);
}
}
```

This doesn't throw the typical InvalidOperationException about multithreading, but there is some debate among myself and my colleagues as to whether this is permissible or whether we should instead do:
var result = DoTheThing(context, 1).Result;
or something else entirely.

Just to add some context: we have three durable functions that share code to varying degrees; the goal is to abstract the common code out to the base class so that we're not duplicating it everywhere.

Official guidance would be greatly appreciated!

[cgillum]

There shouldn't be any problems as long as:

1. The code continues to be deterministic
2. You're not introducing new threads.

Using a base class to make durable async calls doesn't violate either of these, so you should be fine.

I haven't tried this, but I feel like DoTheThing(context, 1).Result could result in deadlocks because our orchestration dispatcher is single-threaded. Also note that should similarly avoid ConfigureAwait(false) because running continuations on non-dispatcher threads will also cause problems.

[ssanderlin]

Thanks for the response. Yes, any code that could potentially be non-deterministic is executed within activity functions.

Regarding your concerns around DoTheThing(context, 1).Result, that was my thought as well - seemed like it was asking for trouble, although I haven't actually tried to create a situation that would be problematic.

A related question, if I may: how does the "not introducing new threads" rule reconcile with awaiting Task.WhenAny() or Task.WhenAll() on a collection or array of Tasks in a Durable? Even though the Orchestration function is awaiting a single Task (using ConfigureAwait(true)) that's returned by those methods, doesn't the TPL create new threads to run the tasks in parallel?

[cgillum]

All Durable async APIs (CallActivityAsync, CreateTimer, WaitForExternalEvent) are executed and have their continuations executed on the single dispatcher thread. We own the TaskCompletionSource<T> objects for all of these which allows us to control this behavior. I don't exactly remember the behavior of Task.WhenAny() or Task.WhenAll() but those are guaranteed to be executed on this thread as well. I believe part of what makes this possible is the fact that the Durable Task Framework has a custom single-threaded synchronization context (code here).

The orchestration itself never does anything in parallel. However, the work that it schedules does get executed in parallel. For example, when fanning out, you're effectively enqueuing a bunch of work item tasks on an Azure Queue, and those work items can be processed concurrently. The responses are then fed back to the orchestrator for one-at-a-time processing in a way that ensures APIs like Task.WhenAny() and Task.WhenAll() always execute in a deterministic way (e.g. when using Task.WhenAny(t1, t2), the same task always wins on each replay).

[ssanderlin]

Thanks for the info. I'm not very familiar with the internal workings of the TPL, but this helps a lot - I'm going to dig through the DTF code, because now I'm very curious as to how you're forcing deterministic execution.

Just to be sure that my understanding is correct, I wanted to just summarize the original topic of discussion. My takeaway is that the issue is not so much the await (since this operator is just syntactic sugar and doesn't actually create any threads) but the nature of the async code that is invoked from the Orchestrator. Thus, awaiting methods in the orchestrator or its base(s) is fine if and only if all of the following are true:

• None of the invoked method(s) invoke any async methods except for the supported Durable async APIs
  
  
  
  • However, repeats of the same pattern are permissible, i.e., RunOrchestrator() could invoke Orchestrator.DoIt() which itself invoked OrchestratorBase.DoTheThing(), as long as both methods didn't invoke any async methods other than methods that follow the same pattern or the supported Durable async APIs
  
  
• All of the returned Task objects (down to and including the Task that is returned by the supported Durable async API) have their awaiters configured to marshal back to the original context either using ConfigureAwait(true) or by omitting ConfigureAwait() entirely
  
  
  
  • As usual, ConfigureAwait(false) is not permitted anywhere in an Orchestrator function
  
  
• No non-deterministic code is introduced anywhere
• No new threads are created
  
  
  
  • Hence the exclusion of ConfigureAwait(false), though this also precludes the use of Task.Run()
  
  
• All other code constraints are followed

Is my understanding correct?

[kevin-secrist]

Would it be possible to add some of the knowledge in this thread to the official docs? I think when I originally read the code constraints article I was given the impression that I could not do what is described by @ssanderlin which would be a pretty significant limitation in terms of code structure if that were true.

[APIWT]

I am very thankful I found this thread. We are adopting durable functions and I had written some code (that works great locally) and I was super concerned it was going to blow up in Azure because I was using await (similar to what it being done in the original post here). We do not make any call to ConfigureAwait, so my guess is that it doesn't actually create another thread. To be perfectly clear, we are only using CallActivityAsync inside the method we are using await on. We do build a service provider as well (to avoid some issues with dependency injection that occur with the out of the box host container), but I have a hard time believing that uses ConfigureAwait(false) under the hood. I guess I'll find out for sure once we deploy our code.