Azure-functions-durable-extension: CallEntityAsync<> causes orchestration execution to 'Complete'

[overunityx]

Description

I have an Orchestration that stops execution when CallEntityAsync<> is awaited. With breakpoints, I can see the entity is called correctly, and the entity state is valid.
I hope to get some understanding of why this is behaving this way.

Expected behavior

I expect the CallEntityAsync<> to assign a value to local variable, as follows.
bool result = await ctx.CallEntityAsync(testID, "enabled");

The 'JobManager' creates a List, then it starts sub-orchestrations and assigns an EntityID to each. Each worker is expected to be able to retrieve the state of their assigned EntityID.

Actual behavior

When the Entity reaches a return point; The Caller Orchestration immediately stops execution and moves to a 'Completed' state. All code following the CallEntityAsync<> is ignored.

Relevant source code snippets

Main Orchestration (run once)

[FunctionName("JobManager")]
public static async Task RunOrchestration([OrchestrationTrigger] IDurableOrchestrationContext ctx, [DurableClient]IDurableOrchestrationClient client, ILogger log)
{
   int maxThreads = ctx.GetInput<int>();

   // add workers
   var adds = await ctx.CallActivityAsync<List<EntityId>>("Activity_Entity", ("add", thread, $"10"));

   // Start Workers
   List<Task> wkrStartTasks = new List<Task>();
   foreach(EntityId id in adds)
   {
      Task wkr = ctx.CallSubOrchestratorAsync("ExtractWorker", $"WKR-{id.EntityKey}", id);
      wkrStartTasks.Add(wkr);
   }
   await Task.WhenAll(wkrStartTasks); // Start all Sub Orchs
}

Entity Class

[JsonObject(MemberSerialization.OptIn)]
public class Entity_Tester
{
   [JsonProperty("value")]
   public bool enlu { get; set; }

   public bool enabled() => false;

   [FunctionName(nameof(Entity_Tester))]
   public static Task Run([EntityTrigger] IDurableEntityContext ctx) => ctx.DispatchAsync<Entity_Tester>();
}

Worker Orchestration (many can exist)

[FunctionName("ExtractWorker")]
public async void RunWorker([OrchestrationTrigger] IDurableOrchestrationContext ctx, [DurableClient] IDurableEntityClient client, ILogger log)
{
   EntityId entity = ctx.GetInput<EntityId>();
   //Objects.Worker wkr = await ctx.CallEntityAsync<Objects.Worker>(entity, "enabled");

   EntityId testID = new EntityId("Entity_Tester", "tester1000");

   //EntityStateResponse<JObject> stateResponse = await client.ReadEntityStateAsync<JObject>(testID);
   bool result = await ctx.CallEntityAsync<bool>(testID, "enabled"); // Execution never continues past this point.

   if(result)
   {
      ctx.ContinueAsNew(entity);
   }
   else
   {
      await ctx.CallEntityAsync(entity, "state", "Completed");
   }
}

Known workarounds

After 2 days on this, I have not come up with any workarounds. Aside from manually saving state to a custom Storage Table, and skipping Durable Entities.

App Details

• Durable Functions extension version 2.2.0:
• Azure Functions runtime version 2.0:
• Programming language used: C#:

Screenshots

NA

If deployed to Azure

not yet. Currently testing locally with Azure Storage Emulator

• Timeframe issue observed:
• Function App name:
• Function name(s):
• Azure region:
• Orchestration instance ID(s):
• Azure storage account name:

TaskHubTables.zip

[overunityx]

Further testing seems to indicate issues around using Durable Entities in general. the following code also causes the orchestration to immediately halt with 'Completed'.
I've also published this to an azure tenant, and the problem persists.
I can write to an entity but cant seem to read.
What sort of things could cause this behavior?

var lockId = new EntityId("LockEntity", $"LOCK-{entity.EntityKey}");
using(await ctx.LockAsync(lockId)) // Breakpoint shows lockId is valid, but execution ends here.
{
   // do nothing
}

[sebastianburckhardt]

Hi @overunityx, I am trying to repro this locally. Some questions about the code you posted. Is it exactly what you were running? There seem to be some mistakes. For example,

[FunctionName("ExtractWorker")]
public async void RunWorker([OrchestrationTrigger] IDurableOrchestrationContext ctx, [DurableClient] IDurableEntityClient client, ILogger log)
{

should really be

[FunctionName("ExtractWorker")]
public static async void RunWorker([OrchestrationTrigger] IDurableOrchestrationContext ctx, [DurableClient] IDurableEntityClient client, ILogger log)
{

(it has to be a static function for the binding to work).

I have confirmed experimentally that omitting 'static' from the orchestration function really does cause the entity call to hang. That is definitely not nice. There should be an error message warning users about accidental omission of static for functions.

[ConnorMcMahon]

It sounds like we should definitely at least add this case to our analyzer.

[cgillum]

Static methods are not required for Functions generally. Instance methods were allowed starting in Functions 2.0 in order to support dependency injection. It would be good to understand why this seems to be causing problems for orchestration triggers.

[withinboredom]

I vaguely remember reading somewhere in the documentation that orchestrations must be static. I can't find it now, but I'm very positive it's in there somewhere. I think it's to avoid being able to use DI which may make execution non-deterministic.

[overunityx]

I ended up dropping this project.
Then upgraded from VS2017 > VS2019. Upgraded all DLLs.
Rewrote the project, & used static orchestrations... I never saw this particular 'bug' again.

Thx to efforts from everyone, keeping things moving forward =)

[ConnorMcMahon]

Glad to hear you were fixed @overunityx. I'm going to leave this issue open, and hopefully we can investigate it just to make sure no one else hits this issue, but I will put it at a slightly lower prioritization since you seem to be fixed in the meantime.