Azure-functions-durable-extension: What is the expected behavior if we are waiting multiple times on the same event name, but with a different shape of data?

[SimonLuckenuik]

Ref: Handling external events in Durable Functions (Azure Functions)

Different scenarios we are looking at for multiple systems synchronization. I know there would be workarounds to prevent receiving the same event name multiple times, but my goal here is to clarify the documentation, so let's assume we cannot change the behavior of the external system. Based on the current doc, I am assuming that both examples below are currently supported. Let's assume for both examples that events are raised based on QueueTrigger and that ordering is not guaranteed.

Example 1: Wait until a processing is completed or until we receive 3 errors. Number of events is known in advance, but Error value will always be different (or the same). Example of events received: _Error_ event: 503, 503, 404. _Completed_ Event: never received.

public static async Task Run([OrchestrationTrigger] DurableOrchestrationContext context)
{
    var completed = context.WaitForExternalEvent<int>("Completed");
    var error1 = context.WaitForExternalEvent<int>("Error");
    var error2 = context.WaitForExternalEvent<int>("Error");
    var error3 = context.WaitForExternalEvent<int>("Error");
    var error = Task.WhenAll(error1, error2, error3);
    var winner = await Task.Any(completed, error);
}

Example 2: Receive multiple events providing progress on an external system, reflect external system status in the current system status. Example of events received: _Progress_ event: 0, 5, 10, 30, 70, 85, 90, 100.

public static async Task Run([OrchestrationTrigger] DurableOrchestrationContext context)
{
    float progress = 0;
    do
    {
      progress = await context.WaitForExternalEvent<int>("Progress");
      // Reflect external workflow status in the current workflow status.
    }
    while(progress < 100);
}

Thank you!

[cgillum]

Thanks for posting. I expect Example 2 to succeed but Example 1 to fail. Example 2 was enabled as part of the Beta 3 release (see https://github.com/Azure/azure-functions-durable-extension/issues/141). This fix allowed waiting for events with the same name sequentially (which is basically your example 2). However, the fix would not enable listening to events with the same name in parallel. A new fix will be needed to support this.

[SimonLuckenuik]

Thanks @cgillum, I was expecting the opposite to be supported, since the first example seems to be more predictable.

With Example 2 on Beta 3 release, I guess that if events are received within a very short timeframe (before the orchestrator executes), the first event would be lost? (based on #29)

It could be interesting to have an Orchestrator level Queue concept that would enable example 1 and help to support any scenarios with external triggers that need to interact with the workflow. For example, a Workflow that reacts to QueueTriggers containing messages coming from external systems.

[cgillum]

Hey @SimonLuckenuik support for both these patterns will be ready in the next release. I'm going to go ahead and close this issue for tracking purposes, though I'm happy to keep the conversation going.

Regarding this:

With Example 2 on Beta 3 release, I guess that if events are received within a very short timeframe (before the orchestrator executes), the first event would be lost? (based on #29)

This is not expected. At this point in time, the only time you should see a loss of events is if the orchestration is in the ContinuedAsNew state.

It could be interesting to have an Orchestrator level Queue concept that would enable example 1 and help to support any scenarios with external triggers that need to interact with the workflow. For example, a Workflow that reacts to QueueTriggers containing messages coming from external systems.

I don't think this would actually be necessary. As it stands each orchestration instance is fed by exactly one queue (though that queue will be shared with potentially many instances) and using external triggers should not be an issue. The main scenarios that we need to spend more time on are:

1. How to deal with the ContinueAsNew race condition (this would require changes to the Durable Task Framework)
2. Investigate a way to support raising an event to a non-existent orchestration instance in such a way that is safe for concurrency and ensures the instance will be created (exactly once) if it doesn't already exist.

Let me know if you have other thoughts on this.