Azure-functions-durable-extension: External event message loss due to async activity in orchestration

[Neil-p-Hughes]

It seems message loss is not only due to continue as new, but happens if any await activity happens inside the orchestration.

the following code should return the alphabet, however it instead returns only part of it.
you can see in the history log that all letters where received however a good number of them are dropped.

if we remove the activity call from the orchestration, and perform the append operation in the orchestration, I observe no message loss.

I have ran this test using version 1.6.2

public static class TestMessagePassing
{
    [FunctionName(nameof(TestMessagePassing) + nameof(OrchestrationAppend))]
    public static async Task OrchestrationAppend([OrchestrationTrigger] DurableOrchestrationContext context)
    {
        string state = "";
        for (int i = 0; i < 100; i++)
        {
            context.SetCustomStatus($"Awaiting append {state}");
            var append = await context.WaitForExternalEvent<string>("append");
            state += await context.CallActivityAsync<string>(nameof(TestMessagePassing) + nameof(ActivityTriggerAppend), append);
        }
    }

    [FunctionName(nameof(TestMessagePassing) + nameof(ActivityTriggerAppend))]
    public async static Task<string> ActivityTriggerAppend([ActivityTrigger] string append)
    {
        await Task.Delay(50);
        return append;
    }

    [FunctionName(nameof(TestMessagePassing) + nameof(Starter))]
    public static async Task<IActionResult> Starter(
        [HttpTrigger(AuthorizationLevel.Function, methods: "get")] HttpRequest req, ILogger log,
        [OrchestrationClient] DurableOrchestrationClient starter)
    {
        var instanceId = await starter.StartNewAsync(nameof(TestMessagePassing) + nameof(OrchestrationAppend), null);
        log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
        var status = await starter.GetStatusAsync(instanceId);
        while (status?.CustomStatus?.ToString().StartsWith("Awaiting append") != true)
        {
            await Task.Delay(200);
            status = await starter.GetStatusAsync(instanceId);
        }
        var abcs = new List<string>() { "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"};
        foreach (var letter in abcs)
        {
            await starter.RaiseEventAsync(instanceId, "append", letter);
        }

        await Task.Delay(40000);
        status = await starter.GetStatusAsync(instanceId);
        return new OkObjectResult(status?.CustomStatus?.ToString());
        // we should see the alphabet but due to message loss, we have less letters
        //actual response: Awaiting append adgjnrvy
    }
}

I feel this issue relates to #207 , and generally with the continue as new bug. #67

[cgillum]

Yes, your assessment is correct. If the orchestration receives an external event and does something other than consuming it (using WaitForExternalEventAsync) that message will be dropped. In your case, instead of receiving subsequent letters, the orchestration schedules an activity call, so any pending external events will be dropped at that point.

[cgillum]

Resolved in v1.8.0 release.