Azure-functions-host: Unexpected delay before message processing in EventGridTrigger

[oleksandr-slys]

Is your question related to a specific version? If so, please specify:

<TargetFramework>netcoreapp3.1</TargetFramework>
<AzureFunctionsVersion>v3</AzureFunctionsVersion>
<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" Version="2.1.0" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" />

What language does your question apply to? (e.g. C#, JavaScript, Java, All)

C#

Question

I see that the code execution time inside the Azure Function is 2s, but event handler shows that the total time is a few minutes
Why do some messages from EventGrid are waiting so long before start actual processing?

public class Handler
{
    [FunctionName("Handler")]
    [Singleton]
    public async Task Run([EventGridTrigger] EventGridEvent eventGridEvent, ILogger log)
    {
        log.LogInformation("Event:");
    }
}

[Hjortsberg]

Hi,
We experience the same behaviour on our functions.
About 0.48 percent of events, gets a delayed delivery of exactly 3 minutes.
We thought we had a computationally slow function and in order to diagnose the behaviour, we created a new function in a separate Resource group, separate Application Insights to log to and a Windows Consumption AppService Plan.
However, this "logging function" shows the same behaviour.

Common between the functions

• Both on windows function plans, one on elastic consumption and one on standard consumption.
• Both subscribed to the same Event Grid Domain Topic, with a function trigger. I.E, Event Grid Trigger.
• Delays happen at the same time for both functions sometimes.

The new "logging function" contains nothing except the following code:
``` C#

r "Microsoft.Azure.EventGrid"

using Microsoft.Azure.EventGrid.Models;

public static void Run(EventGridEvent eventGridEvent, ILogger log)
{
log.LogInformation(eventGridEvent.Data.ToString());
}
```

The service that publishes the events of the topic should not be the culprit according to these Azure EventGrid metrics where you can see the publisher has a constant publishing latency and the destination functions fluctuate wildly. The functions get triggered approx 50000-100000 times a day.

The weird thing is that the slow function shows the same behaviour as for the author. Here is an example of an function execution that takes 10 seconds, all dependency calls are intra-azure within the same data center location.

I've also found this in the docs: Event Grid automatically adjusts the rate at which events are delivered to a function triggered by an Event Grid event based on the perceived rate at which the function can process events. This rate match feature averts delivery errors that stem from the inability of a function to process events as the function’s event processing rate can vary over time..
Is this what we're experiencing?
Any ideas are greatly appreciated.