Azure-functions-host: Azure Functions runtime slow since 2.0.12408.0

[adobedoe]

Describe the bug
I have around 15k orchestration instances running daily with sub-orchestrators and many activity functions that was running very fast on sdkVersion=2.0.12382. My function host had FUNCTIONS_EXTENSION_VERSION=~2 so it automatically took the latest version of the runtime.

I can see in App Insights that the execution time increased significantly when the runtime went to 2.0.12427 and is still slow on the latest version 2.0.12438

Investigative information

• Durable Functions extension version: > 2.0.12382
• Function App version (1.0 or 2.0): 2.0
• Programming language used: C#

If deployed to Azure

• Timeframe issue observed: From 19th April to today
• Function App name: fnPP-Processor
• Function name(s): Orchestrators_ProcessEventPayload <-- entry point
• Region: UK South
• Orchestration instance ID(s): f2b6f571-3fc1-4b1a-9efd-2d00789fe6f7 (slow), cc1d0173-0391-4da9-96a0-3418c97bc2d3 (fast)

I have seen that the control queues grow very large with the later version of the runtime and are not evenly distributed. They also take a very long time to clear. This does not happen with the earlier version of the runtime.

[cgillum]

In both cases, you were using the same version of Durable Functions, right? So this is more likely a regression in the Azure Functions runtime and not Durable Functions itself?

[adobedoe]

That's correct. Same version of Durable Functions and same codebase. The only thing different is the FUNCTIONS_EXTENSION_VERSION so it does point to an issue with the runtime

[olitomlinson]

I too am currently trying to get to the bottom of what appears to be a performance regression with my DF, But I’ve not yet fully ruled out regressions in my own dependencies to say for sure that orchestration performance has regressed.

I’m on ~2 , UK South.

I have also observed very unbalanced control queues. Thousands on 3-4 queues and nothing on the remaining 12-13 queues, despite all my orchestration instances executing exactly the same code-path and hitting the same dependant external services in the activities. I’m using a random GUID as the Orchestration ID so this should be giving a uniform distribution of load across the control queues.

[olitomlinson]

My findings corroborate with the slow down you are experiencing. https://github.com/Azure/azure-functions-durable-extension/issues/779#issuecomment-496657372

2.0.12408 is the last known good version for me.

[adobedoe]

Just a quick update. Since forcing my function app to use version 2.0.12382 (for nearly 2 weeks) I have not experienced the slow down that I was seeing every day with the later runtime versions.

[cgillum]

@adobedoe, @olitomlinson the Functions runtime team informed me that some Application Insights changes were made which may impact the performance of HTTP triggers and potentially outbound HTTP calls as well. More details here: https://github.com/Azure/azure-functions-host/releases/tag/v2.0.12408

Can you try the suggested workaround, which is to disable the new tracking functionality and see if that improves your performance?

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "httpAutoCollectionOptions": {
        "enableHttpTriggerExtendedInfoCollection" : false
      }
    }
  }
}

[olitomlinson]

@cgillum changing the setting as you prescribed has made a huge difference!

Posted the results here : https://github.com/Azure/azure-functions-durable-extension/issues/779#issuecomment-499651847

[cgillum]

@olitomlinson thanks for confirming!

@adobedoe are you able to confirm as well?

@brettsam FYI, I wonder if we need to reconsider the defaults here if it's having such a noticeable impact on performance.

[olitomlinson]

@cgillum you’re very welcome, and thank you for getting to the bottom of it.

It was a very nervous time for me because our use-case relies on predictable throughput so we can plan to meet our Customer SLAs.

If this Runtime upgrade turns out to be the root cause of the regression, will there be actions to understand how DF can better protect itself from regressions in the Host Runtime?

I’m not against fixing the version of the Host Runtime I integrate against, and I should probably do that as my own act of diligence - lesson learned for me.

But I believe in aspirations, so I would like to not fix the Runtime version, but instead have confidence that a minor/patch runtime upgrade would not affect performance as drastically as it did in this example.

[adobedoe]

@cgillum - I've updated the host.json as described and using the latest runtime (2.0.12493), the throughput is much faster so I'm pretty confident that this is the cause.

For me, this caused a huge degradation in performance which probably would've prevented me from using DF if this was a PoC and not something I'd had running in production for a while so I think it would be good to default the enableHttpTriggerExtendedInfoCollection to false in the runtime at least in the short term.

Thanks for resolving this issue.

[cgillum]

@adobedoe thanks for confirming. I agree that this is a pretty serious problem. Unfortunately there is nothing Durable Functions can do to make it better since Durable Functions is just an extension. I'm going to transfer this issue to the Azure Functions team for a resolution.

[brettsam]

@lmolkova -- is there anything tracking this issue on the App Insights side?

[lmolkova]

@brettsam we are tracking this in internal VSO backlog. I also created github issue https://github.com/microsoft/ApplicationInsights-aspnetcore/issues/900

[brettsam]

Has anyone else hit this recently I'm trying to track down repros so we can check our mitigations. Thanks in advance.

[ericsampson]

I think this could be closed now as it was resolved AFAIK?

[brettsam]

Agreed.