Azure-functions-durable-extension: Orchestration instance stuck at Pending status

[umamimolecule]

Description

We have a few HTTP-triggered Azure Durable Functions that are failing to start orchestration instances. Today we had 7 in a row fail to start, spaced 3 hours apart. Over the last 5 days we observed 45 instances that are stuck in the Pending state.

Currently we are only seeing this behaviour in the Australia Southeast region, our other production functions in West US and North Europe are functioning as expected.

Expected behavior

Orchestration instances should start.

Actual behavior

Orchestrations are stuck in Pending state and do not execute any activities. Querying the status for the stuck orchestration instances shows something like the following:

{
    "instanceId": "d1f0757635634486b5873038e9ee1f80", 
    "createdTime": "2020-10-05T19:00:58.3516335Z", 
    "runtimeStatus": "Pending", 
    "name": "CreateTeamOrchestrator", 
    "lastUpdatedTime": "2020-10-05T19:00:58.527866Z"
}

Known workarounds

None that we know of.

App Details

• Durable Functions extension version (e.g. v1.8.3): 1.8.1
• Azure Functions runtime version (1.0 or 2.0): 3
• Programming language used: C#

Screenshots

Application Insights shows the following pattern where our HTTP triggered function is called, successfully calls StartNewAsync on the orchestration client, but no further activity from the orchestration is logged:

• Timeframe issue observed: From 2020-10-01 onwards, but most appear to be within the last 24 hours (2020-10-05 to 2020-10-06 UTC)
• Function App name: ncpprd01tmsfncaus01
• Function name(s): CreateTeamOrchestrator
• Azure region: Australia Southeast
• Orchestration instance ID(s): 5dbb9b9890a14107b886922a17d7f5c7, c93b847cf67c4890aacf308c5f4f21d1, 801f0f844c424c7697bc7896d465fdca, d1f0757635634486b5873038e9ee1f80, af6e72cfe7f74ed0bad6055dcddeb298
• Azure storage account name: ncpprd01tmsstraus02

[olitomlinson]

@umamimolecule

You are running a very old version of D.F in production, there have been many patch updates since then that target these kind of scenarios.

I strongly recommend updating to the latest 1.x, which I think might be 1.8.6. Even better, get onto 2.x If you can 😊

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.