Azure-functions-durable-extension: Getting 409 conflicts in calls to Azure storage

[jeffputz]

With the 1.3.1rc update, I'm getting 409's from Azure storage when the orchestration service tries to call it:

System.Exception: [demo]  ---> Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (409) Conflict. ---> System.Net.WebException: The remote server returned an error: (409) Conflict.
   at Microsoft.WindowsAzure.Storage.Table.Protocol.TableOperationHttpResponseParsers.TableOperationPreProcess(TableResult result, TableOperation operation, HttpWebResponse resp, Exception ex)
   at Microsoft.WindowsAzure.Storage.Table.TableOperation.<>c__DisplayClass4.<InsertImpl>b__2(RESTCommand`1 cmd, HttpWebResponse resp, Exception ex, OperationContext ctx)
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result)
   at Microsoft.WindowsAzure.Storage.Table.CloudTable.EndExecute(IAsyncResult asyncResult)
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.<SetNewExecutionAsync>d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.<SendTaskOrchestrationMessageAsync>d__90.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DurableTask.Core.TaskHubClient.<InternalCreateOrchestrationInstanceWithRaisedEventAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.DurableOrchestrationClient.<StartNewAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at AMS.Functions.HttpBackgroundTaskDispatcher.<Run>d__0.MoveNext()
   --- End of inner exception stack trace ---

Interestingly, the function still runs. In this case, it's generating double exceptions. First, the one above, and second, because it causes a 400 when I call the Http triggered endpoint of the function.

Also, in one environment (Azure), clearing the various durable functions blobs and tables resolved the issue. Locally, with the emulator, it's not resolving it.

[cgillum]

Interesting. I was also seeing occasional 409 conflict failures in our CI, but it was not consistent. I'll try to see if I can repro locally and root cause. Let me know if you happen to notice a pattern for when it happens.

[jeffputz]

It's definitely not just my local environment, or just http triggered functions. Are you using Insert or InsertOrReplace to write? I'm assuming it's to table storage.

[cgillum]

Ah, it looks like we are doing a TableOperation.Insert in this code path. I imagine this would break the singleton pattern if a singleton is being recreated. Could that be what you are running into? I suspect it should be safe for us to internally do a TableOperation.InsertOrReplace instead.

FYI @gled4er

[jeffputz]

Yes... they are implementing the singleton pattern. That's the key reason
we use this framework, so we don't have to try and manage all that state
ourselves.

Is this fast-fixable the way yesterday's problem was? :) Our sprint ends today.

[gled4er]

Hello @jeffputz,

We will fix the issue as soon as possible.

Please follow this thread for the bug-fix release.

Thank you!

[cgillum]

Sorry for the delay, and for the regression. I've added several new tests that cover the singleton scenario to avoid regressions like this in the future.

Here is the updated version of DurableTask.AzureStorage which has the fix: https://www.nuget.org/packages/DurableTask.AzureStorage/1.1.8-beta.

We'll subsequently update Microsoft.Azure.WebJobs.Extensions.DurableTask to include a direct dependency on this version. Until then, you can explicitly add the newer DurableTask.AzureStorage dependency to your project to get the fix.

[cgillum]

The updated nuget dependency with the fix is automatically included in the latest release (v1.4.0).