Azure-functions-durable-extension: Abnormal request count to Azure Queue when the host Starts

[shibayan]

Use Durable Functions, There are unusually many specific requests to Azure Queue when the host starts up. The same behavior at HttpTrigger sample.

I think the DurableTask's BackoffPollingHelper is executing GET requests to queue with no weights in 5 seconds.

[cgillum]

There is currently no waiting for the first 500 milliseconds. The code is here: https://github.com/Azure/durabletask/blob/f9cc450539b5e37c97c19ae393d5bb1564fda7a8/src/DurableTask.AzureStorage/Messaging/TaskHubQueue.cs#L52-L53

Recently people have been complaining about too many storage requests. I will change this behavior to reduce the number of GetMessagesRead and GetMessageRead operations.

[shibayan]

Thanks. sleepToWaitRatio is specified as 0.1 by default, it is actually 5 seconds no waiting?

https://github.com/Azure/durabletask/blob/f9cc450539b5e37c97c19ae393d5bb1564fda7a8/src/DurableTask.AzureStorage/BackoffPollingHelper.cs#L35

https://github.com/Azure/durabletask/blob/f9cc450539b5e37c97c19ae393d5bb1564fda7a8/src/DurableTask.AzureStorage/BackoffPollingHelper.cs#L77-L82

[cgillum]

Oh, you are right! Thanks for catching that!

[dsm0880]

Yeah the sooner the better, we are actually getting charged not insignificant amounts on our queue transactions. We are ~120k transactions a minute and I'm worried the architecture wont scale. Hope this fixes it!

[anhhnguyen206]

We're having this issues as well. We heavily use DurableTask in our application and the number of requests is abnormally high and this in turn cost us more money for using Storage Account v2. Looking forward to the fix.

[cgillum]

This fix will go out with the next release. I expect it will result in a significant reduction in queue transactions.

[ielcoro]

We are seeing a variation of this issue, however it doesn't stop after initialization, we are seeing 6000-7000 request to queue service per minute:

This is costing us aprroximately 6-7$ per day (with four function apps), in a month we estimate a cost around 217$, wich defeats the purpose of using functions at all.

UPDATE

Just tried with version 1.6.3, the fix reduces slightly the number of request, I still don't understand what is going on to reach 6000 request on queue service every minute or so... We estimate a bill of 126$ with this version for keeping track of one orchestrator function per function app that runs at two minute interval:

As this is totally insane, we are replacing durable functions with a custom solution based on service bus until we see a significant change in reducing operations against storage service.

[dsm0880]

@ielcoro One thought just so you don't lose all hope. You could try General Purpose Storage v1 as oppose to V2. I haven't tried it myself but looking at the billing the Queue is what is causing the high cost and v1 general purpose storage is 1/10th the cost. so $126 would be $12.60. With that being said @cgillum this is seriously a critical issue in my mind so anything you can do to push this fix would be greatly appreciated. We are now at one month since the initial report and your users are incurring significant costs.

[ielcoro]

Thanks @dsm0880

However we are on production now and we cannot bet on this being fixed, or that it will not increase the usage in the future with some minor change to our orchestration or activity functions. So we need a stable solution that gives us predictable usage costs.

What we are doing is really easy to replicate with just a scheduled message to service bus, we just have one orchestration function with two activities that runs as new at 2 minute intervals, seems to us that per minute request number to the storage queue should be infinitely lower, 10 requests per minute? 100? But several thousand?

Something looks very wrong, and 1.6.3 does not fix it. We hope it's fixed soon, and will re-evaluate the framework then, but until this is fixed durable functions is a no go for our customers and short term projects.

[cgillum]

@ielcoro Just to clarify, your said 1.6.3 does not fix this, but how are you running 1.6.3 when we haven't released it yet? Are you using your own custom builds?

Also, can you share with me an instance ID and Azure region for one of your 2 minute interval orchestrations? I'd like to understand your situation better.

[ielcoro]

Yes @cgillum, before we were experiencing this issue in 1.6.2, so after seeing your PR and comment here, we downloaded and built the 1.6.3 branch, published as nuget package in our Azure DevOps private repo, and published our app with the new dependency. The logs confirm we are running with the 1.6.3 version of the durable extension, without significant change in behavior.

If I understand by Instance Id, you refere to a function execution ID, if you need any other info, please send me how to contact you via private channel.

• Region: West Europe
• Function Execution ID: f7f4c9a90f35468b87c1838c50f98c46

Be advised, we plan to remove this environment as soon as posible (the costs are too high for our client at this rate).

Thanks @cgillum for your time and help.

[cgillum]

Thanks for the info. I believe the changes you want are not yet part of the 1.6.3 payload. They're actually still sitting in the Durable Task Framework repo here: https://github.com/Azure/durabletask/pull/217. Once we get around to merging that, we'll be able to update the Microsoft.Azure.DurableTask.AzureStorage dependency to include these fixes.

Apologies for the long delay on this. It's definitely a top priority for us - just a matter finding the time given constrained resources on the team.

[ielcoro]

Ok, good news @cgillum

The fix does indeed solve the problem, here is the metric graph that shows requests before uploading the new version and after:

Now the usage makes more sense, staying at 266-300 requests per minute, it's not going to be free either, but we estimate 360,000 requests per day, at 0.0034$ per 10,000 request, the cost for a full day run will be: 0.1224$, that totals to 3,79$ per 31 day month, that cost can be justified. I wonder if still room for improvement, it was unexpected, at least for us, to incur in storage costs after selecting durable functions as our developing framework.

As you said the fix was not merged into durable task library, https://github.com/Azure/durabletask/pull/217, so we cloned the branch behind that pull request (v.1.6.3), changed the versions of Microsoft.Azure.DurableTask.AzureStorage, Microsoft.Azure.DurableTask.Core and WebJobs.Extensions.DurableTask, built and published packages in our private repo, and published the new function apps with those dependencies.

Doing this has bought us time to wait for the official release.

[cgillum]

Thank you @ielcoro for going out of your way to do this validation! I'm pleased to see the usage go down to acceptable levels. This gives me confidence that we'll be able to move forward with this very important release more quickly.

We will continue to look for ways in which we can further reduce Azure Storage costs while at the same time maintaining a responsive framework. Apologies for unpleasant surprises here.

[cgillum]

This is fixed in the upcoming release.