Azure-functions-durable-extension: Excessive storage cost running durable functions

[vincentvanderwalt]

I'm running a IOT based durable function application which runs off a storage eventgrid (http trigger)

I'm using a dedicated storage account for the durable function by using the AzureStorageConnectionStringName host setting.

After running the process for not even a day we accrued a bill on the storage account for close to £100 for 3 functions apps.

going at that rate we'll be paying over £3500 a month just on durable function storage.

That doesn't sound right to me since the running of the functions will be around £500 for the month.

is there some way of reducing that cost or reducing the IO hits ?

[vincentvanderwalt]

Doing some investigation about the costs and it looks like the following operation is causing the huge costs.

On 1 storage account total cost £56

Class 2 Operations - Queues v2 - £43.62
LRS Class 1 Operations - Queue.... - £11.49

[cgillum]

This is a known issue, currently being tracked here: https://github.com/Azure/azure-functions-durable-extension/issues/462. Basically we're polling the queues much harder than necessary and this is what seems to be impacting the bill.

This will be fixed in the next release, and recent testing shows that the I/O against queues will be dramatically reduced, which should similarly reduce the bill. Apologies about the costs you have incurred!

[cgillum]

This is fixed in the upcoming release.

[adavir]

This is still an issue for me, I have a simple orchestration

It gets called every 10 seconds, saves a record to DB then sends message to an event grid. Works a charm, the only issue my bill was 500 bucks for 1 month. 60% or this attributed to the function backing storage account. How can i reduce this cost?

            [FunctionName("orch-productionlog-save")]
            public static async Task<bool> Save(
                [OrchestrationTrigger] DurableOrchestrationContextBase context, ILogger log)
            {
                var input = context.GetInput<ProductionLog>();
                var retryOptions = new RetryOptions(TimeSpan.FromSeconds(5), 10);

                try
                {
                    await context.CallActivityWithRetryAsync("saveProductionLog", retryOptions, input);

                    // Now that the log is confirmed to be saved the next step is to send the event 
                    // to our event grid topic
                    await context.CallActivityWithRetryAsync("postToEventGrid", retryOptions, new EventGridInstruction("production-log-saved", "productionLog", "created", input));

                }
                catch (Exception ex)
                {

                }


                return true;
           }

[SergiySeletsky]

The same issue for me, storage has a lot of Blob and Queue IO even when no triggers running.
We not using blobs/queues in the function itself, so the durable function generates it.
every hour about 16К blob transactions!!!

[cgillum]

How many partitions do you have configured? If your app does not require multiple VM instances and you reduce the number of partitions, then you can significantly reduce the number of transactions.

More info here: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-billing#azure-storage-transactions

[SergiySeletsky]

I use default settings - I hope it is 4.
Also it is running on consumption plan.
I changed default 30s to "maxQueuePollingInterval": "00:04:30" and it helps to reduce load from 16К to 2K per hour. But still a lot. All calls now related to BlobLease and happen every 10 seconds. This setting is hidden inside durable task host source, so I can not change this 10s. blob lease calls.