Azure-functions-durable-extension: Ability to run DF on-prem w/ configurable storage/queuing

[brandonh-msft]

Is your feature request related to a problem? Please describe.
We are evaluating options for a microservices architecture with Cx for handling message flow and invoking an ML model. Durable Functions would be a great fit, except a limitation from the customer is that network connectivity is not guaranteed. Therefore, the microservices must be orchestrated on-prem. Currently we are going to do this with docker-* and RabbitMQ for the messaging. Durable Functions could potentially replace all of that, but it's not an option given it can't be run on-prem with something like Redis.

Describe the solution you'd like
Ability to use durable functions with configurable storage on-prem.

Internal Tracking

CSEF 333050

[cgillum]

Would an on-premises SQL Server dependency work?

[brandonh-msft]

attn @engineersamuel

[engineersamuel]

@cgillum Yes absolutely!

[cgillum]

Great! I’m working on a proof of concept that does exactly this. I’ll update this thread when I have more to share.

[priyaananthasankar]

+1 Seeing more customers requesting this feature

[vflorusso]

+1 it is becoming a common requirement as customers need to move their serverless workload on other environments or on the edge

[brandonh-msft]

Latest Cx ask:

This project leveraged Azure SQL as its primary database. The durable orchestrations & entities were used to facilitate long running state machines that drove the shipping process for product. Ideally, we would have liked to have the data used/produced by these state machines stored alongside the rest of our data in Azure SQL. This would have allowed us to query on this information and join it to other data within the database. It could have also allowed us to serve read requests of our API using this data.

However, since Azure Storage is required and is the only option, we had to manage this data across two databases. Within the orchestrations that drove the entity operations, we implemented a replication step to replicate the entity state to our Azure SQL database at key points in the state transitions. Additionally, the solution required the cost of two storage solutions (both Azure SQL and Azure Storage).

Most importantly however, having the option to integrate global scale storage solutions such as CosmosDB would make the solution more viable for global scale systems and also be able to leverage the enhanced feature set of that solution (such as cross-region replication, partitioning, latency SLA's, etc).

I was hoping something similar to what Orleans provides here.

[mail2hjain]

+1 we are also looking forward for this feature. Kindly update the expected timelines about the same.

[guru98]

Other scenario here: think tu use durable on edge to work and interact with local iot devices and sensors

[cgillum]

Thanks for the input, everyone! We've decided to go ahead and commit to doing this work during this calendar year. In fact, I've already started prototyping and verified the feasibility of a SQL Server backend. We'll update again when we have more specific details to share.

/cc: @anthonychu

Also, if folks could tell us more about scenarios they'd like to use this for (local IoT, collocate data with existing DB, etc.), please let us know so that we can gather feedback and plan accordingly.

[anthonychu]

Should we use this issue to track the work or create a new one? Some related issues that can be addressed by a SQL backend.

• Remove Azure Storage dependency (this issue)
• More partitions, especially for Durable Entities #776
• Add more work item processing strategies than simply FIFO #1067

[cgillum]

Yeah, I could see how having a new one might make it more discoverable and better facilitate targeted goals/design discussions.