Azure-functions-durable-extension: Post orchestration lifecycle events to Azure Event Grid

[cgillum]

There are various scenarios where it would be useful to have an external system subscribe to orchestration lifecycle events. The most interesting events would probably be orchestration started and orchestration completed (success or fail).

One idea is to allow configuring an Event Grid SaS key as an application setting, and the extension could then post to the corresponding topic.

[TsuyoshiUshio]

I'd love to contribute this as a DevOps guy. :)

[cgillum]

よろしくお願いします！:)

[TsuyoshiUshio]

Hi @cgillum I have a question.
For sending event to EventGrid Topic, we can use SDK or implement REST by myself. When I see the dependency, Durable doesn't have a lot of dependency. Can I use SDK with new nuget package or prefer to keep less dependency of nuget with direct REST call?

[cgillum]

What is the nuget pack for the SDK? Is it Microsoft.Azure.Management.EventGrid
? I looked at these dependencies and I'm a little worried about them. For this particular case, I would be more comfortable with targeting the REST API.

[TsuyoshiUshio]

Sure!

[yuka1984]

I'm seeking opinions on the behavior when an Exception occurs when accessing EventGrid's API with HttpClient.

https://github.com/TsuyoshiUshio/azure-functions-durable-extension/blob/feature/lifecycledev/src/WebJobs.Extensions.DurableTask/DurableOrchestrationContext.cs#L464

[cgillum]

いい質問ですね！That is a good question. I think there are two options:

1. Ignore the exception and let the orchestration continue running.
2. Let the exception cause an orchestration failure.

Both options can have significant data-loss implications. With (1) we can lose notifications, and this can cause problems if your end-to-end application relies on these notifications. With (2) Event Grid availability problems can cause orchestrations to fail, which is another form of data-loss.

One problem right now is we don't know how to handle failures because the Azure Event Grid documentation does not have this information. I opened a documentation bug to track this: https://github.com/MicrosoftDocs/azure-docs/issues/6660.

I think the best approach would be to go with (1). However, if we see an error with Event Grid, we should retry if it is a transient error. Some errors could last a long time, so maybe we should consider a new feature for Durable Functions which allows us to "abandon" an execution if the notification fails and retry it later.

Does anyone else have opinions on this?

[cgillum]

I went ahead and merged this PR. Thanks for this great contribution! I opened up a separate GitHub issue for tracking error handling strategies: https://github.com/Azure/azure-functions-durable-extension/issues/241

[CastleArg]

Hi guys, this is great that this feature exists, I just came to request it after a lot of google fu got me nowhere.
Is it due to go G.A and be documented anytime soon? I'm a bit wary of installing r.c version.
Essentially I want to trigger functions on error AND have realtime web dashboard for some long running task status so pub sub would be ideal.
Cheers!

[akourbat]

Is this feature working? I cannot get it to work at all

[brandonh-msft]

Is this feature working? I cannot get it to work at all

Assuming you're following the docs I don't know of any reason for it not to work.

paging @TsuyoshiUshio

[cgillum]

@akourbat If you’re running into problems, please open a new issue and let us know what version of Azure Functions runtime and what version of the Durable extension you are using.

[akourbat]

I actually got it working now, but the host.json that is indicated in the docs is obsolete. Instead of

{
    "durableTask": {
        "eventGridTopicEndpoint": "https://<topic_name>.westus2-1.eventgrid.azure.net/api/events",
        "eventGridKeySettingName": "EventGridKey"
    }
}

as suggested in the docs, one should use newer version with "notifications" object-

{
"extensions": {
  "durableTask": {
    "hubName": "MyTaskHub",
    "notifications": {
      "eventGrid": {
        "topicEndpoint": "https://topic_name.westus2-1.eventgrid.azure.net/api/events",
        "keySettingName": "EventGridKey",
        "publishRetryCount": 3,
        "publishRetryInterval": "00:00:30",
        "publishEventTypes": [
          "Started",
          "Pending",
          "Failed",
          "Terminated"
        ]
      }
   }
}

I also think "Failed" is not being published.
CORRECTION: "Failed" is also working

[CastleArg]

the docs still contain open feedback that the feature does not work when set up as per the docs.
@akourbat or anyone who has actually successfully used this it would be great if you could make a pull request to the docs with updated info once you have it running!