Azure-functions-durable-extension: [Feature] Ability to set Custom Status via DurableActivityContext

[brandonh-msft]

Some users have expressed desire to be able to set the Custom Status property on an orchestration from activity functions. Filing.

[cgillum]

What is the source of this desire? Why is setting it in the orchestrator not good enough? Said another way, what is it users want to accomplish that cannot be done in the current model?

[brandonh-msft]

In general, I’m having a really hard time understand how to monitor the execution of these orchestrations, especially since we call other orchestrations which call other activities. It’s not difficult to track just the top level orchestration using App Insights or Storage Explorer, but I haven’t figured out how to trace execution through all of the sub-functions. Is that possible? If A calls B which calls C, to have a list of all messages from A, B, and C?

I suggested to them using CustomStatus and/or TelemetryClient. Response:

We looked into using CustomStatus, but the API isn’t available from activities where most of the work is done, only at the orchestration level. So it appears to be more of a high-level status rather than a logging API.

And yes, we’re going to be adding logging, but I was looking to see what tools came out of the box to track an orchestration workflow.

hence this request. hope that helps.

[cgillum]

I understand now. I think the real ask is how to do end-to-end tracing. My suggestion is to use ILogger and the orchestration instance ID to correlate logs.

Updating the custom status from activity functions won't really help with end-to-end tracing since it's intended for high-level status and not tracing.

[sehcheese]

@cgillum With a longer-running activity function, it would be helpful to be able to report a more granular status for the GetStatus API to show to the user. Not for error tracing, but for information on the progress of the task. Does that make sense?

[cgillum]

I agree this would be generally useful. I'll reopen this to track.

If I were to design such an API, it would look something like DurableActivityContext.SetCustomStatusAsync(object status). Internally it would call a yet-to-be created API in the Durable Task Framework which directly updates the orchestration's custom status in the tracking store.

The downside of something like this is that you now could have multiple clients racing with each other to update the custom status (today no such race exists). It would therefore need to be used with caution.

[petr-stupka]

@cgillum absolutely useful function. However as you mentioned link it directly to CustomStatus may lead to racing conditions.

As we are running multiple I/O operations in parallel, i would suggest to have it implemented in different way so we can consolidate those outputs in orchestrator and then calculate the CustomStatus.

I’m using entity for the values, but not sure if this is the best way as the entity calls are very rapid (have no possibility to throttle the occurrence)

[srikanthnd]

I recently came across a situation where this feature will be very helpful and indeed having to set custom status from activity function makes a lot of sense.

Also, currently, every time the custom status is set, it over writes the previous value. I would prefer to have an append functionality to the custom status as well instead of overwriting the previous value OR is there a specific reason why the custom status was designed to have a single value?

[cgillum]

Also, currently, every time the custom status is set, it over writes the previous value. I would prefer to have an append functionality to the custom status as well instead of overwriting the previous value OR is there a specific reason why the custom status was designed to have a single value?

Custom status can be any object type that fits within the 16 KB limit. If your payload size is limited, you can simply use an ordered list as your custom status object.

If you need larger payloads or if you need activities to be able to save status, then another workaround would be to store these logs in a Durable Entity. That would also solve the race condition problem since entities are single-threaded, and may actually be a more appropriate long-term solution for this scenario.

[KavithaKommuri]

@cgillum - We have a similar situation as others where setting the custom status on activity function as our activity function is a long-running process and it would be really helpful for the users to know the progress of that process. Is there any way to achieve this?

[NickSevens]

@cgillum would there be a way in that scenario to retrieve and update the Entity from within an Activity?

[cgillum]

@NickSevens and @KavithaKommuri yes, definitely. Here is one potential solution that I think should work:

1. Use a durable entity to store custom status - there would be one entity per orchestration. The ID of the entity can be derived from the orchestration instance ID for simplicity.
2. use the [DurableClient] binding and SignalEntityAsync() within an activity function to send "update" operations to the appropriate status entity.
3. If the activity needs to read the status, it can similarly use the ReadEntityStateAsync() method via the [DurableClient] binding.
4. Use the orchestration's custom status field to publish the name/location of the status entity so that external clients can also find it and query it.