Azure-functions-durable-extension: Can't combine DurableContext + DurableClient in the same function

[aaronpowell]

_Disclaimer: I may be approaching this incorrectly so it might be partially a design issue in my solution but I'm unsure how to do it otherwise._

I'm experimenting with using Orchestrators + Entities to create a cache of HTTP functions, here's a basic example:

    public interface ICounter
    {
        void Add(int count);
    }

    public class Counter : ICounter
    {
        public int Count { get; set; }

        public void Add(int count) => Count += count;

        [FunctionName(nameof(Counter))]
        public Task Run([EntityTrigger] IDurableEntityContext client)
        {
            return client.DispatchAsync<Counter>();
        }
    }

    public static class HttpCachingFunction
    {
        [FunctionName(nameof(HttpCachingFunction))]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = "count")] HttpRequest req,
            [DurableClient] IDurableClient client)
        {
            var entityId = new EntityId("Background", "Cache");
            var state = await client.ReadEntityStateAsync<Counter>(entityId);

            if (state.EntityExists)
                return new OkObjectResult(state.EntityState.Count);

            // Generate some cache from the backing store
            var cachedData = 1;
            // Hand the data over to be cached
            await client.StartNewAsync(nameof(CacheOrchestrator), cachedData);

            return new OkObjectResult(cachedData);
        }

        [FunctionName(nameof(CacheOrchestrator))]
        public static async Task CacheOrchestrator(
            [OrchestrationTrigger] IDurableOrchestrationContext ctx,
            [DurableClient] IDurableClient client,
            ILogger logger
            )
        {
            logger.LogInformation("Starting background job");

            // Get the data to be cached
            var count = ctx.GetInput<int>();

            var entityId = new EntityId("Background", "Cache");

            // Cache on entity
            await client.SignalEntityAsync<ICounter>(entityId, proxy => proxy.Add(count));

            // Start a timeout
            var timer = ctx.CreateTimer(ctx.CurrentUtcDateTime.AddMinutes(1), CancellationToken.None);

            await timer;
            logger.LogInformation("Orchestrator shutting down");

            // Remove entity now that the cache has "expired"
            await client.TerminateAsync(entityId.ToString(), "Cache Cleaning");
        }
    }

Essentially what happens is:

• Request comes in
• Check if there's an entity, if there is, return it
• If there isn't, do _something_ to get some data and hand it to an orchestrator
• Orchestrator takes the data, gives it to an Entity and then starts a timeout for the cache
• Once the timeout has expired we terminate the Entity so it gets removed <-- this is what I'm not sure about, but I can't figure out another way to "kill" an Entity

This code explodes because the IDurableOrchestrationContext and IDurableClient run their stuff on different threads, which it doesn't like (it goes 💥 on this line).

I can rework the code a bit to use the IDurableOrchestrationContext to signal the Entity (although that doesn't support the proxy-method approach, may be worth opening a separate bug for that), but I still can't work out a way to call TerminateAsync without using IDurableClient.

So, how should I be going about solving this problem?

[cgillum]

Adding @sebastianburckhardt

[sebastianburckhardt]

There are two issues here:

• DurableClient should not be used from within orchestrations. All entity functionality we support from within orchestrations is already in IDurableOrchestrationContext.
• TerminateAsync is not a safe way to clear an entity.

We should probably clarify these points (provide better error messages and documentation).

The correct way to clear an entity is to add a clear operation to its interface and implementation:

public void Clear() { Entity.Current.DestructOnExit(); }

You can then call or signal this operation from the orchestration.

Soon we will have an even better way to do this pattern - by supporting timers inside entities (#716). Then you won't need the orchestration at all. Your caching pattern could actually be a very nice sample for this feature.

[ElanHasson]

I hit the DurableClient issue and already documented it last week via https://github.com/MicrosoftDocs/azure-docs/pull/38208.

[aaronpowell]

@sebastianburckhardt ah ok, I had seen the DestructOnExit method on the Entity but I assumed that there this was to mark the Entity as "destructable" and there was something else you needed to do to trigger an OnExit event.

I'm assuming that the method name isn't important, it's not looking for Clear or anything specific?

[sebastianburckhardt]

Yes, the method name is irrelevant. DestructOnExit can be called in any operation and will delete the entity right after the operation completes. We should perhaps rename it to something more descriptive, e.g. DeleteEntityAfterThisOperation.

[aaronpowell]

I think it's more a documentation issue, but given that it's preview there isn't much documentation so I'm learning by experimenting and digging through the source. Reading more closely and having your explanation makes it clear what the method intent is.

[aaronpowell]

FYI - here's a working sample of a cache:

    public interface ICounter
    {
        void Add(int count);
        void Clear();
    }

    [JsonObject(MemberSerialization = MemberSerialization.OptIn)]
    public class Counter : ICounter
    {
        [JsonProperty]
        public int Count { get; set; }

        public void Add(int count) => Count += count;
        public void Clear() => Entity.Current.DestructOnExit();

        [FunctionName(nameof(Counter))]
        public Task Run([EntityTrigger] IDurableEntityContext ctx) => ctx.DispatchAsync<Counter>();
    }

    public static class HttpCachingFunction
    {
        [FunctionName(nameof(HttpCachingFunction))]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = "count")] HttpRequest req,
            [DurableClient] IDurableClient client)
        {
            var entityId = new EntityId(nameof(Counter), "Cache");
            var state = await client.ReadEntityStateAsync<Counter>(entityId);

            if (state.EntityExists)
                return new OkObjectResult(state.EntityState.Count);

            // Generate some cache from the backing store
            var cachedData = 1;
            // Hand the data over to be cached
            await client.StartNewAsync(nameof(CacheOrchestrator), (cachedData, entityId));

            return new OkObjectResult(cachedData);
        }

        [FunctionName(nameof(CacheOrchestrator))]
        public static async Task CacheOrchestrator(
            [OrchestrationTrigger] IDurableOrchestrationContext ctx,
            ILogger logger
            )
        {
            logger.LogInformation("Starting Cache manager");

            // Get the data to be cached
            var (count, entityId) = ctx.GetInput<(int, EntityId)>();

            // Cache on entity
            ctx.SignalEntity(entityId, "Add", count);

            // Start a timeout
            var timer = ctx.CreateTimer(ctx.CurrentUtcDateTime.AddMinutes(1), CancellationToken.None);
            await timer;

            logger.LogInformation("Cache cleaning");

            // Remove entity now that the cache has "expired"
            ctx.SignalEntity(entityId, "Clear");
        }
    }

I'll close this issue now as it's not really an issue, just a clarification, which I now have 😁

[ElanHasson]

@sebastianburckhardt,

What happens if you call DeleteEntityAfterThisOperation and the method fails? Will it behave like defer in go (almost like a finally in C#)?

[sebastianburckhardt]

Yes, it is similar to finally: if the operation calls DeleteEntityAfterThisOperation and then does something that throws an uncaught exception, the deletion still happens.

This is consistent with how we treat exceptions inside entity operations. The entity scheduler considers them to be just "special result values", not really failures that trigger special behavior. For example, changes made to the state are not rolled back on exceptions.

[riezebosch]

There are two issues here:

• DurableClient should not be used from within orchestrations. All entity functionality we support from within orchestrations is already in IDurableOrchestrationContext.
• TerminateAsync is not a safe way to clear an entity.

We should probably clarify these points (provide better error messages and documentation).

The correct way to clear an entity is to add a clear operation to its interface and implementation:

public void Clear() { Entity.Current.DestructOnExit(); }

You can then call or signal this operation from the orchestration.

Soon we will have an even better way to do this pattern - by supporting timers inside entities (#716). Then you won't need the orchestration at all. Your caching pattern could actually be a very nice sample for this feature.

DestructOnExit and DeleteEntityAfterThisOperation don't exist? There is an DeleteState but I see in my task hub the entity still in a running state after that.

[Areson]

@riezebosch I noticed that as well. It appears that it was removed in a recent commit.

[martinkearn]

I agree with the previous two posts from @riezebosch and @Areson, DestructOnExit does not appear to exist in the latest NuGet package as at Feb 2020 (Microsoft.Azure.WebJobs.Extensions.DurableTask version 2.1.1).

The only thing I can call is Entity.Current.DeleteState(); but that does not appear to delete the entity from storage.

Instead DeleteState clears the CustomStatus property and updates the Input property to something like {"sorter":{"ReceivedFromInstance":{"35a2fdf321ac44849511271d1851b072":{"Last":"2020-02-24T07:22:41.8033573Z"}},"ReceiveHorizon":"2020-02-24T06:52:41.8033573Z"}} but the entity (or a form of it) still exists in storage.

How can I remove the entity from storage?

[AsifulNobel]

@martinkearn According to this comment, entity removal from storage is not implemented yet!

[martinkearn]

Thanks @AsifulNobel

What is the best practice right now for clearing the actual storage of the entity rather than just clearing the data contained within the entity ("garbage collection" as it is described in #931)?

Do we need to write code that works at the Table Storage layer to somehow delete the underlying storage record as part of any delete method in the entity?