Orleans: Managing grains without subclassing: Getting the IGrainActivationContext?

[bddckr]

Summary

I need a Grain instance to pass to IGrainRuntime.DeactivateOnIdle and other public, accessible API - how do I get that without forcing the target grain type to have a reference to a GrainServiceClient that then itself can get to the IGrainActivationContext?

Should Grain.Data be public instead of internal? Is it a bad idea?

Issue

How do you handle such things like using the API that is (seemingly) only available to grains themselves? E.g. RegisterTimer, DelayDeactivation and DeactivateOnIdle. I want to call such API from some managing grains "from the outside".

Example

I have some patterns like a grain that should be kept alive. I want to re-use the implementation of that pattern but I don't like having this as a grain superclass to inherit from - composition is really needed here instead, as I have various differences in the kinds of grain implementations and we can't have multiple superclasses of course :smile:

What I ended up with are three things basically:

• An interface with a single method StayActivated that literally does nothing and just returns Task.Completed. (Thanks to C# 8's default interface implementations this is basically just a marker interface and needs no copy-pasta in each implementing grain!)
• A GrainServiceClient which any implementing grain gets via constructor injection, then keeps a reference and calls KeepActivated(this) or AllowDeactivation(this) whenever necessary. This client is added as a scoped service to ensure client instance gets the correct IGrainActivationContext.
• A GrainService to keep a list of all grains to repeatedly call StayActivated on.

The only reason this works is because I can get to calling the Grain instance in the client via IGrainActivationContext.GrainInstance. After the client delays the activation it also calls into the GrainService as I don't want the scoped client to do the keep-alive logic - it dies with the grain getting deactivated after all AFAIK.

While this allows me to re-use this pattern implementation now by just

• Tagging the grain types with the interface.
• Adding a constructor argument + reference to the GrainServiceClient.
• Calling the client's methods.

I am still having a need to allow "outsiders" to tell the service to keep a different grain activated.

[SebastianStehle]

I think this is in general a good approach. Grains should be more composable. Right now you have to use inheritance to accomplish a lot of things and the different interface like IGrainWithStringKey and IGrainWithGuidKey make it even more harder to implement generic extensions.

[jason-bragg]

While this doesn't directly address all of your concerns, regarding cross cutting concerns you wish to be injected onto any grain, you may want to consider grain extensions.

Below are some links to test code where we register and use grain extensions to add code to grains at runtime. This behavior is primarily for cross cutting concerns as there isn’t any way to limit these extensions by class or interface, any grain extensions registered in the container can be installed on any grain.

In the below test, we define a simple grain extension (IAutoExtension) and register an implementation of it in the container (AutoExtension), then cast a test grain which does not have this behavior to the extension, and call it. This triggers the installation of the grain extension onto the grain where the extension functions act like normal grain calls.

Test grain extension interface

public interface IAutoExtension : IGrainExtension
{
    Task<string> CheckExtension();
}

Test grain extension implementation

public class AutoExtension : IAutoExtension
{
    public Task<string> CheckExtension()
    {
        return Task.FromResult("whoot!");
    }
}

Register behavior in the container

hostBuilder.AddGrainExtension();

Call extension on any grain

IAnyGrain grain = GrainFactory.GetGrain<IAnyGrain>(GetRandomGrainId());
IAutoExtension autoInstalled = grain.AsReference<IAutoExtension>();
await autoInstalled.CheckExtension();

The tests simply ensure that the extension works, but don’t really provide a good practical example because they don’t illustrate how to get access to the grain, which is probably needed. For an extension to get access to the grain, it will need to take an IGrainActivationContext in it’s constructor. This should allow the extension access to the grain as well as other details. Something like:

public class AutoExtension : IAutoExtension
{
    private readonly IGrainIdentity grainId;
    private readonly Grain grain;

    public AutoExtension(IGrainActivationContext context)
    {
        this.grainId = context.GrainIdentity;
        this.grain = context.GrainInstance;
    }

    public Task<string> CheckExtension()
    {
        return Task.FromResult($"{this.grainId} says whoot!");
    }
}

The activation context also has an ‘items’ dictionary that different extensions or the grain can use to share data between each other without having to know anything about the other components.

https://github.com/dotnet/orleans/blob/master/test/Grains/TestGrainInterfaces/ITestExtension.cs
https://github.com/dotnet/orleans/blob/master/test/DefaultCluster.Tests/ProviderTests.cs

For other examples, both Orleans streams and transactions use grain extensions to add needed grain calls to application grains to support those feature sets.

Please let me know if you’ve any questions/comments/feedback.
Best,
jbragg

[jason-bragg]

Regarding composability of grains, the direction we've been going is to expose future feature sets as Orleans 'facets' rather than via inheritance.

https://github.com/dotnet/orleans/issues/3223

The facet system was introduced in 2.x and is currently used for transactions and IPersistentState. Unfortunately the facet system did not manifest as simply as we'd hoped, so we've not made it part of the official programming model and documented it as such, but it is fully implemented and usable.

[SebastianStehle]

This is great. Would be good to get access to the runtime as well to manage timers, reminders, activation and so on.

Can an extension also be an incoming grain call filter?

[jason-bragg]

timers, reminders

These systems should be refactored to be facets, so they can be injected into the grain extensions. We're just gated by dev time. Contributions are welcome! :)

Edit: correction, these probably shouldn't be facets, but should be made injectable. Some may already be, haven't looked at them in a while.

Can an extension also be an incoming grain call filter?

No, they are different mechanisms. Grain extensions add new calls to a grain. Grain filters wrap all grain calls, both those on the grain's interface as well as any added by grain extensions.

[SebastianStehle]

I had an issue with high memory usage because of too many grains. I have implemented a service that uses a cache to deactivate the least recently used grains when a threshold is reached.

The only model I found so far was via a base class because I need to call DeactivateOnIdle from outside. Furthermore I need to update the cache for every call. So I have a grain call filter for that. Would be great to have a single component for that. I think the facet system would be great for that.

[jason-bragg]

I think the IGrainRuntime is currently injectable. So you should be able to put it in the constructor of any grain extension and use it.. but I've not tried that :/

[jason-bragg]

I had an issue with high memory usage because of too many grains.

Can you provide more details? How many grains? Were memory issues 'out of memory', garbage collection, something else?

[SebastianStehle]

It is not related to Orleans. I use Orleans for https://github.com/squidex/squidex and each content item is a grain. Usually it has a lot of benefits, but when you do a "mass" import of data like 100k items with a lot of content per item you would end up with all your data in memory and you can get an out of memory exception.

I was thinking to have one grain per partition of content items, but it is hard to distribute the workload (perhaps with worker grains) and would require a lot more changes and my approach seems to work good as well.

[bddckr]

timers, reminders

These systems should be refactored to be facets, so they can be injected into the grain extensions. We're just gated by dev time. Contributions are welcome! :)

Edit: correction, these probably shouldn't be facets, but should be made injectable. Some may already be, haven't looked at them in a while.

I think the IGrainRuntime is currently injectable. So you should be able to put it in the constructor of any grain extension and use it.. but I've not tried that :/

I believe all of these should just work being injected - I can already confirm IGrainRuntime working as part of a GrainServiceClient constructor at least. There are usable interfaces (they work on Grain instances) for each and they are just singletons after all:
https://github.com/dotnet/orleans/blob/613b215c90382b90031e3cc81f2478b3fabc8ac4/src/Orleans.Runtime/Hosting/DefaultSiloServices.cs#L104-L107

Thank you for bringing up the grain extensions! I've seen them and the facet PR and was actually just forgetting about them. In the case of the extensions I wasn't sure whether I could use it outside myself.
So sounds like there's a reason for all of this not being documented yet - but I'm happy to do some tests and will probably either share my existing grain service + client implementation for "keep alive" management, or if I can get it to work, an IGrainExtension.

It seems like the extension pattern is allowing me to do everything I wanted, so 🥇 if it does! 😄

[bddckr]

Oh it looks like IReminderService should be referenced actually - the reminders are using the grain service pattern and we can only work with the service if we want to pass grain (references) around ourselves for it...

Looks like that interface is only a thing internally and working with IReminderTable or IReminderTableGrain directly seems like working too closely with the "internals". Maybe @ReubenBond can chime in here (I saw you recently split the former into the two...)?

[ReubenBond]

@bddckr I believe IReminderRegistry is the interface you want for interacting with a grain's reminders

[bddckr]

Yeah but that's a grain service client, so only can be called from a grain, right? Or will the IGrainExtension still work with that idea of the grain service client's CallingGrainReference?

[ReubenBond]

It should work from an IGrainExtension in the same way, since it's running under the same context

[bddckr]

Great! I'll do some tests of this all and will post my findings here. I believe if it all works I can close this issue for sure and perhaps raise a PR documenting some of this, if the core team agrees with these things already being documented is a good idea. I don't want to get in the way of bigger refactorings etc. getting into the spotlight or something.

[bddckr]

Having no issues doing all that was discussed in here previously. Sadly I don't currently know when I'll have some time to contribute some docs back, but this issue is definitely resolved from my POV.

Thanks for all the help!