Orleans: Provider model refactor

[jason-bragg]

In an effort to clean up our provider model to support more advanced scenarios required by features like streaming, we'll be making changes to our provider model in the following areas.

_Areas of focus:_
_Lifecycle_ - In general, the silo lifecycle should be sufficient to address any provider startup/shutdown needs, even for features like streaming.
_Naming_ – Many providers, such as storage and streams, are named and may have multiple instances. For this we plan to use named services and named options. This will require some prototyping.
_Dynamic providers_ – Stream providers are currently the only providers that support dynamically adding/removing and stopping/starting outside the initial silo lifecycle. For providers of this nature we’ll need to either generalize the logic used for this for stream providers or work out how to do this safely.

Legacy support is planned for third party providers.

To vet the approach in phases we're starting with the simplest provider model, bootstrap providers. The main goal of the bootstrap provider change is to vet that the silo lifecycle is sufficient for such purposes.

Next we'll be looking at storage providers to work out the use of names services and named options (in conjunction with the lifecycle, vetted in the bootstrap refactor) to support extensions to Orleans core actor model.

Last we’ll look at stream providers, using the lifecycle to improve their startup/shutdown logic and working out how to generalize dynamically adding/removing providers at runtime.

[jason-bragg]

Boostrap refactor can be found in "Added support for legacy bootstrap providers. #3738"

[ReubenBond]

Do we need dynamic providers - could we drop support for it?

[jason-bragg]

@ReubenBond, Good question. Let's discuss in planning meeting.

[jason-bragg]

Update:

• We're dropping dynamic providers for 2.0 release. May revisit in future.

• Named provider prototype is mostly working. Will be under review soon, but will provide some details here now.

_Existing_
To start, it may be helpful to revisit some of the existing Orleans capabilities this work relies on.

_Keyed services_
Since we've chosen to limit our dependency injection usage to that provided by the Microsoft.Extensions.DependencyInjection abstraction layer, we do not have named services at the container level. However, named services are quite useful, so we' built some minimal support for them on top of the DI abstraction.
For instance, if we needed to access different implementations of IFruit by name we could do this by registering the instance as name services in the container, then accessing them as such.

// register in service collection
serviceCollection.AddTransientNamedService<IFruit, Apple>("apple");
serviceCollection.AddTransientNamedService<IFruit, Orange>("orange");

// request service by name from service provider 
IFruit fruit = serviceProvider.GetServiceByName<IFruit>("orange");

_Silo lifecycle_
For better control over the startup and shutdown sequences we've introduced a (still primitive) silo lifecycle. Silo lifecycle participants can register to take part in various stages of the silo startup and shutdown sequence. See "Minimal silo lifecycle #3670". This capability has also been exposed to application developers.
For instance, if an application has a component that needed to be initialized as startup and shutdown with the silo (like bootstrap providers) one needs only implement the ILifecycleParticipant interface and register their instance in DI with that interface.

// implement ILifecycleParticipant<ISiloLifecycle> interface
public class MyComponent : ILifecycleParticipant<ISiloLifecycle>
{
    public void Participate(ISiloLifecycle lifecycle)
    {
        lifecycle.Subscribe(SiloLifecycleStage.ApplicationServices, OnStart, OnStop);
    }
}

// register types with service collection
serviceCollection.AddSingleton<MyComponent >();
serviceCollection.AddFromExisting<ILifecycleParticipant<ISiloLifecycle>, MyComponent >();

With that, MyComponent's OnStart would be called at the application services stage of the silo lifecycle on silo start, and it's OnStop would be called at that stage when the silo is being shutdown.

_New_
Since providers are mostly just named services that take part in the silo lifecycle, the only thing we need to add is some additional support for names services so they can take part in the silo lifecycle.

To address this, the prototype introduced the new keyed service calls:
_AddKeyedSiloLifecycleParticipant_ and _AddNamedSiloLifecycleParticipant_
These calls act like the preexisting keyed services registration calls, but also wire the registered service up to the silo lifecycle.

The prototype for this was done against the storage provider infrastructure, specifically the AzureTableStorage. Since the existing storage provider interface couples the grain facing calls (Storage bridge facing really) with the initialization pattern from IProvider, it was necessary to decouple these. The grain facing calls were moved to a new interface IGrainStorage and the IStorageProvider interface now joins IGrainStorage with the IProvider interface to preserve the legacy provider pattern.

public interface IGrainStorage
{
    Task ReadStateAsync(string grainType, GrainReference grainReference, IGrainState grainState);
    Task WriteStateAsync(string grainType, GrainReference grainReference, IGrainState grainState);
    Task ClearStateAsync(string grainType, GrainReference grainReference, IGrainState grainState);
}

public interface IStorageProvider : IGrainStorage, IProvider
{
}

The grain storage infrastructure was adapted to interact with IGrainStorage, while the legacy provider infrastructure continues to initialize IStorageProvider.

Most of the logic of the AzureTableStorage class was ported to AzureTableGrainStorage which implements only the IGrainStorage interface and gets it's configuration from an AzureTableStorageOptions provided by named options.

AzureTableStorage is now a thin wrapper over AzureTableGrainStorage which converts the IProviderConfiguration into an AzureTableStorageOptions to configure the storage class.

To use the new AzureTableGrainStorage one need only register it as a named lifecycle participant (assuming configuration is in place).

// standard named options configuration
serviceCollection.AddOptions<AzureTableStorageOptions>("AzureStore")
    .Configure(option =>
    {
        option.Name = "AzureStore";
        option.ServiceId = serviceId;
        option.DataConnectionString = "<data_connection_string>";
        option.DeleteStateOnClear = true;
    });

// Register AzureTableGrainStorage by name as a named lifecycle participant.
serviceCollection.AddNamedSiloLifecycleParticipant("AzureStore", AzureTableGrainStorageFactory.Create);

If we go with this approach, there is relatively little work required to reach parity with our existing provider pattern, most of the work is in maintaining the legacy pattern.

[galvesribeiro]

First of all, great work @jason-bragg!

There is only one thing that got me confused with this work and the related issues... How is this correlate (if it does) with Facets?

I thought when we were discussing facets system the main intention was to (1) eventually remove the inheritance on Grain<T> and (2) make the core services and providers injected thru the facet.

Was facet system dropped?

[veikkoeeva]

Would it be possible to interact somehow with the new providers? Related to storage providers, I have two use cases in mind:

1) A general one the same for all, where for instance, if I wanted to dynamically to alter the serialization and deserialization of grains to and from the persistent storage (based on type, ID, silo ID and whatever other parameters). This would require changing the configuration dynamically somehow, which is quite difficult to do currently. One use case I can think of is switching format from, say, JSON, to Parquet in the storage or perhaps the shape of data has changed and needs a custom transformation when being read.

2) I might want to alter the way the operation is done in the storage. This might be be more storage-specific, i.e. in ADO.NET could be adding a conditional by type/silo ID (whatever) to the appropriate clauses that operate with the storage (and then the DBA for the storage could, for instance, create a new Storage table for some split data to a separate filegroup and schema for disaster recovery, security (GDPR) purposes etc.).

[jason-bragg]

@veikkoeeva
The current refactor deals mainly with the extensibility of the Orleans actor model. Provider support is an existing extensibility point which is somewhat limited and requires significant infrastructure. This refactor deals mostly with improving Orleans extensibility to the point where the provider infrastructure is no longer necessary, then porting the existing providers to this new pattern.

The sort of changes you're referencing would, imo, need to be provider specific, In the case of storage providers, I perceive many issues with our current providers, especially in regards to serialization and backwards compatibility. A general pattern to address these issues across all storage providers may be valuable, but is beyond the scope of this effort.

If you see issues with our existing storage provider implementations or patterns, or have suggestions for improvements, I'm confident the team and community would be interested in learning of them. There have been a number of issues raised regarding storage providers, but most have not reached the critical mass necessary to be addressed in code. I encourage you to create an issue where your suggestions can be explored in depth or revisit some of the previous issues.

[jason-bragg]

@galvesribeiro

"How is this correlate (if it does) with Facets?"

This is independent of facet system, though they should eventually be linked. There are plans to enable grain storage via a facet rather than via inheritance, and that will likely touch on this refactor, but for the most part, this work is more about obsoleting the current provider model.

[jdom]

Couple of comments/questions:
In the example, you created the named option "AzureStore", but then also assigned an options.Name property to the same value. That shouldn't be case, right?

You didn't get to the details of what the AzureTableGrainStorageFactory.Create is but I assume it's a delegate that takes a string (the name) and returns the created object? If that's the case, probably for most cases this can be automatically inferred to just provide a concrete T type and the implementation will just use ActivatorUtilities.CreateInstance, correct?

[jason-bragg]

@jdom

but then also assigned an options.Name

The option.Name was just a detail of the prototype. Its not part of the general pattern.

You didn't get to the details of what the AzureTableGrainStorageFactory.Create

As you surmised, that's a factory function. In the prototype AddNamedSiloLifecycleParticipant and AzureTableGrainStorageFactory.Create signatures are as such.

    public static void AddNamedSiloLifecycleParticipant<TService>(this IServiceCollection collection, string name, Func<IServiceProvider, string, TService> serviceFactory)
        where TService : class

    public static IGrainStorage Create(IServiceProvider services, string name)

A concrete factory class can be used instead, but a concrete service is more difficult as there isn't a way for DI to resolve the 'name' which is necessary to lookup the named options. Currently that is the responsibility of the factory function.

    public static IGrainStorage Create(IServiceProvider services, string name)
    {
        IOptionsFactory<AzureTableStorageOptions> optionsFactory = services.GetRequiredService<IOptionsFactory<AzureTableStorageOptions>>();
        return ActivatorUtilities.CreateInstance<AzureTableGrainStorage>(services, optionsFactory.Create(name));
    }

Very open to other solutions should they have advantages. I was mainly focused on the feasibility of the approach and what the public surface would look like. Implementation details can be iterated on and improved over time.

[jason-bragg]

I had hoped to be able to iterate over the named options by option type and build the associated provider from that pair, but I found no way to enumerate them, hence the need to register a named service that could look up it's options by name. Since it was, imo, not safe to assume all providers would have named options or options at all, supporting a factory function responsible for these behaviors on a per implementation bases seemed like a safe approach.

[galvesribeiro]

not safe to assume all providers would have named options or options at all

Well, real life shown us that there is no way to have a provider without configuration/options. Even the dev/test ones like memory grain state or SMS has it. However, its always good to not _assume_ such things as it may shoot on our feet in the future.

[jason-bragg]

Closing - New extensibility model of using names services, with options, which participate in service lifecycle for asynchronous initialization seems to be working as intended. Legacy providers are still supported via the legacy libraries.