Orleans: Support for custom (default) placement strategies

[hstuart]

Currently the default placement strategy is rather hardwired and limited and it is only possible to tweak it using the built-in placement strategies. I'd like to propose that we open up PlacementDirectorsManager, PlacementStrategy, and PlacementDirector so grain solutions can opt to create their own placement strategies for more specialised setups. Simultaneously, it would be beneficial to be able to specify a custom default placement strategy in the configuration and have that take effect globally rather than having to go through all the grains and add attributes to them.

Some of the custom placement strategies that might be of interest could be to limit new activations to not overwhelm a fixed machine setup (e.g. in a non-cloud/non-virtual setup), place activations according to expected communication coherence, etc.

[sergeybykov]

Let me try to break it into pieces. Correct me if I'm wrong.

1. Make default placement globally configurable.
2. Add ability to add custom placement policies with application code.
3. Placement strategies that would benefit scenarios of fixed deployments.

1 - no-brainer.
2 - seems like a medium size workitem to augment AssemblyLoader load custom strategies and lazily associate them with grain classes that use them.

1. Curious about specific examples and how far PreferLocal can go in these scenarios.

Overall, great suggestions!

[gabikliot]

The default placement configuration is already configurable. GlobalPlacement.DefaultPlacementStrategy.

About adding custom placement strategies: it is very easy now. You add a new PlacementStrategy attribute, write the PlacementDirector that implements this strategy and hook it up correctly in the PlacementDirectorsManager. What else is needed?

New placement strategies - great, we would love to see more.

[sergeybykov]

As I understand, the proposal is to bring custom placement strategies to the application level, so that one doesn't need to rebuild Orleans to add a new strategy to use by their grain classes.

My bad - I forgot that DefaultPlacementStrategy is already in config.

[hstuart]

I should have been clearer with regards to configuration support, sorry. What I meant was to extend PlacementStrategy's GetDefaultPlacement with assembly/type load support for externally specified assemblies. I agree that it is already exposed in the configuration.

And yes, my proposal is to bring custom placement strategies to the application level, as I can take much more informed choices there than in the general framework, and I would prefer not having to maintain my own fork of Orleans just to get custom placement.

@sergeybykov, as to whether PreferLocalPlacement is adequate: One of the additional things that I can discern between my grains in the application layer is whether they are CPU, I/O, or memory bound (or other external, limited resources, which is where my main interest for this feature is directed) and divide them more evenly than both local and random placement can do. Creating a placement director that takes I/O and CPU load (perhaps expected runtime of the grain as well) into consideration, might be beneficial to add to the Orleans runtime, but I would very much prefer that I can take even more control of activation placement than the generic placement algorithms can provide.

I agree with @gabikliot that it would be nice with more placement strategies inside Orleans, but I would like to keep that to a separate discussion from the issue of application controlled default placement strategy support.

@sergeybykov I can provide a more specific explanation of the scenarios I envision in a private mail, if need be.

Thanks.

[gabikliot]

@hstuart Perhaps you scenario can be solved with one generic placement policy which can be parameterized by the application? For example, look at https://github.com/dotnet/orleans/blob/master/src/OrleansRuntime/Placement/ActivationCountPlacementDirector.cs.
It balances by the number of activation, but it would be easy to balance by some undefined RESOURCE and allow the application to specify this resource. The resource will be specified in the PlacementStrategy attribute (something like GeneralResourcePlacement["Resource"]) and in addition silos will be configured to report this resource periodically as part of periodic stats publishing (DeploymentLoadPublisher).

Would that solve your problem?
I think that conceptually it is easy to do. The hard part is that not all resources require the same balancing logic. For example, for CPU you need to smooth it. For other resources it may be something else. So the trick would be to come up with a unified balancing algorithm that can work across a wide range of resources. This might be hard, but maybe possible.

If you can come up with such an implementation, it would be very interesting.

[hstuart]

@gabikliot That does not solve the problem of having to adorn each and every grain with extra information (needlessly). Adding extension points to ActivationCountPlacementDirectory might be a way to accomplish custom default placement, but not if it only consideres a single resource to balance on. It might work, but the solution does not particularly appeal to me.

I would probably prefer making a placement strategy that builds up knowledge of expected resource strain based on grain type over time, but I can only do that because I know my application will exhibit the same resource drains on the same grain types consistently, otherwise I would need to do something a whole lot more intelligent. In short, I believe we can take much better informed choices as to placement at the application layer.

I agree that it is not a particularly easy problem to solve (balancing on different resources fairly), but nevertheless something I plan to attempt, and with a bit of luck share, provided I can get permission to do that.

For completeness sake, I also have a number of non-resource constraints that will influence the placements I want to make, which might be a bad fit for a simple opening up of ActivationCountPlacementDirector.

I think the best approach will be to hold a series of useful general purposes placement strategies in Orleans, while allowing applications to choose their own freely.

[gabikliot]

Agree, sounds like allowing to add custom placement policies with application code is a useful capability.

[hstuart]

@gabikliot great, thanks.

[gabikliot]

Henrik, I am not sure we will have time to push the implementation of that feature forward, in the near time frame. So if you would like to suggest a design (how exactly to expose it programmatically), we will happily review it and later on hopefully someone will come up a potential implementation.

[hstuart]

@gabikliot sounds fine, my main goal right now was to get agreement for the feature. I'll outline a proposal for a solution as soon as time permits, and if you agree with that then implement it.

[jkonecki]

Just adding an idea to the thread:
I have a multitenant application with tenant being part of the composite grain key. I would like to implement a custom placement strategy that would place grains from the same tenant together. There is very little overlap between tenants.

The above would obviously requite the knowledge of which silos are used by grains from each tenant.

[gabikliot]

@jkonecki , would PreferLocalPlacement help?
If your grains are structured in a way that every tenant lands first on a unique per tenant grain and then this grain uses a number of other, per tenant grains, then you have the 1st grain be placed randomly and then other grains use PreferLocalPlacement and that way you will have the whole per-tenant cluster always local. Without requiring the knowledge of which silos are used by grains from each tenant.

[jkonecki]

Thank you @gabikliot for the idea. But does it mean that _all_ the calls to each grain have to be proxied through the 'tenant' grain first? That would be a bottleneck, wouldn't it?

[gabikliot]

Yes, all the call to each grain have to be proxied through the 'tenant' grain first.
It will not create a bottleneck. Since I presume you have a lot of tenants, there wont be any particular one tenant grain that will be a bottleneck and since I assume again every tenant is relatively small, you won't stress any one particular tenant grain.

It would indeed create an extra overhead - extra CPU cycles spent and extra latency. At the first it might seem like it will double the latency and CPU. But in reality it won't. The 2nd call is always local, so much faster latency-wise and CPU wise in our benchmark a rule of thumb is that we can do around 5 times more local calls then remote call. So CPU wise it will be about 20% increase. So the overhead overall will increase by 20%. And you can further optimize it by using Immutable arguments.
So overall this may not be a big issue.

[yevhen]

@jkonecki I use the approach that @gabikliot suggested. It's good one. Thus, it means, that absolutely every call need to be re-routed by the gateway (tenant) grain. And this is where Orleans with its method-based OO interfaces doesn't cut. You need message-passing so welcome to Orleankka :wink:

[gabikliot]

Yes, API wise this now becomes like Akka's router grain I guess. Or alternatively you put some logic in this gateway (tenant) grain and only delegate to other local grains for parts you cannot do in the tenant grain. But if you prefer a uniform interface approach, then indeed Orleankka is your solution. And you have the flexibility for both: you don't have to choose one over the other.

[yevhen]

Yep. I just wish that C#7 will be available sooner than 2016 so that the price (verbosity/ceremony) of defining message (type) is comparable to the price of defining method signature. We can then throw away a lot of BS in Orleans and just use uniform interface with explicit message passing and be happy! :)

There is a vast body of knowledge and patterns available for message-based designs, like the ones described here https://www.safaribooksonline.com/library/view/reactive-messaging-patterns/9780133846904/ Shame, that due to that OO-like method-based interface approach we're throwing away all of this goodness out the window.

Can you believe that all of this is just because C# is an old farty (crap)? :smile:

[danvanderboom]

If I may chime in (a year after the last comment here), I haven't quite grokked every part of Orleans yet, but I think Orleans could provide complete placement control with a single placement strategy in which a score is provided for each silo (with logic provided by some app-specific plugin), and the silo on which a new grain gets activated is the one with the highest score.

The logic plugged in for a specific grain type (or across all grain types) could provide the highest scores for silos that have the least busy CPUs, most available memory, or some composite of various factors and subscores.

It seems that a properly designed and sufficiently abstract implementation of such a placement director could serve as the base implementation of just about any placement strategy imaginable.

[kai-zeng]

@gabikliot @sergeybykov We have a similar requirement of adding custom placement strategy with application code. Currently it seems that some of the relevant classes like PlacementDirectorsManager, PlacementStrategy, and PlacementDirector are all internal.

Since we do not want to fork Orleans, I was wondering if there is any plan to make these APIs public?

+@rapoth

[sergeybykov]

@kai-zeng I see no reason to keep placement classes internal. We could start by making them public as is, and then consider using DI for that. Would you be interested in working on a PR for that?

[ReubenBond]

I've got an open PR with placement being injectable - I just haven't made
them public yet. It's a part of the statics removal work

[sergeybykov]

https://github.com/dotnet/orleans/pull/2345, right? Perfect.

[kai-zeng]

Awesome, this is exactly what we need. Thanks a lot, @ReubenBond . I'm also wondering could you make the APIs public in your PR?

[ReubenBond]

The functionality is now present, but it's hidden behind internal keywords.

The question is how do we want to expose it, particularly with regards to IPlacementContext, GrainId, and their constituent pieces.

As a reminder, here's the interface for placement directors:
```C#
internal interface IPlacementDirector
{
Task OnSelectActivation(
PlacementStrategy strategy,
GrainId target,
IPlacementContext context);

Task OnAddActivation(
PlacementStrategy strategy,
GrainId grain,
IPlacementContext context);
}

internal interface IPlacementDirector : IPlacementDirector
where TStrategy : PlacementStrategy
{
}
```

IPlacementContext is a complex interface, but some of the functionality it exposes could be moved into other interfaces which are injected separately (DI is used to construct placement directors). That would allow us to keep some parts internal for now and expose just what we think is needed.

v1.4 is already a large milestone, but if we decide that all we need to do is flip some access modifiers between internal & public, then we can squeeze it in there. Otherwise maybe 1.5?

[gabikliot]

:-)
I remember we discussed that in the past as wellm

Up to you guys, but it does feel like recently more and more of the "let's just flip the modifier" approach is prevailing. A public surface of your API is obviously an ethernal obligation, to support, to document, to fix all the different unpredictable usage of it, .... The end is that it forces you to spend more and more efforts on supporting all this huge surface and takes time from higher priority tasks. On the flip side,..., if the community asks, we are here to serve the community/users after all.

It's a somewhat strategic decision that you must make. I obviously have my opinion (keep the surface small), but it's not my decision at this point.
For this concrete question of placement, I do think we need to open it, but via bigger refactoring and not via flipping modifier approach.

[ReubenBond]

@gabikliot thanks for the input :) I pretty much agree with it, as usual. I want to also add this overlaps with the discussion about GrainIds, i.e: how should they be exposed & should we make them more general (instead of the closed format we currently use).

This needs design with design review rather than someone (eg me) jumping in like a cowboy & flipping bits. Since we're OSS, the design should also be openly debated & we can consider holding online design walkthrough meetings.

[luomai]

We have a scenario for adding a new placement strategy and wonder what is the suggested way to implement it within Orleans.

We are using Orleans to build a data processing system that needs to read big log chunks from a distributed file system. In this system, there is a management grain asking the master node of the file system for a list of worker nodes' locations that stores requested file replicas. Then we want the management grain to spawn a reading grain that is preferably sitting together with these replicas to leverage data locality. Within the reading grain, the actual file read method is using a location transparent file path. So it does not break the virtual actor abstraction of Orleans. If the worker locations are not available for the Orleans cluster, either because there is no silo on that machine or it is too busy right now, the placement director would fall back to random placement. Conceptually, this placement is similar to PreferLocalPlacement and can be interpreted as PreferLocationPlacement.

We try to look at if the current grain identity implementation can allow us to add the location information into the grain identity for the placement director to leverage. A straightforward way might be to create a custom compound key class that contain the suggested locations. There is also a proposal regarding GrainMetaData that can be used for facilitating the write operations of storage provider. We are also wondering if this proposal can help us to facilitate the read operations on the other way.

[jdom]

Just as a heads up, this is being implemented in #2932

[sergeybykov]

Support for custom placement strategies/directors was added in #2932. You can use https://github.com/dotnet/orleans/blob/master/test/TestGrainInterfaces/CustomPlacement.cs as an example until the docs are added.