Orleans: Service Fabric Mesh support

[mholo65]

I started thinking about adding support for Service Fabric Mesh.

So far the docker-aspnet-core example works like a charm, example here. But I want to take this further and use Service Fabric Mesh for clustering, and (when Service Fabric 6.4 lands) use Standalone Reliable Collections for persisting grain state.

For clustering, I started looking at current the current implementation in Microsoft.Orleans.Clustering.ServiceFabric and _"I think"_ that a similar approach would be doable, but instead of taking dependency on System.Fabric.FabricClient, one should try using the Service Fabric Client REST Api instead.

Now to the questions:

• Is adding support for Service Fabric Mesh on your roadmap? AFAIK, Mesh will not bring the Actor programming model currently available in Service Fabric Native. People, (including me) are looking for alternatives if moving to Service Fabric Mesh. I would love to help adding Mesh support for Orleans.
• As I'm new to Orleans (and specifically extending Orleans), I might need some help on getting started. To add support for new clustering providers, am I right in saying that I need to implement the following services?:
  
  
  
  • IMembershipOracle or IMembershipTable
  
  
  • IGatewayListProvider.
  
  

[sergeybykov]

@ReubenBond was about to publish a Mesh sample, but ran into an issue running it locally. Let's see what's the overlap here.

[ReubenBond]

Thanks for opening this, @mholo65 :)

I have been working on various aspects of Service Fabric support lately.

For now, Mesh support is just in the form of a sample. When Standalone Reliable Collections are released, it would be fantastic if you wanted to add support for grain persistence. If you're working on clustering for Mesh, then implementing IMembershipTable backed by Standalone Reliable Collections would be the best approach, IMHO.

IMembershipOracle is used by the current SF clustering provider, but that isn't being shipped because of issues that users experienced which essentially come down to the provider not guaranteeing what IMembershipTable does. I'm looking into relaxing the requirements.

I have a branch where I'm looking at implementing support for Stateful Services here https://github.com/ReubenBond/orleans/tree/feature-sf-stateful - ignore the timestamp on the last commit, it's not accurate because I just squashed more recent commits.

So far, I have a proof-of-concept for hosting Orleans in an SF Stateful Service using SF for clustering and grain persistence. I'll be working on it more today and might have a PR soon.

[mholo65]

Thanks for your insights @ReubenBond

I'm looking forward to implementing clustering (using IMembershipTable) and grain persistance on top of standalone reliable collections once it's available in Service Fabric Mesh. I'll let you know if I run into any problems and will send a PR once it's ready for review.

[ReubenBond]

Great! We're here to help, so feel free to drop in on Gitter if you want to discuss any aspect of it.

[mholo65]

@ReubenBond @sergeybykov so, I've started implementing the MembershipTable based on ReliableCollections. Some code are here. Also, glad to find ready made tests which was easy to hook up for new IMembershipTable implementations.

I was wondering, what's the benefit of supporting the "extended protocol"? I can see that e.g. the Consul based Membership Table doesn't support it, while e.g. Azure Table Storage based does. Should we target supporting that with ReliableCollections? Also, I'm somewhat confused over TableVersion and etag. Are there any spec that I could look at, which clarifies the usages.

[mholo65]

@ReubenBond regarding IGatewayListProvider. Because of limitations in Mesh (ReliableCollection can only be accessed from the owning service, e.g. the Silo) we’d need a way for the Grain Client to query (any) Silo to get the list of gateways. Is there a possibility to leverage Orleans messaging for this communication, or should I just ”roll my own”?

[ReubenBond]

@mholo65 how will clients discover the endpoint they need to query? Would the Silos need to host an HTTP endpoint (presumably behind a shared VIP/load balancer) to provide that information?

[mholo65]

@ReubenBond, I think we should be able to leverage SF Mesh platform for disovering any Silo (e.g. via DNS). I don’t think it would be necessary to expose HTTP, any other protocol should work.

[ReubenBond]

Can Mesh serve up SRV records for the gateway IP:Port for all silos at once, or will we just get a round-robin IP in return?

[mholo65]

I think round-robin.. I need to do some testing/PoC..
//cc @MikkelHegn

[mholo65]

...or we could just introduce another service, MemberShipService which would own the RC. Both the Silo and the Grain Client would go through that service for Membership read/write.

[ReubenBond]

Exposing a metadata endpoint for gateway info from the silos would be fine in my opinion. Either way is ok as long as the user experience is good. Having a separate MembershipService may require more work, so I'd lean towards the alternative.

[MikkelHegn]

SF DNS does provide SRV records with port numbers. For stateless services, I'm not sure if we round robin or give you all, someone will have to check https://github.com/Microsoft/service-fabric/tree/c326b801c6c709f36684700edfe7bb88ceec9d7f/src/prod/src/Management/DnsService/src :-)

Looks like we are returning an array of endpoints for all record types, and including port numbers for SRV records, but I remember having discussions about whether the ip and ports are actually mapped correctly in the array.

[mholo65]

@ReubenBond this leads me back to the initial question:

Is there a possibility to leverage Orleans messaging for this communication.

I.e. leveraging Orleans (specifically Orleans.Core.Messaging) for exposing (and consuming) a service on the Silo which exposes the functionality for IGatewayListProvider.

[ReubenBond]

@mholo65 the clients would need to know some gateway endpoints in order to be able to reach the silos to learn about other gateways. I believe the DNS SRV record route may be preferable, but I'm not certain how frequently they would be updated. Either way, it sounds as though DNS would need to be involved in some capacity.

[MikkelHegn]

DNS in SF has no cache, each request is being routed to the naming service, so there should be no lag.

[ReubenBond]

Nice, thanks @MikkelHegn

[ReubenBond]

Moving to 2.4.0

[dazinator]

What was the final outcome of this? I was just considering whether to use Mesh in an upcoming project ;-)

[ReubenBond]

You can run Orleans on SF Mesh if you want, similar to running in any container environment