Akka.net: Integrate Akka.Persistence with Azure Service Fabric

[rogeralsing]

Claudio (@clca) from Microsoft is working on integrating Akka.Persistence with Azure Service Fabric.
This work will be done in a separate branch in this repo.
https://github.com/akkadotnet/akka.net/tree/azure-fabric

We have also allocated some developers from to help building this integration.

cc @r1sc, @nethousejela, @AndersSahlin

[BenGale]

How do you guys see the integration with azure service fabric going forward?

[rogeralsing]

We are waiting for Claudio's initial commit.
First task is to get Akka.Persistence working with Service Fabric.

After that we will start exploring how we can integrate Akka.Cluster

[clca]

I should be able to push the initial commit by tomorrow. The initial commit contains a barebone implementation of a Akka Persistence plug-in (that needs a lot of more work) and a Service Fabric service that can be used as Persistence storage so it should be a good starting point for building the real thing.
Will update this thread as soon as I'm able to push the code.

[rogeralsing]

Any progress on this @clca ?

We wouldnt mind getting our hands on it even if things are not complete

[clca]

Waiting for the final step in the approval process. I should be able to push the code out next week (Mon is Holiday in the US)

[varghesep]

I don't understand the benefit of using Service Fabric in AKKA.net except it can be used to deploy actor systems to Azure. Is it to integrate two actor systems deployed in multiple machines so they can work together? For example, there are two Microservices having two different actor systems in it. They can work in a pub-sub way through Service Fabric?

[jeremylcarter]

Not to be nasty @varghesep but that comment isn't very constructive. You don't have to understand the benefit of using Azure Service Fabric because nobody asked for your opinion in this thread. Lets all try to stick to topics and be positive to one another!

[varghesep]

@jeremychild Sorry, you felt that my comment is not constructive and not positive. Sorry for asking a question on the benefit of using Azure Service Fabric with AKKA.NET.

[clca]

we have an opportunity to leverage some of service fabric features to provide a great runtime environment for Akka.net. For instance we can make akka.net cluster management easier on service fabric by leveraging some of the Application Lifecycle Management feature it provides. For persistence we can leverage SF stateful-ness to provide reliable/replicated and low latency storage option for Akka.net. Akka.net can benefit from having access to a low latency storage infrastructure if, for instance, the app is running in the Service Fabric cluster. And there are probably other opportunities we can explore.

[Aaronontheweb]

@varghesep I didn't interpret your comment that way - it's a good question! And @clca did a great job answering it.

[clca]

I pushed the code for the first very early prototype/PoC of the Service Fabric Persistence service. Only Journal is support for now, Snapshot is work in progress.

[varghesep]

Thanks to both for the efforts. I checked for the PoC in the link that @rogeralsing provided but could not find any code. I'm eager to see the implementation.

I'm excited about the integration of the AKKA.NET with SF (Service Fabric). I'm exploring the possibility of using the Service Fabric Reliable service with AKKA.NET actors. The scenario is to use Service Fabric Reliable Service to queue up the requests in a Reliable queue from a REST end point. Then process those requests with AKKA.NET actors. The great benefit I see with SF is it's deployment easiness. The Service Fabric explorer is a good tool to see the partitions and where the service is running. It can be run local box too.

IMHO, the request/response model in Reliable actors makes me to think it is not reactive and is true to some extend unless the code is written that way. They fit many of the applications that has POCO object models that need scalability. The AKKA.NET programming model is reactive with first class message passing and is suitable for complex financial applications. It has FSM which is beneficial too in financial applications where the actors (or objects) that has many state changes.

It would be great if the Reliable actors in SF can be switched with the AKKA.NET actors through some provider pattern and use the existing benefit of garbage collection (if actor has not been used for a while).

[ReubenBond]

Browsing GitHub for code using Service Fabric, I found this (hi everyone :smile:). I checked out the PoC code, nice work.

At least some people from the Akka.NET community and from the Orleans community are interested in hosting/persistence with Service Fabric. Maybe we can work together on the common parts.

The persistence mechanism currently used in https://github.com/akkadotnet/Akka.Persistence.ServiceFabric is acceptable for a small number of events, but it doesn't scale beyond that. Currently, every operation (including reads) will cause replication of the entire journal for a given actor. Read operations could op-out of replication using the [ReadOnly] attribute, but write operations will still replicate everything.

@clca, could we have a store which allows us to replicate with a finer-grain? An event journal only needs simple semantics: it could be provided by one of the Reliable Collections with minimal modifications.

Better yet, something with the relatively simple semantics of Azure Table Storage would be great :smile:.

I've been toying with a very simple store based on ESENT which can coexist with the other Reliable Collections (i.e, it can take part in the same transactions). Very early PoC at the moment: it's just a side project.

[clca]

HI Reuben,

We can optimize the code in various areas as you suggested: ReadOnly attribute, .... On the scaling side: the structure is 'partitioned' per PersistentId so performance could become a problem is an actor have a very long tail of events that it wants to persist. And we are working on the Service Fabric side to allow 'delta' state management on actors so the replication will be a lot more efficient with the next release.
Anyway nothing is set in stone so happy to discuss alternative designs and any contribution will be welcome.
(progress has been slow so far but I should be able to allocate more time on this so happy to continue the discussion and come up with a common design that can work for Orleans as well).

[vladkosarev]

Is this dead?

[Danthar]

Unless someone is willing to pick this back up again, And bring it back up to speed. https://github.com/akkadotnet/Akka.Persistence.ServiceFabric
I would guess so.