Faas: Feature Request: CQRS, SAGAs and Event Sourcing as a director function pattern modeled as state machines in yml

[dghadi1]

Expected Behavior

• Model the behavior of director functions (acting as orchestrators or transactors) as a series of commands, handlers, events and states that can allow for complex enterprise use-cases where handlers can be simple functions combined together by modelling a cohesive business process encompassing multiple steps.

Context

Orchestrations or transactions when formally modeled gives predictability (through precise design) and maintains behavioral guarantees, when executed in a distributed manner for mission critical use-cases.

• Retries and predictable recovery from a failed state at each interaction level
• fallbacks for resiliency
• deduping data fetching calls (especially when executing reads in parallel)
• compensating transactions (or rollbacks when mutating) - SAGA
• snapshotting assistance for troubleshooting undesirable behaviors through a distributed event journal
• strict separation of model and code
• strict separation of read/write workloads (CQRS)
• event driven execution always (Event Sourcing)

• Orchestrators are non-side effecting chain of functions (as handlers) and the workflow is modeled as a state-machine using a Command-Handler-Event-State pattern (CQRS). The commands can be any core business event (fetching data) or an agnostic event such as authentication, authorization, introspection, caching, validation etc)
• Transactors are side effecting chain of functions (as handlers) and the workflow is modeled as a SAGA state-machine using a command-Handler-Event-State patter (CQRS). The handlers can either fetch data, or mutate data.
• An Event Journal persisting every commands, event, and the state and could be replayed (Event Sourcing pattern)

Possible Solution

Build a cli command and integrate into the faas-cli by providing a orchestrator-name.yml or a transactor-name.yml and a process to generate the director function image. The orchestrator or transactor can use other function definitions already deployed in faas.

Example model

will add a sample model here soon

I'll update the issue with more detailed design soon. Any help/direction on how to do this faas natively will be greatly appreciated (using kubernetes).

Related issues

91

[alexellis]

Hi thanks for your interest in the project. What you're asking for here is a huge and complex piece of work. Maybe we need to start at the beginning? Could you introduce yourself and give some context about the proposal and usecase?

[dghadi1]

Thanks for your quick reply

I'm a devops/microservices engineer consulting for a large enterprise running hundreds of micro services. We have a custom engineered platform which has a workflow model as I described above. I'm primarily skilled in JVM technologies but interested to try my hand in golang (which I'm still a beginner)

more context and use case

I'm looking to evaluate/pilot OpenFaas (or server less compute model in general) for enterprise production use cases for my clients and put it in their roadmap for 2018. I looked at OpenWhisk, kubeless, aws lambda and Openfaas. I'm most impressed with the design and the community around openfaas, hence I chose this as my first experiment.

Many of the enterprise applications I consult with have multi-step orchestrations where each step does one of the following and cohesively produces an API (over Swagger or GraphQL or sometimes both)

• Fetch data
• modify data
• save data
• transform/process data (run business rules)
• compute data
  etc

For those kind of services I hacked together a simple specification to model the workflow as a command-handler-event-state pattern using Akka Persistent FSM in scala. I later abstracted that into an yaml spec so that I can autogenerate the code and make it friendly for business users to model their service without writing code.

This model allowed us to churn out services very quick without needing a lot of developers repeatedly writing code. We have modeled and running hundreds of such services now in production.

I want to extend this model by externalizing the handler functions in the pattern above into openfaas functions running in containers (practically any programming language). Currently they are local procedure calls in Java. This allows for a strict separation between developer/designer responsibilities and allows for further agility and automation.

Solution options

If we stick with Akka and Scala and JVM - it already has everything we need to provide the CQRS and Event Sourcing capabilities including the Journal. However, I want to evaluate lighter weight options with a golang equivalent if possible. We might still have to deal with the cold-start latencies either eitherways.

The proposal

1. Generate a function image from a yaml spec (acting as a director/orchestrator/transactor). The spec defines state transitions (similar to aws step functions, except that it uses reactive execution compared to imperative execution)
2. Reference other handlers in the spec as images and execute them as events either in Sync or Async manner
3. Optionally build a visual flow editor for the yaml spec
4. Package the whole thing as a plugin for openfaas if possible (that way - we can work independently)

I have couple of friends who are willing to hack together something in the next two-three weeks, so we are looking for some general direction on the use case fitment and interest for this feature in this community.

[alexellis]

Have you got examples of your Akka / YAML services you can share here?

[alexellis]

Pinging @dghadi1

[alexellis]

Derek close

[alexellis]

Closing due to inactivity. IMHO the scope is too broad and should be broken down into individual items if there is any demand.