Eshoponcontainers: [Feedback] Event Sourcing and CQRS with eventstore

[paulopez78]

Hi,
Great job with that project. I'd would like to add my feedback about the future of domain events and integration between microservices. I would like to suggest to apply event sourcing in the orders microservice to have an event log that can be consumed by others (CQRS) creating different views of the system. eventstore is perfect for that scenario because you don't need 2PC between SQL and RabbitMQ and you can consume the events using competing consumers which behaves pretty much like RabbitMQ publish/subscribe. From my point of view Event Sourcing and CQRS are key in microservices architectures.

[CESARDELATORRE]

Thank you for your feedback!
Yes, we're aware of that approach and agree on your statements.
The event-sourcing approach is something I did include in the Guide/eBook when talking about integration events, so from that point of view, it is covered in the guidance (but not in the sample app).

It is probably one of the bests approaches when you are already using Domain Events and Event-Sourcing for storing the Domain's data as events while it can also be used for integration events. If using a "single" event-store database when saving aggregate's events, you already have atomic operations and then the subscribed microservices to the Event Store would get the integration events.

However, my thinking here is that EventStore for integration events fits great when you are already also using event-sourcing for the Domain data persisted as events (Event-Sourcing = No "current state" in SQL), because if you use EventStore just as a view/projection of integration events but maintain the "current data" in SQL, you might have the same issue in regards the atomic operation. I mean that if after saving data in SQL aggregate's tables the operation saving the event into the EventStore fails or any other error happens in between, you'd have an inconsistent state.
How would you solve that if you don't use the EventStore also for the Domain Events data? if not using full event sourcing which means no "current data in SQL"?

Can you elaborate if you mean the first or the second case?

My initial thinking is that when choosing Event-Sourcing for the integration events it would be better to have the "source of truth" of your Aggregates stored as events, too, so it is easy to maintain consistency, no 2PC issues there.

On the other hand, if your "source of truth" is SQL with "current state data", as we currently have, then an approach based on an Event Bus with RabbitMQ underneath plus an Integration-Event Table in SQL used in-between and under the same transaction (so you avoid inconsistency risks), feels more natural for our initial implementation.

But I'm really interested on how you'd implement it.
If you'd like to fork the project and show any implementation using EventStore for the Ordering microservice's integration events, we'd love to review it and take it into account.

To sum up, it is for sure in our radar. If not the first approach we might implement because we already chose to use SQL at the Ordering microservice, it might be an additional approach that would be very cool to show, as this is a case where event-sourcing based on EventStore offers differentiating values and fits great.

Even further, I really like the idea because microservices/containers should live in a polyglot world (multi framework and multi language) and multi DB environment. We aim to evolve to a more diverse sample application and using EventStore which is a great asset from Greg Young for the Event-Source community, is a great example.
We might do it, eventually.

Thanks a lot for your feedback! :)

[davasquezp]

Just to add another "flavor", we are using a combination of State-based Aggregates stored in Mongo and EDA as overall architecture, so in cases we use the produced events to also generate CQRS Read Models. We would prefer event sourcing for scenarios when history is important (such an Orders Bounded Context) and for simple, more crud-like scenarios, storing a whole aggregate per collection give us agility to put the effort into modeling instead of thinking too much about ER data model.

[paulopez78]

@CESARDELATORRE The idea is to have an event log as source of truth for the orders bounded context implemented only with eventstore, so no relational database involved here and no 2PC. That would require to change the aggregates for being event store aware and implement a repository for the event store, but that's pretty standard and easy to find in different repos out there. I would like to add a PR with a basic implementation of that idea as soon as I have some time. Again thanks for starting that interesting project and for accepting all the community feedback! :)

[davasquezp]

Main trick when implementing event based aggregates or sagas (sending commands) is to store in a public list the "uncommitted events" (for aggregates) or the "undispatched commands" (for sagas) and let the infrastructure-aware code take care of them ( to publish those events and to dispatch the commands). One great secondary effect of this is that you can unit test now your aggregate or your saga by checking the produced events or commands, while keeping the internal state private.

[CESARDELATORRE]

@paulopez78 Sure, please, add a PR or point to any fork for reviewing first and taking it into account.
So, you meant what I mentioned, using Event Sourcing as the "source of truth" for the Ordering microservice. We could always do that or since microservices in general should be autonomous/independent, as long as we don't change the external interfaces (Http interface, etc.) we could also have two different implementations of the ordering microservice (for proof of concept purpose), one ES and the other using the relational SQL DB.

However, there are two different things here.

1. Implementing the Ordering microservice with ES. That could be done, eventually, because I like to have diversity here.

1. Integration Events across microservices: Sure, if using the ordering event-store directly for pub/sub, we wouldn't have the 2PC needs as the event-store can act itself as the source for the integration events. If thinking only about that particular microservice with ES and subscribers, everything sounds cool.
   But, unless I'm not getting your whole story, the issue that I see here if using pub/subs based on a particular EventStore like the Ordering one, is that we'd be coupling the other microservices to a specific event-store's subscriptions. Kind of similar to if SQL Server were capable to work as an integration-event system and my other microservices were directly subscribed to the microservice's SQL-database ? That doesn't look great.

And if we are using SQL for rest of the microservices plus an Event Bus (let's say based on RMQ or whatever SB), we'd have kind of two different integration-events systems...
That is why I'd like to have a decoupled event-bus taking care of event-messaging no matter how microservices are implemented underneath (SQL, Event Store, NO-SQL, etc.).

Finally, if using ES in the ordering microservice plus the decoupled generic Event-Bus (RMQ, etc.), then, we're in a similar state than the original SQL Server based microservice in regards the integration events. You need to still assure consistency based on any technique like the "additional List/Table of uncommitted events" that I mentioned above and @davasquez-newshore also mentions.

My point is, sure, you could avoid having an Event Bus and use direct subscriptions to the event-store databases, like in a star topology with no central Event Bus (RMQ, etc.). However, that works well if all the sources of integration-events are based on ES (Event Sourcing with Event Store).
But, if we also have the SQL based microservices (heterogeneous world) as the source of integration events, then, we'd need a second/different mechanism for raising the integration events.
That's why, for a heterogeneous microservices application, I'd like to decouple the integration-event messaging channel from the microservice and their storage system (Event Store, etc.).

I should be subscribing just to "events" in a generic Event Bus instead being coupled to a specific event store or any other persistence database for getting the integration events.

I might not have gotten your whole story, am I missing anything?

[CESARDELATORRE]

@davasquez-newshore Yes, that "public list of integration events" that you mention I think is pretty similar to the approach I described in the guide when talking about Integration-Events.
It could be implemented as a SQL table when using a SQL based microservice, or any other list based on No-SQL or Event Store, as long as you can assure atomic consistency between that list and the original Aggregate's data.

See below extracted from the Guide/eBook.
I still have to refine the terms and add the implementation we'll end up doing, though.
"
You would use states such as a “ready to publish the event” state that you set in the original event when you commit it into the integration events table, and then you would try to publish the event to the Event Bus (based on queues or any bus implementation). If the publish event action succeeds, then you would start another transaction in the source or original service and move the state from “ready to publish the event” to “event already published”.
If the publish event action in the Event Bus fails, the data still won’t be inconsistent within the original microservice because it is still marked as “ready to publish the event”, and in regards to the rest of the services, it will be eventually consistent. You can always have background tasks/jobs checking the state of the transactions or integration events and if they find a “ready to publish the event” state, they can try to re-publish that event into the Event Bus.
Notice that with this approach you are persisting just the integration events per origin microservice, and only the events that you want to communicate to other microservices or external systems. On the contrary, in a full Event Sourcing system, you store all Domain Events as well.
Therefore, this approach can be considered as a very simplified event-sourcing system. You need a list of integration-events with their current state (ready to publish vs. published), but you only need to implement it for the integration events. In this case, you don’t need to store all your domain data as events in the transactional database as you would in a full event sourcing system.
"

Agree on what you also say about "great secondary effect of this is that you can unit test now your aggregate or your saga by checking the produced events or commands, while keeping the internal state private". That is cool. :)

[davasquezp]

Probably I was not going so far in terms of how to store those Integration Events before publishing.
What I meant with "public list" was simpler, in terms of just having an IEnumerable/IList in the Aggregate base class ( or interface). So every time you do some aggregate.Action(someParams) you would produce events in that List like "SomeStateChanged". This for unit testing and for infrastructure ignorance purposes.

You could easily mark those events with IIntegrationEvent and/or IDomainEvent and then your infrastructure code (after persisting I suggest) would publish the uncommited integration events to the cross-boundaries event bus and the domain events to some internal bus/coordinator (inside the boundaries) if you would like to do so. No really need to persist them, but possible.

[CESARDELATORRE]

@davasquez-newshore
I think we're talking about almost the same thing:
Somehow, you'd need to persist that list of events, too, you mean as part of the AGGREGATE it self, right? other than that, if there's a failure before publishing any integration event to the Event Bus , you would have an inconsistency.
But having another event list/entity as part of the Aggregate means kind of the same thing I was saying, you'll have an additional table underneath (if using EF) that would persist those events, then mark them as committed or published and act accordingly.
The only difference might be that you suggest to have a list of every domain event and then mask it as domain-event or integration-event.
When using domain-events, that approach sounds pretty similar to the following, suggested by Jimmy Bogard.
https://lostechies.com/jimmybogard/2014/05/13/a-better-domain-events-pattern/
It would just be "extended" with "marks" like : IntegrationEvent and/or DomainEvent

My only doubt is if an integration event could be composed by multiple domain events instead of simply "marking" a domain event as integration event, as well. In that case we'd need a separate list of integration events.
Very similar approaches, in any case. But could be simpler and interesting to just "mark" the events within the Aggregate instead of further complexity that might not be needed.. Good feedback!
We'll brainstorm about it and decide, but it will be pretty similar. :)

[paulopez78]

@CESARDELATORRE I understand the idea of having an abstraction for consuming asynchronous events the same way HTTP is an abstraction for sync communication and not coupling to any storage technology, that's clear for me.

That abstraction will have probably a rabbitmq implementation or other message broker technology, event store can be also implemented as an abstraction or kafka for example. As as consumer you only want to implement a basic subscription interface that behind the scenes connects you to the broker using TCP, for example event store competing consumers and rabbitmq pub/subscribe have exactly the same interface when used in integration scenarios so they could be replaced easily.

But the main difference between them is the order and the history of events, rabbitmq is not and event log, event store and kafka goes more in that direction so they can be used for implementing event sourcing or event streaming where you can reply all the events to all the subscribers and them can rebuild their projections of the data, because the source of truth is in the event log, that's for example how you can have a database replica and you can recreate it over and over. I think a good abstraction for consuming this kind of event streams could be Reactive Extensions where the other microservices subscribes to an IObservable stream of events.

Anyway in integration scenarios like you describe I think you can mix both messaging systems, message broker and event log and use one or another based on your use case. An as I mention the 2PC is key here and that's an incredible advantage comparing to SQL + RabbitMQ, even you implement a custom solution with states and a job publishing events on the background, that will create much more contention than writing to an event log. To sum up from my point of view event store could be seen like SQL + RabbitMQ but not made to be queried if not consumed like an stream of events creating projections aka as CQRS.

I'd like to share my favourite references about that topic here:

• Really good presentation about event sourcing:
  https://www.youtube.com/watch?v=avi-TZI9t2I
• Really good repo about event sourcing with message broker and .net core microservices : https://github.com/AdaptiveConsulting/ReactiveTraderCloud

Thanks for all this feedback, hope to contribute with code soon :)

[CESARDELATORRE]

@paulopez78 Thanks for your input. All your feedback is really interesting. :)
We understand ES and event source. I've been in many great discussions about it with Greg Young, like when we created the P&P CQRS Journey, also at DDD eXchange, etc. So, I'm sold on the idea of using ES and event store for certain scenarios where a history and a "event replay" is important.
Also, thanks a lot for your references, anyone reading this thread would benefit of it.
I didn't know the ReactiveTraderCloud. It looks awesome. :)

However, what is not clear to me in what you're telling is (and related to the eShopOncontainers context):
A. Are you saying you would use an Event Store database for implementing ES just for the Ordering microservice to store its "source of truth" data (Aggregate's events) ?
or
B. Are you saying that you would implement the central Event Bus with Event Store as a central storage of all the integration events while supporting Pu/Subs instead of RabbitMQ or any other higher level Service Bus?

In your original emails, I thought your were saying more about CASE A, but I'm not sure now with your last response.

We'll have further brainstorming about this subject and I'll come back here with our conclusions, ok?
Thanks a lot for your feedback and thoughts! :)

[paulopez78]

@CESARDELATORRE I know your strong background in DDD and ES that's why I like to hear your feedback because is hard to find real experience in this kind of architectures.
As you pointed out, my first thought was the option A, use event store locally in the Orders microservice and have a consumer on the same event store publishing to rabbitmq the integration events.
But eventstore is a message broker itself so why not to use it removing the need of another moving part like rabbitmq, so that lead us to the option B. I understand that has a bigger impact than option A and you've already chosen a broker technology for solving that problem.

Anyway I will be looking forward to hearing from your feedback and let me know if you need any other information or help. Thanks and great job with that repo!

[mvelosop]

Closing this as conversation can be considered finished!

Making an internal note to consider this in the roadmap.