Axonframework: Support Kafka as EventBus

[jangalinski]

I just recently learned that my approach of connecting command service and query service via rabbitMq won't allow me to use replay on query side due to the nature of the rabbitmq.

So my feature "request" would be to support kafka as an eventbus (and also store). Are there any plans so far?

[jangalinski]

see https://groups.google.com/forum/#!topic/axonframework/9QS-NO8Cdi8

[hakamairi]

Hi,
@abuijze you mentioned that kafka support is planned in the ticket https://github.com/AxonFramework/AxonFramework/issues/401
I assume that the 375 (this) ticket is where the magic should happen. Is it?

What's the progress on this? Do have it on the road map? Which version is planed to support this?

[abuijze]

Hi, you're right: this is the ticket to track for the implementation.
So far, we're not planning further than the next release (3.2). I don't expect this feature to be part of it. Main reason is that we haven't started investigating the best way to provide support for Kafka, just yet. We expect to be providing some form of connector (forwarding published events from the event bus to Kafka) as well as a KafkaMessageSource, similar to the SpringAMQP components that are already there.
We don't expect to implement a KafkaEventStorageEngine, as the design of Kafka doesn't seem as fit for Event Sourcing as we initially thought (see https://www.youtube.com/watch?v=ipBb29xnFb8).

As soon as we schedule this ticket for a release, we will set the milestone on it. Until then, we obviously welcome PRs. ;-)

[nklmish]

I can take this issue, already started working on adding Kafka as an Event Bus .

[ghost]

Where are you with this one? Any plans to see Kafka Event Bus any time soon?

[nklmish]

@petriapaf Publishing and consuming messages from Kafka is done. In upcoming days, will focus on consumer rebalancing and refactoring. Hopefully we will have it soonish

[caindy]

@abuijze the YouTube link appears broken. Is the rationale for "the design of Kafka doesn't seem as fit for Event Sourcing" documented elsewhere?

[smcvb]

@caindy I can't point you to the right YouTube vid (as I'm unsure which Allard was referring too), but this read why Kafka is not for Event Sourcing is a good one to check out as well.

[siamak-haschemi]

Hello, I still thinking kafka can be used for that. Just that „event-sourcing“ has to be interpreted as publishing Snapshot-events.

When a command handler fires an event, this event can be added to kafka by a kafka producer. The event however would contain the the status of the aggregate, as well as the original intention (command-name, command-id).

This aggregate snapshot event can then be consumed by a consumer group to persist the last status of the aggregate. We could use a DB, or Kafka Streams KTable for that.

The command handler then would load the repository object from this DB using repository.load(...).

Strictly speaking, that’s not the „event-sourcing“ definition of Axon, so maybe you would not use Event-Sourced Aggregates.

The benefit however of this approach is, that you rely on kafkas durability, and low-latency.

Just my 2 cents

[smcvb]

Firstly I'd like to point this is not exclusively Axon's definition of Event Sourcing.

Secondly, the suggested solution is, as you point out, not really Event Sourcing either. This approach seems to use Kafka a a state storage mechanism, but rather then storing it once you are storing the entire Aggregate on each command being handled + the actual command. I'd argue that state storing the aggregate is not identical or resembles Event Sourcing, as you're now not working of of deltas/events but complete possible versions of the end product.

Taking these suggestions, that thus not make it an EventStore, but a regular aggregate Repository.

I trust in you that the solution you sketch could work as a Kafka version of an aggregate Repository.

We do agree that Kafka has very nice benefits we can and should leverage though, don't get me wrong there. That is exactly why as of Axon 3.3 (in issue #573) you can use Kafka as a StreamableMessageSource. Do note that is thus still not providing it as a Kafka version of an aggregate Repository however, but as a way to receive events in a distributed set up.

[smcvb]

Closing this request as the PR #573 introduces Kafka as a StreamableMessageSource into Axon 3.3.

[fredgate]

Axon 3.3 provides support for Kafka as an event bus.
But adding support for Kafka as event store will be a great feature. Problem is performance to load an aggregate from its identifier as topics in Kafka should be by aggregate type and so include events of all instance of this aggregate type. But to solve this problem, we can load current state of an aggregate from kafka local store : this is a kind of snapshot.
Example in this article : https://dzone.com/articles/event-sourcing-with-kafka

It would be great to have complete Kafka support as it is powerful in a microservices architecture, as detailed in this serie : https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and-services/

[smcvb]

Thank you for your input @fredgate, I can assure you we will take every suggestion serious.
We currently still remain with the stand point that Kafka is a great event bus, but not so much a great solution for event sourcing, as is for example pointed out here.

If anything within this space changes so that it makes sense to introduce an Aggregate Repository implementation like I commented earlier, we will definitely open up this ticket.

[fredgate]

Tanks @smcvb for your reply.
I knew the article that you mentioned, but it is quite controversial and not necessarily impartial as it was written by the author of serialized.io, a potential competitor of Kafka.
The main problem with Kafka is to load current state of an aggregate from its past events. Kafka will not work well if a topic is created by aggregate, so the best solution is to create a topic by aggregate type. So to retrieve events of an aggregate, we could browse all events of all aggregates and filter these belonging to the specified aggregate. Of course its not pratical. The good alternative, that is not suggested in the article, is to build a local table from the stream of events and so to directly have current state of aggregates.
So a implementation of storage for Kafka could append events to a stream and retrieve current state (but not past events) from the local table.

Kafka has so many advantages like :

• write / publish atomicity
• streams : easy to add new services that can replay events
• exactly one guarantee
• public / private topics
• schemas (avro)
• services load balancing...

[shafqatevo]

I fully agree with @fredgate . I think this is a huge opportunity loss not to use Kafka as an Event Store, too, taking advantage of Kafka Streams and KSQL.

In 2018, Kafka became more suitable as an Event Store: please see this, this and this.

Please also see the following comments on the outdated post that was referred to as a basis of the view that Kafka is not suitable as an event store:
https://medium.com/@sergey.pichkurov_78659/hi-jesper-663aed571009?source=responses---------2--------------------- and
https://medium.com/@jimstone_75945/hi-jesper-632ba52090b6?source=responses---------3---------------------.

[smcvb]

Thanks for your @shafqatevo.
As time moves on, we understand that Kafka might eventually become more suitable to be used as an Event Store. I will comment similarly to you as I did to @fredgate: we definitely take suggestions like this from our community serious. All the information shared thus far is worthwhile to make this decision.