Axonframework: Is (Storing events + publish events) Atomic in UoW?How?

[mehdichitforoosh]

Hi.
Based on Sending Events to Event Bus post in Google Groups.

I think the best way to control the atomicity is having an Outbox table or transaction log tailing in DB , EventBus can simply poll the table for new events and remove events from outbox when published.
what is your opinion?
Thanx.

Transaction log tailing
microservices-aggregates-events-cqrs
Using sagas to maintain data consistency in a microservice video '43:00'

[mehdichitforoosh]

Excuse me, I did not really understand how Axon control atomicity with attached Transaction(SpringTransactionManager,...) in UoW.
What happens if the system fails or eventbus fails before events are released?
1.EventStore success
2.publish & consume events failure

Thanx.

[abuijze]

Hi @mehdichitforoosh,

please do not use the issue tracker for questions. You can post questions on the mailing list: [email protected].
To briefly answer the question: appending an event to the log is one operation. Handling it is another. If handling fails, you either retry or skip... Furthermore, each handler has its own transaction on the consuming side. Also, "consuming" doesn't remove the message from the queue. Consumers are "free" to revisit streams if they wish.

[mehdichitforoosh]

Hi @abuijze
I agree with you.
Let me say another way:
One solution to this problem is to add an extra column to the DOMAIN_EVENTS table(DomainEventEntry) that tracks whether an event has been published. The event publisher(EventStore) would then use the following process in Retry Scheduler(polling):

1 Find unpublished events by executing this SELECT statement:
SELECT * FROM DOMAIN_EVENTS where PUBLISHED = 0 ORDER BY event_id ASC
2 Publish events to the message broker or an event bus.
3.Add a callback for example onSuccessPublish() to EventStore or call a method markAsPublished() on EventStore.
4 Mark the events as having been published when callback is called:
UPDATE DOMAIN_EVENTS SET PUBLISHED = 1 WHERE EVENT_ID in(?,?,...)

reliable-messaging

Do the developer must do this?or It should be in the Axon Framework?

[OLibutzki]

I guess this question reflects what I recently discussed in the forum.

The issue arises whenever the event store and the component which emits the event cannot take part of the same database transaction. The outbox pattern closes this gap as it publishes the event to the event bus asynchronously.

The corresponding thread: https://groups.google.com/forum/m/#!topic/axonframework/_VNvbIATXj4

[mehdichitforoosh]

Hi.
@OLibutzki I agree with you.
I solved this problem by seprating Event store from Event Bus.
1.first transaction persist events or entity (Without publish events)
2.a scheduler calls rePublish() method in EventBus and then polls event store or Outbox table.
3.publish events to (RabbitMq,Kafka,...) and then calls markAsPublished() in EventBus when second transaction successfully commits.Best Efforts 1PC

I think Axon Framework ultimately needs to use this method to be reliable.
Of course this is my opinion.
@smcvb @abuijze
Thanks.

[smcvb]

What @OLibutzki signals with the Outbox Pattern is quite identical to having a separate process poll for new events being published.

Reading that, this seems to be exactly what the TrackingEventProcessor does.

The thing you're pointing too, @mehdichitforoosh, is thus already implemented by the framework if you'd ask me. You should just refrain from utilizing the SubscribingEventProcessor for guaranteed message delivery, is that simply doesn't provide it.

Note that the Kafka extension is a StreamableMessageSource solution, thus suitable for a `TrackingEventProcessor.

The nature of RabbitMQ in the AMQP extension is however such that it would require quite some custom code to be a polling mechanism. Hence, it is implemented as a push solution covered by the SubscribableMessageSource, suitable for SubscribableMessageSource.

As already discussed, the SubscribableMessageSource does not give you the security you're looking for, @mehdichitforoosh. I'd thus suggest you move away from that pattern if you require ensured delivery.

So, to reference your comment just now:

I think Axon Framework ultimately needs to use this method to be reliable.
Of course this is my opinion.
I believe the framework already does this, this conversation just started off in the wrong direction.

If any of my description above does not fit the explanation you, @OLibutzki, or you, @mehdichitforoosh, have provided, then please correct me.

[mehdichitforoosh]

Hi.
Thank you @smcvb
Ok.I'll check TrackingEventProcessor and we will discuss about it.
Good luck.

[smcvb]

No problem @mehdichitforoosh, we're here to help!
Hope it covers what you're looking for.
And, as pointed out, if it doesn't, please do not hesitate to reach out.

I do want to ask though that questions like that will be posed either on Stack Overflow with the axon tag or on our User group.
We'd like to keep Github a place for features, enhancements and bugs mainly; hence the question.

[OLibutzki]

Hi @smcvb,

thanks for answering.

What @OLibutzki signals with the Outbox Pattern is quite identical to having a separate process poll for new events being published.

Reading that, this seems to be exactly what the TrackingEventProcessor does.

Exactly, it's similar to what the TrackingEventProcessor does. But there are slight differences:

• It's possible that events are processed by multiple TrackingEventProcessors.
• In an outbox a message/event is processed exactly once. There is a single tracker which is responsible for:
  
  
  
  1. detecting new outbox entries
  
  
  2. publishing them to the event bus
  
  
  3. and deleting deleting the outbox entries or marking them as read
  
  

In my opinion you need both: The TrackingEventProcessor and an _OutboxTracker_ (or whatever it might be called).

I do want to ask though that questions like that will be posed either on Stack Overflow with the axon tag or on our User group.

Sure, that's what I did in the first place. I just stumbled upon this issue which seemed to be similar to my request.

[smcvb]

Thanks for your added feedback @OLibutzki, I'll see to this suggestion will be discussed at on point or another.