Axonframework: Allow for Ephemeral Events

[Blackdread]

In axon, events are always stored in Event Store and persisted for forever.

It would be useful to have an annotation (something like @TempEvent) that is put only on classes.
Only EventHandlers can receive this event (aggregate/saga do not receive as those events will be deleted in the near future and so cannot be used in CQRS/ES to re-build state of aggregate/saga).
Events with the annotation:

• would be persisted in event store and distributed to event handlers
• deleted from event store after the specified time specified in data of annotation

A microservices that restarts may receive again those events if those were not deleted already in event store.

Of course it could be possible to manually put a condition on timestamp of event in handlers and skip the event if older than X time.
The main benefits of this approach is to have events propagated like ordinary events but would be deleted later and forgotten completely from the event store -> gain in space and faster replay as those events will be deleted and not replayed.

[Polish-Civil]

I think that this might add accidental complexity to many of the projects, I also think there are better approaches to manage the performance rather than just deleting events. It kinda breaks purpose of axon and it's event sourcing route.

[Blackdread]

It might add complexity but depends on what.
At runtime (boot time), the class that handles registration of event handler methods can check that the event with @TempEvent is not within a @Aggregate (and entity etc) or @Saga class, simply throws an exception to cancel boot.

What are your different approaches ?

Not sure it breaks purpose of Axon, it is after all only a new type of events that are handled a bit differently by the EventStore and the route logic.

The benefit of this kind of short lived event is that it is still routed by Axon instead of using a custom Kafka/Akka/RabbitMq solution (which brings other issues like setting up server, replicas, localisation, more config, etc).
No need to have different use of annotation/classes/etc to be able to receive events from other bus than Axon.

[Polish-Civil]

Problem with that for me is that i cannot really find any use case of such feature, we could discuss it further if you could give me some hint on how would you use such.

You mention the performance gains so i'm guessing this feature would be non-trivial one as it's main purpose is to optimize performance so i'm assuming that you have alot of those kind of events.

Well it might not necessary break axon purpose but it does break the EventStore / Event sourcing and thoose concepts are driving the core concepts of the axon framework.

It looks like it brings the complexity by having additional type of event which life cycle is somewhat quantum state (it might be there but who knows), everyone that would jump into axon, could get confused without explicit knowledge what are those events and what are the use cases, also following that the developers would need to know how to use those events.

To summarize, without any real world use case presented that might be solved using your approach, i'm not sure that axon would consider such.

(But that's just me :), i might not see what you do)

[Blackdread]

Use cases are many.
These are still events, just "short lived", axon can at runtime check if registered event listener are not using those short lived events in wrong part of code (aggregate/saga/etc).

With this feature you do not need to have your own Kafka/Akka/RabbitMq/etc which brings many other issues to take care of.

Ping @smcvb

[smcvb]

We have been having discussion on stuff like this, as it becomes more common place to have events as pure notifications of things happening in the system, which aren't your day-to-day domain event.
Thus, they are not required to drive business logic latter on, just as a trigger for certain actions or logging statement.

Whether this will be an annotation as an interesting concept, as my personal angle on this was a little different. Regardless, the exact implementation is not important just yet.
I'll ensure we'll investigate the idea further, @Blackdread, within the AxonIQ team.

Thus thanks for filing the issue. I can assure you that anybody tracking this issue will be kept up to date.
Feel free to further discuss the sense or non-sense of the issue and potentially ways to tackle it (maybe even with a PR to support the approach).

[Polish-Civil]

Just to clarify my criticism,

There are few things that @Blackdread say which strike me

deleted already in event store

forgotten completely from the event store

The EventStore and EventSourcing main concept is that the events shall be persisted forever. They are truth of everything, therefore it breaks the use case of those components.

However, as @Blackdread mentions, we would not need to implement the Kafka or any other MQTT capable components to manage these kind of events. This makes me thinking that the main reason here is to make axon all-in-one framework that would be capable of handling separate type of events without external components.

I still think that clear use case scenario might help out to decide the implementation detail and i strongly disagree with saying that event may be deleted from event store.

[Blackdread]

I understand that you may not want to delete events.
Maybe there could be more feature required, this needs to be split.

Something like to allow to delete events after a certain amount of time => one annotation -> this fits some use cases -> when you only care about sending events that have no business logic but more like information purposes or real-time dashboard (do not care about history).
Annotation is put on event class, runtime refuses to boot if event class is used in any aggregate/saga/etc.
Is there any way to check if a previous event that is in event store was used in aggregate/saga/etc? If yes then at bootime, it should check and throw to make sure we can not do a mistake of deleting events that were previously used for an aggregate/saga/etc (but I think it should be the responsibility of developers and not the framework). It comes with high risk but again developers responsibility I think.

Another annotation, maybe not on event class anymore but new one (inside EventListener or just put side by side on each method) that "filters" events based on timestamp so method does not receive "old" event (or filtered events (not only time based filter) ->would make the annotation more complex as it will evolves in a complex annotation of filtering but is useful)
Filter annotation is forbidden as well in any aggregate/saga/etc and cancel boot (runtime check).

I think Axon needs to be more than CQRS/ES, it has great features of abstracting location, duplication, scale, etc so less need of Kafka/etc

[Polish-Civil]

@Blackdread Could you elaborate more on the use case and focus on why the event need to be persisted for x amount of time instead just published? Does it have to do with the delivery mechanizm or the replays.

You mention the real time dashboard, which, well, I'm doing atm using axon's existing features and the 'volatile events which, are just notification of things happening, so they are not persistent really, they are just there to notify you about that something happened so client can update the view accordingly. I call them query subscription events

I think that issues with how to annotate this and ensure that developers use such events properly are trivial at this stage. I would really like to continue on trying to find clear use case of such that cannot be done with current features of axon or provides more clear implementation for the things that people are doing with axon currently.

The only thing that comes in mind are these real time systems but all i can see is client-server integration so i'm just using subscriptions because they are oriented on the query model because that's what client wants - the data and updates on that data.

100% of the events i have in my system are driving my logic, so they need to be persisted in order to recreate the state of the application.

It seems like we are touching the point of the axon's current implementation of the query system and it's subscription capabilities.

1. You can define what type of query you are interested in.
2. You can define what type of data you are going to receive on the subscription (lets call it subscription event for simplicity)
3. You can drive your updates based on these kind of events

The only point missing here is the 'temporary' persistance, so im not really sure what would be an use case of it. Another thing with subscription queries is that they should not be used in your domain layer (they are part of query layer) so they cannot drive anything that can do changes in your domain, however they can be used by application layer just fine.

So what exactly the impact of the system by such volatile event would be? In what layer this event resides in? What can it control? What it is controlled by? Where controller of it lives?

I'm sorry that it might seem like im really not a big fan of this feature, I like the concept of it but i also see alot of issues that it might bring if not put in the proper context.

[Blackdread]

I understand that query subscription event could be used but it might
be too specific to the query (or even queries that want same information but display that info differently)

One simple example would be to have different http filters in spring, one to send event for each incoming request with data like url, controller called, current user authenticated or not, ip, params, headers, etc.

Just this example shows that you might end up with millions of events every day that the business logic do not care to keep "forever" and do not need to have replay capabilities after x time (one week, one month, etc).

There are many tools that can achieve as well percentiles display of API, etc (Prometheus, Graphite, etc) but sometimes it is easier to send events.
Same event can be used in different place but with query subscription, the producer needs to know all different queries and send appropriate update for it whereas with event producer do not need to know who uses this events, neither how.

I will try to describe more when I have more time.

[vab2048]

If it adds any value there was a somewhat related discussion on the google groups regarding this here: https://groups.google.com/forum/#!topic/axonframework/UGe0mjUMcrA

From my perspective I agree with OP that the ability to specify an event as "expiring" or "to be deleted" after a certain amount of time would definitely be handy for those situations where you want to distribute generic events which do not relate to an aggregate/saga/event-sourcing through axon server (rather than having to go through the headache of additional mq infrastructure).

[smcvb]

It definitely adds value @vab2048; more insights can help for a better judgment when it comes to implementing this feature. :+1:

[msavy]

I have an interesting use-case that touches on this.

We ingest business-related emails and store them in the event store -- this is handy as we can replay them later to pay attention to stuff we didn't look at previously. This step is a 'private/internal' event for that service, because other parts of the system would never consume the raw email (they would consume only a normalised event or view).

However, as will likely not surprise you, a certain amount of the email turns out to be junk. People sending non-business stuff, spam, circulars, etc.

For those cases, it would be good if we could zap entries labelled as volatile. For example, the operator identifies these entries as spam (i.e. they are definitely useless); we would then remove associated aggs/events from the event-store, as they have no possible useful purpose. They are just occupying space without being helpful.

In my mind, the volatile descriptor kind of makes sense for this use-case also; it's an event/aggregate that may not continue existing for legitimate business reasons.

May be law of unintended consequences caveats associated with this, I realise :-). Perhaps snapshots could be rebuilt, though, in simpler cases?

[smcvb]

For followers of this thread, we're currently required to move the release of this feature from 4.4 to 4.5.
This follows from changing prioritization at our end which made it so that we cannot resolve this implementation in a timely manner.
Thanks for your understanding and we will keep everybody posted under this thread as soon as work commences.

[smcvb]

Adjusted priority to a "should" and removed milestone. We envision this issue to require an API adjustment which is in line with the requirements of #1106. Namely introducing another Message Source which interleaves with the regular Streamable/Subscribable Message Source. As #1106 its priority has been adjusted and it intertwines with the requirements of this issue, it is only right to adjust this issue's priority too.

Furthermore, it is not feasible that this will be resolved before the desired 4.5 release schedule, hence the milestone removal.
This issue will be reevaluated for the following release.