Axonframework: Significant performance degradation in event propagation to projections with Axon Framework 4.4.1

[dmurat]

After upgrading to Axon Framework 4.4/4.4.1, I'm experiencing performance degradation with events propagation to projections. It is about 500 ms slower than with the 4.3.5 version, where it was around 20 ms.

As I also upgraded to Axon Server 4.4, I tried several combinations of Framework/Server versions trying to pinpoint the source. According to my observations, it looks like the Framework is problematic since, with Server 4.3.5/4.4 and Framework 4.3.5, there was no degradation.

For diagnostic, I'm using distributed tracing with Wavefront UI and simple comparison of timestamps in logging output.

[smcvb]

Thanks for filing this concern with use @dmurat. To be honest, I am guessing it is the new AxonServer Connector which has replaced the majority of work initially done by the framework itself to form the connection. Interestingly enough initial validations didn't uncover a gap of 500ms compared to the previous approach. Regardless of where the issue lies, this deserves a better look of course.

Slightly upfront, but would you be able to share thread dumps with us on your validations?

[dmurat]

Maybe I can do better than that :-)

Here is a project where you can try yourself: https://github.com/croz-ltd/klokwrk-project. Starting up applications is described in https://github.com/croz-ltd/klokwrk-project/blob/master/support/docs/startingUp.md.
Axon related dependencies can be changed in https://github.com/croz-ltd/klokwrk-project/blob/master/gradle.properties, namely axonFrameworkVersion and axonServerDockerImageVersion.

If you like, we can even arrange some skype/zoom call for me to demonstrate the issue.

[smcvb]

Thank you very much for the sample project @dmurat!
I'll check it out myself, seeing what the results are.

As soon as I have new information I will update this ticket accordingly.

[smcvb]

I believe I've been able to replay the behaviour as you've described it.
Thus, with framework 4.3.x everything is fine, whilst with 4.4 it turns out 500ms slower at virtually every occassion.
As I've not used Wavefront before (assuming this is where you've grabbed your info from), neither have any of my peers at AxonIQ, I thought it wise to get a showcase from your end to be a 100% sure that what I am seeing is the same as you are seeing. Or, are you just validating console output between your command and query applications?

Additionally, I have a hunch what the issue might be on our end. Just need to validate _it_ is the culprit. Hence why I am looking for validation if I am checking those results in a similar fashion as you are. Please contact me on [email protected] so that we can discuss this further in a video call.

[smcvb]

Closing this issue as it has been resolved in pull request #1484.