Axonframework: Integration with Spring Sleuth

[meong1234]

it nice if axon provide integration with distrubuted tracing use spring sleuth
https://cloud.spring.io/spring-cloud-sleuth/

[meong1234]

`@Aspect
public class TraceEventHandler {

private static final String EVENT_HANDLER_COMPONENT = "eventhandler";

private final Tracer tracer;
private final TraceKeys traceKeys;
private final Pattern skipPattern;

public TraceEventHandler(Tracer tracer, TraceKeys traceKeys, Pattern skipPattern) {
    this.tracer = tracer;
    this.traceKeys = traceKeys;
    this.skipPattern = skipPattern;
}

@Around("execution (@org.axonframework.eventhandling.annotation.EventHandler  * *.*(..))")
public Object traceBackgroundThread(final ProceedingJoinPoint pjp) throws Throwable {
    if (this.skipPattern.matcher(pjp.getTarget().getClass().getName()).matches()) {
        return pjp.proceed();
    }
    String spanName = SpanNameUtil.toLowerHyphen(pjp.getSignature().getName());
    Span span = this.tracer.createSpan(spanName);
    this.tracer.addTag(Span.SPAN_LOCAL_COMPONENT_TAG_NAME, EVENT_HANDLER_COMPONENT);
    this.tracer.addTag(this.traceKeys.getAsync().getPrefix() +
        this.traceKeys.getAsync().getClassNameKey(), pjp.getTarget().getClass().getSimpleName());
    this.tracer.addTag(this.traceKeys.getAsync().getPrefix() +
        this.traceKeys.getAsync().getMethodNameKey(), pjp.getSignature().getName());
    try {
        return pjp.proceed();
    }
    finally {
        this.tracer.close(span);
    }
}

}`

example by use anotation @EventHandler

[jorgheymans]

just a thought, IIRC axon3 supports dropwizard, so you might be able to get something working combining https://github.com/opentracing-contrib/java-dropwizard , https://github.com/openzipkin/brave and the zipkin connector of spring cloud sleuth https://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html#_spring_cloud_sleuth_stream_zipkin_collector

[dzonekl]

Hey Guys,

We want to have a go at this. The impl. I am thinking of:

On a: QueryGateway / CommandGateway:

A MessageDispatchInterceptor, which gets Tracing injected, from which we get the current span, and add it as a MetaData to the Message.

Handler interceptors

A MessageHandlerInterceptor which gets Tracing injected , from which we spin a new child span.
This could potentially be a CorrelationDataProvider, so it maps the MetaData to other messages.

I notice CorrelationDataProvider is registered on the commandbus in AxonAutoConfiguration, and in EventHandlingConfiguration. Is it correct these are the only two places, where MHI can be registered?

Netty/Grpc: (Axonhub specific or distributed in general).

The MetaData could potentially also be written/read has b3 headers in Grpc or HTTP/1 communication. Not sure if happens by default somehow, or an Trace injector/extractor should be used.

Enabling the headers, could let other tools use the tracing information. (In our case we will use Spring Cloud Sleuth to send the tracing data to Zipkin).

Finally, some auto configuration could be created, to enable the Axon instrumentation with a good set of defaults.

I hope, for your feedback on this approach...

[smcvb]

@dzonekl on face value that doesn't sound to off for me to be honest.
Mainly posting a response here to notify you that we'll look in to your suggestion, but time-frame-wise I am hard pressed to tell you when that response will come.
We have to focus on our 4.0 deadline for next week Friday first, after that, I assume things like this will be fair game again.
So, that's my heads up for you. Thanks for showing interest of course!

[dzonekl]

Hi Steven, Thanks for the head-up.
I am actually working on it, made some good progress ... At current stage I implemented the
Dispatcher and Handler and an Injector/Extractor which puts traceId, spanId and parentSpanId into MetaData. This is propagated between VM's using AxonHub, so it already kind of works.
It's a rough cut, have time to continue on it now, and I will likely offer this as a PR.

Below you see two span's between two axon apps, separated by Axonhub.

Cheers Christophe

screenshot

[dzonekl]

@smcvb question, where does the tracing code fit best? As an extension or as part of the framework? There will be deps. on opentracing.io

[smcvb]

We'd very much like this to be part of our new 'extensions' idea, so yes, making it a separate repository to contain the code would be very nice. Might be worthwhile for you to start it of as a personal repository, for which we can discuss the migration (if you want to migrate ownership to us that is) at a latter stage.
Would that work out?

[dzonekl]

Sure it will, it's actually easier, are API's for extensions? What I would otherwise do, is to do an Autoconfigure, conditional on opentracing or sleuth dependency, and then register the various handlers. BTW, for the event processors, registering a handler in a generic way is tricky. The processor group names are not available at application configuration time. (They are resolved later) .

What I would like to do is this method in a @Configuration class.

    @Autowired
    public void registerHandlers(EventProcessingConfiguration config, Tracer tracer){

        OpenTraceHandlerInterceptor openTraceHandlerInterceptor = new OpenTraceHandlerInterceptor(tracer);
        Map<String, EventProcessor> stringEventProcessorMap = config.eventProcessors();

        // map is empty! 
        stringEventProcessorMap.keySet().forEach( processorName ->
                config.usingTrackingProcessors().registerHandlerInterceptor(processorName, configuration -> openTraceHandlerInterceptor)
        );
    }

[jorgheymans]

@dzonekl just curious, is this an integration with sleuth or opentracing. They are not the same. Sleuth uses brave under the hood and is able to bridge traces to an opentracing backend.

[dzonekl]

@jorgheymans yeah, I am aware they are different and I have a few tested.
There are actually more combo's (as far as I know).

• sleuth -> zipkin/brave (tested, working).
• sleuth -> Jaeger (Not tested)
• opentracing -> jaeger (Testing as we speak).
• opentracing -> zipkin (Tested working, Which uses the brave bridge as you mentioned).

The focus is on opentracing/jaeger combination.

Rgds.

[dzonekl]

@smcvb , I am able to do a trace for a single request on a gateway (Either command or query), unfortunately when chaining requests in an async flow, the tracing scope gets lost.

Typically when making a client call (In this case calling .send() or .query() ), a child span is created which needs to be finished() when the request is done.

With the MessageDispatchInterceptor I can populate the Metadata with the span context, but I can not wrap the request so that the child request is finished as discussed here:

https://opentracing.io/guides/java/spans/ => "Moving a span between threads"

So what would be needed, is to wrap the query or command on the gateway as part of the code which executes the request. What are the options?

I can think of extending the Gateway and do something with the FutureCallBack, any other hooks into the framework to wrap a Gateway request?

[dzonekl]

@smcvb fyi, problem is clear, the scope is maintained in ThreadLocal, as we have an async flow of completable futures, we loose the scope [1], Do you see an option to hook into the thread executor services used by Axon, to somehow propagate the content of Threadlocal?

[1]
2018-10-25 16:56:54.802 INFO 42987 --- [nio-8080-exec-1] i.j.internal.reporters.LoggingReporter : Span reported: c1362a2611982392:427564ce0d87ead4:c1362a2611982392:1 - query 2018-10-25 16:56:54.802 INFO 42987 --- [nio-8080-exec-1] e.b.e.tracing.gw.TracingQueryGateway : Trace id after call: c1362a2611982392:c1362a2611982392:0:1 2018-10-25 16:56:54.893 INFO 42987 --- [ault-executor-0] e.b.e.tracing.gw.TracingQueryGateway : Thread: Thread[grpc-default-executor-0,5,main] 2018-10-25 16:56:55.130 INFO 42987 --- [ault-executor-0] i.j.internal.reporters.LoggingReporter : Span reported: 3e59f3b881f11944:3e59f3b881f11944:0:1 - dispatch-message 2018-10-25 16:56:55.869 INFO 42987 --- [nio-8080-exec-2] i.j.internal.reporters.LoggingReporter : Span reported: c1362a2611982392:c1362a2611982392:0:1 - createAssetDefinition

[smcvb]

I am not entirely sure about the 'wrapping issue' your sharing here @dzonekl. The thing I get from your description is that you have send a command, and you want to populate the information from one (type of) message to a new (type of) message your publishing on the Command/Event/Query bus.

Axon internally uses the CorrelationDataProvider mechanism to move a correlation ID and trace ID from one message to another. It might be worthwhile to check wheter you can use that mechanism to give the span context over from one point to another.

[dzonekl]

@smcvb, there are/were two issues.

1. How to get a new child span context into the Metadata.
   2.How to preserve the span context, when chaining async requests (Or calls to the Command or Query Gateway).

The first issue, I solved with a custom TraceQuery and TraceCommand Gateway.
The 2nd issue, is tricky, this is the one mentioned in the email I have send you, and here [1] is the link again here. It seems this requires a customisation of Axon when producing Completablefutures, with a custom executor service. (In similar fashion as how SLF4J solves the MDC issue, propagating ThreadLocal, will look for more info there and be back ...).

[1]
https://github.com/opentracing/opentracing-java/issues/318

[abuijze]

While going through older issues, we noticed this one has been superseded by the OpenTracing Extension for AxonFramework (see https://github.com/AxonFramework/extension-tracing)

If any features are missing in that module, please raise an issue there.