Spring-cloud-sleuth: Feature Request: Supporting custom headers to be traced along with X-Trace-Id & X-Span-Id

Created on 29 Mar 2016 · 19Comments · Source: spring-cloud/spring-cloud-sleuth

Currently, tracing X-trace-id and other related headers get traced just fine between applications (provided all are sleuth enabled). It'd be nice to have the ability to add custom headers in application.properties such that they can be traced at the entry and the exit point of a given application. This ability will help under scenarios where a spring sleuth enabled application interacts with legacy applications which manually track some specific headers and the spring sleuth enabled application could track those same headers without really passing them through across all the method calls manually.

enhancement

Source

shivangshah

Most helpful comment

We will be working on this feature as the next big feature in Sleuth. It's often referred to as a Baggage. We are aware that it's an important one :)

marcingrzejszczak on 8 Jul 2016

👍4

All 19 comments

marcingrzejszczak on 30 Mar 2016

This is sometimes referred to as baggage. Basically propagating other
values in the same context.

codefromthecrypt on 31 Mar 2016

Isn't this already supported via spring.sleuth.keys.*.headers?

dsyer on 31 Mar 2016

good question.. @shivangshah are you asking to have custom headers show up in the zipkin ui?

codefromthecrypt on 31 Mar 2016

@adriancole actually no. I haven't actually used zipkin ui yet. What we needed is a way to somehow specify custom headers so that they can be traced through entry/exit points (via Client Interceptors and Servlet Filters) by Sleuth. These headers can be dynamically picked up based on what the specific application configures (probably in their application.properties) to be traced through it's entry/exit points.
@dsyer I am not sure I know how this supported (or currently works). Is there an example around?

shivangshah on 31 Mar 2016

Also wanted to add. It'd be good to trace and see them through UI as well (but not necessary). Essentially, these parameters (X-trace-Id and other custom headers) get logged (per MDC) and then eventually can be tracked in Splunk or ELK for debugging purposes. What we do have right now is legacy applications (not using sleuth) manually tracking some specific headers, and we would like to not do that manually if we are already using sleuth, and can probably piggyback on sleuth to provide this and log them.

shivangshah on 31 Mar 2016

I think I need a more detailed example of what the legacy apps are doing and how you want to use the data the provide. I don't really follow the use case yet (the existing support lets you change the names of the X-Span_id (etc.) and also log additional headers in the Span (but those don't get propagated since they aren't needed downstream). A legacy app knows nothing about sleuth so how would you use any features that we provide anyway?

dsyer on 7 Apr 2016

Sure. So let's say we have legacy Application A, sleuth enabled Application B and legacy Application C.
Application A calls Application B with headers "sessionId" and "clientIP". Application B calls Application C. Application C has filters which will filter out "X-Trace" , "sessionId", "clientIP" (because it knows it is being called by B, which eventually is called by A). For me to trace the "sessionId" and "clientIp" headers from B --> C, I could potentially manually write filters and interceptors, or somehow piggy back on Sleuth used in B so that I don't have to manually do this. Essentially, the call that comes in from A --> B --> C should be traced in logs for all 3 applications along with custom headers. Does that give a good explanation? In short, legacy applications were using their own version of "traceId", and because the new application can use sleuth, we want to piggy back on it so that we don't have to manually trace these custom headers we used in legacy systems for tracing. Let me know if not clear and I can probably draw something out.

shivangshah on 7 Apr 2016

@dsyer :arrow_up: (forgot to tag you earlier). Let me know if it's not clear

shivangshah on 14 Apr 2016

It's clear, thanks.

dsyer on 15 Apr 2016

+1 for this feature.

I'm trying to pass additional headers based on certain conditions.

Use case: Web Application W1 --> calls service S1 --> calls service s2.

Using logback filters, I was able to get web application W1 to log messages in DEBUG mode for a particular logged in user (for all other users, the log mode will be in ERROR). To assist in debugging issues, I need the log messages in debug mode from services S1 and S2. In addition to correlation ids, I would like to send additional headers (e.g.: x-log-mode: debug) in calls to services. If the header can be passed, I will include additional filter to change the log mode to debug when the header is present. This will allow me to get log messages in DEBUG mode for the entire path of the request.

I will be happy to make code changes and submit a pull request. Any pointers on how this can be implemented?

devasat on 8 Jul 2016

We will be working on this feature as the next big feature in Sleuth. It's often referred to as a Baggage. We are aware that it's an important one :)

marcingrzejszczak on 8 Jul 2016

👍4

@marcingrzejszczak - can we log the baggage- headers we pass in ? How can this be seen on the logs for correlation ? Does Slf4jSpanLogger honor this ?

alwaysastudent on 10 Jul 2017

I'm pretty sure you've already asked this question somewhere and I responded to you... You have to extend the Slf4jSpanLogger to print the baggage information. It doesn't come out of the box.

marcingrzejszczak on 12 Jul 2017

So all the headers that starts with "baggage" will be passed and that can be logged. Is there any way for passing arbitrary headers such as "x-api-device" in the span?