Zipkin: Question: Can zipkin be used to replace logging?

[guettli]

Can zipkin be used to replace logging?

I am unsure if it is useful to have two infrastructures. One for logging and one for tracing.

Both have a lot in common.

But maybe I am missing something. I am new to this topic.

[basvanbeek]

Logging and Zipkin (a type of instrumentation) are actually very different but due to scope creep can have overlaps if not used properly. These overlaps usually are consumer induced and signal a lack of domain separation.

A good article to read on the difference between logging and instrumentation (metrics / tracing / etc) can be found here:

https://peter.bourgon.org/blog/2016/02/07/logging-v-instrumentation.html

Some confusion might rise due to OpenTracing which while often called a Tracing API is actually more and also allows for abstracting instrumentation like logging and metrics as there are some common denominators making it a candidate to share correlation data and context propagation. See:

http://opentracing.io/documentation/pages/instrumentation/common-use-cases.html

Zipkin strictly deals with Tracing and tracing consumption focus in your applications should be at proper granularity of timing metrics while limiting the amount of attached metadata (that could overlap with logging if done excessively). Metadata is typically added in Zipkin as binary annotations.

Good rule of thumb, only add the ones needed for querying Zipkin and investigation/assessment of traces and spans. leave out all others or suffer potentially severe performance issues (just like you'd have when using logging as a means to handle needed metrics, for those concerns use a proper metrics system like Prometheus).

Hope this helps. If not feel free to drop by on gitter and discuss some more.

[adriancole]

@guettli this is a good question. From an infrastructure point of view, Elasticsearch is sometimes a good choice for people, if they are already using ELK. We have a number of folks using the same cluster for log and trace data, with correlation between them. This is a common enough question that we should have a more prescribed answer, but I like the hints made by @basvanbeek

@openzipkin/elasticsearch any others want to take a stab?

[basvanbeek]

on that note... yes everything could end up in the same storage backend but from a developer consumption viewpoint you might want to keep the instrumentation in your applications separate due to differences in framework handling (unless green fielding), usage granularity and best effort (sampled tracing) vs. must have (logging of actionable items).

[guettli]

I am in the happy situation: Green fielding (we do not have tracing yet, only logging) and our servers are far from being saturated (we can handle the overhead created by tracing).

That's why I think, that in my situation it better to focus on tracing. I hope to get a directed acyclic graph of processes involved handling one request.

Yes, metrics are a different topic. But for me tracing and logging share a lot.

[hwinkel]

@guettli we are in a similar situation and I would agree that logging and tracing shares a lot. The was a good talk of @bensigelman at kubecon 2016 https://youtu.be/n8mUiLIXkto?t=876 explaining logging and tracing where spans in opentracing are "just" log messages. but if it comes to graphs extracted and aggregated from Log entries like ELK is doing I could imagine this could be better done with metric collection at the source and not by counting, slicing and dicing log entries. IMHO.
https://twitter.com/copyconstruct/status/809239415558017024

[adriancole]

Zipkin will not compete with Splunk and ELK. Some companies may have the
staff to take on such a burden, but that isn't the case here. Scope is
extremely important, and taking on the responsibility of a logging stack
(and replacement for SLF4J etc) is way beyond reasonable for this project.

Those wanting to log events can use annotations, but that's far different
than replacing logging.

[adriancole]

Closing because the question has been answered. Don't expect Zipkin to assume the responsibility of a full-bore logging system.

The log-like features that exist in zipkin will continue, though, and we will continue helping people integrate more tightly with logging systems, for example the recent change to support log urls.

https://github.com/openzipkin/zipkin/tree/master/zipkin-server#configuration-for-the-ui

Zipkin's scope is different than OpenTracing, which has recently started entering the general purpose logging space. For example, Zipkin is a system and OpenTracing is a library api. Some providers of OpenTracing api have logging features and will encourage their use. That doesn't imply Zipkin will. These are different projects, and the logging aspect of OpenTracing isn't universally popular.

[hwinkel]

@adriancole thanks for your statement, makes sense to me.

[nicmunroe]

@guettli I'm late to the discussion but wanted to share with you what we do at Nike, since we loosely coupled logging and distributed tracing concerns in a way that keeps them separate but provides significant value.

We make sure the distributed tracing's trace ID ends up in the SLF4J MDC for each request, configure our logging system's pattern to output the trace ID from the MDC for each log message, and then log normally. All the log messages from all the services end up in a log aggregation system like Splunk which we can then use to do searches for all log messages associated with a given trace ID, or perform useful reports, etc.

This combination has proven to be quite powerful and I would highly recommend it. You get to leverage your distributed tracing and logging capabilities to their fullest without compromising anything or locking yourself into awkward ways of logging info, and the combination gives you benefits you wouldn't otherwise be able to take advantage of - e.g. all your log messages get tagged with the trace ID, including log messages from third party libraries, which may have the critical bit of info you need to understand why a given request failed (for example).

[guettli]

@nicmunroe for me this topic is still hot. Thank you for sharing your way. I have a question: Do all your requests have a trace-id, or do you trace only one of (for example) thousand requests (to reduce performance penalty)?

Adding the ID in every logging output makes sense.

Since the span-ID is more detailed than the trace-ID, wouldn't it make sense to log the span-id instead of the trace-id? You see, I am still learning :-)

[nicmunroe]

@guettli at the moment we track every request. The performance penalty for individual services is usually negligible (YMMV depending on instrumentation and/or throughput), and since we don't have to funnel all spans to a collector but rather do all our distributed tracing analysis via searches and reports from the log aggregator there's no performance concern on the collector. Of course there's a monetary cost in our case for having the log aggregator pick up every span from the logs, but so far the tradeoff has been worth it for us. You'll have to weigh all the performance and monetary tradeoffs yourself to see what will work best in your situation, but before assuming tracing every request will be too much of a performance burden I'd recommend doing some performance testing. You might be pleasantly surprised.

We have our services return the trace ID in the response headers so that if we run into a request that took too long or had an error or otherwise needs investigation we can dig into it immediately. When investigating these issues the question is almost always "what log messages are associated with this request (trace ID)?", not "what log messages are associated with this small portion of the request (span ID)?". That's why we've only worried about spitting out trace ID for every log message, not span ID. That said, it might be a good idea to spit out both trace ID and span ID for every log message. I just wouldn't recommend outputting only span ID - it's likely to be much less useful in general.

[guettli]

@nicmunroe AFAIK it's easy to get the trace-ID from a span-ID. I am afraid to loose information if I would log only the trace-ID. Up to now I am still learning. At university I was told to avoid redundancy. At my current (naive) viewpoint it would be redundant to log trace-ID and span-ID. What do you think?

[nicmunroe]

@guettli yes it is easy to get the trace ID from span ID, but it's an extra hop that is not needed the vast majority of the time (at least in our experience - YMMV). And like I said, I think it's a good idea to log both.

Avoiding redundancy ("don't repeat yourself", or DRY) is good for mainline code, but it's not a mantra that should be blindly followed in all cases. For example, IMO in unit tests it's more important to have the tests be readable and understandable (and therefore maintainable) than it is to wipe out all redundancy. It's a balancing act. If DRY-ing up your unit tests makes them difficult to understand for other people (or yourself in 6 months) then you've probably gone too far. See this stackoverflow question/answer for a good overview of this idea.

In regards to logging trace ID and span ID - it might be sort of redundant to log both since you can technically get to trace ID from span ID but the drawback is nil (a few extra bytes per log message) and it doesn't hurt maintainability since it's just logs (the usual reason for DRY-ing mainline code), so might as well log both.

Of course that's all just my opinion. If you're unsure, try both ways and see which makes more sense for your use cases.

[guettli]

@nicmunroe I like the "trace id vs span id vs both" discussion. Do you have a idea where (forum, mailing-list, ..) this could be discussed with other experts?

[adriancole]

how about grabbing a slot here?
https://docs.google.com/document/d/16qNbn6IU43FuNMIie0rgWswIjWUCZUU0yNHxeR1vlRk/edit#

we can do a remote talk (bear in mind that Nic is in pacific time iirc)

[adriancole]

I've added world write to this (until people give me email addresses
to do a better job). I hope folks can help outline and clarify what's
been captured here. In many cases, it is copy/paste from this issue.

https://docs.google.com/document/d/1TdgBjMI0KhyXPmlrfVQ_d3wbtbydWQh5dcNzYe4oD2Y/edit