Thanos: Support Jaeger tracer

[d-ulyanov]

What happened
At the moment Thanos aleady is instrumented with OpenTracing and GCT used as hardcoded implementation.

What you expected to happen
It would be great to support Jaeger tracer in all Thanos components because its quite popular too. So, I propose to make tracer pluggable as it works for stores and add also implement Jaeger support.

We're using Jaeger a lot, so I can take care of this issue.
What do you think?

[bwplotka]

Yea, I think it makes sense, but again, we want to limit ourselve in terms of maintainance.

We don't want to support every tiny tracing provider with different API etc.

We have 3 options generally:
A) Create our own API, so backend has to adapt not us.
B) Add most popular ones
C) Use some common REST API

Since C is non existent, and we now we aim for two... B might be what we want for, so it's green light from my side :heavy_check_mark:

cc @domgreen as he is interesting in tracing side

[povilasv]

I think we should use https://opentracing.io/ they have a library for go

[bwplotka]

Yea, we use it already: https://github.com/improbable-eng/thanos/blob/1f043ec4a97709939aa3addbeca5b6510b7c6da6/go.mod#L35

But still it requires different providers having different implementations.

[rafaeljesus]

I would recommend looking into https://opencensus.io besides tracing it has API for metrics as well, in addition, you can export your data to multiple backends simultaneously i.e Prometheus, Datadog, HoneyComb, Jaeger, NewRelic and so on, for both metrics and traces

[rafaeljesus]

just as a info, opentracing will get merged to https://opencensus.io pretty soon

[d-ulyanov]

Making Thanos for Prometheus and using anything else for metrics collection is very strange idea :)
So I'm not agree with it :)
Also, Thanos already partially instrumented by opentracing and its also allows us to use different backends. I propose to add Jaeger backend support. This feature also covers Zipkin backend cause Jaeger client supports it.

[bwplotka]

Agree with @d-ulyanov

[rafaeljesus]

Making Thanos for Prometheus and using anything else for metrics collection is very strange idea :)

True thing, I was more referring to other Tracing providers, but it might be too much for thanos since we already have almost all in place basically

[brancz]

OpenCensus and the resulting merger allow to just use the tracing capabilities. Kubernetes is likely to do the exact same thing, where it will continue to use Prometheus for metrics and the merger project for tracing. As for Thanos, we have an immediate need for jaeger tracing, so if people are not opposed, I'd like to implement jaeger support in Thanos and once the merger project exists evaluate whether we should do a switch. Does that sound reasonable?

// edit

Just to be clear, by “we” I meant us at Red Hat.

[simonpasquier]

cc @jpkrohling @pavolloffay @objectiser for awareness as they're working on Jaeger...

[rafaeljesus]

we have an immediate need for jaeger tracing

sounds good to me, I'm running Jaeger, and it would be pretty handy to operate Thanos with tracing in place.

[povilasv]

:+1: @d-ulyanov AFAIK you are working on this, right?

[brancz]

@d-ulyanov let us know if there is anything we can help with, I'd very much like to see jaeger being integrated :)

[d-ulyanov]

Hi, @brancz!
We've started to work on this issue recently. In Russia we have almost a week of holidays in first half of May, sorry :) But I expect that we'll finish it in the middle of May.
Our Thanos fork already integrated with Jaeger, just need some time to clean up code.

[IKSIN]

https://github.com/improbable-eng/thanos/pull/1147
Pull request for this issue.

[GiedriusS]

This has been merged so we can close this. Will be available in the next version. Thanks everyone! ❤️