Thanos: OpenTSDB support

[szalai1]

Goal: Query openTSDB using the query component.

I have POC implementation and managed to query openTSDB. What is the preferred way to add this new functionality to Thanos?
Options:

• New subcommand
• Subcommand under sidecar
• Flag on the sidecar command (default prometheus)
• Plugin

Long term implication: how does thanos want to treat other tsdbs than prometheus?

(In our case we want to read from openTSDB, but not write.)

[bwplotka]

Thanks for raising this.

Thanos was designed to allow different metric backends, by introducing generic StoreAPI gRPC API that anyone can implement.

Effectively Thanos should work fine with any implementation that hides behind StoreAPI, so the only question here is: should this code be hosted in Thanos project or externally.

TL;DR: I feel that we should aim for hosting those as plugins in separate repositories.

Details:
There are numerous advantages of this for Thanos project AND store API implementations. Let's enumerate some:

• Users can indepdendly develops plugins. We don't limit what language it is and what dependencies plugin adds to build & test it properly (avoids dependency hell).
• Thanos can focus on main goal: Enable Thanos functionalities and support main TSDB: Prometheus TSDB. This allows for better quality product and maintanance expierience.

There are though some clear difficulties in terms of having external plugin in separate repo. Let's enumerate those and see how we can help to workaround them:

• Compatibility between versions: our only contract is gRPC API that gives us flexibility in versionsing (backward and forward compatibility), so it should not be a big problem
• Discoverability, so how to tell users that there exists open TSDB implementation? I think we could host in Thanos repo/webiste an explicit list of those maintained plugins.
• Code reusability: If the implementation is in Golang maybe you would like to reuse some packages and logic. I think it's fine for plugin to depend on various Thanos packages. We can even move code around to make it even easier.
• CI/deployment path reusability: Thanos uses CI for open source, so it should not be hard to mimick CI/deployment path. There are numerous of open docker registry to host docker images as well (: As Thanos community and maintainers we can help to hook all up.

What do you think? @brancz @domgreen @improbable-ludwik @FUSAKLA @adrien-f any other concern?

Overall I am happy you touched this issue as this might be not well known use case for Thanos, and we should document it more (BYOS - bring your own storeAPI (: )

[brancz]

I have had the same "bring your own StoreAPI" solutions for these types of features in mind.

[FUSAKLA]

Plus one to keeping those implementations separately. Off-course that could raise question if the TSDB backend shouldn't be also separately to leave Thanos just as a query layer (compactor, store and sidecar are actually Prometheus/TSDB specific).

The other thing is downsampling maybe? Would this work for the openTSDB for example? (not familiar with the implementation if the the query logic is in the store or query part)

But yeah I can imagine this would be great the possibility to have one source aggregating data from more timeseries databases :+1:

[jojohappy]

I remember that the issue brought a proposal for using Go cloud to implement the bucket store, just like this discussion, but it is no writing flow here. Maybe we can discuss both of them (read and write).

Agree with @FUSAKLA to keeping those implementations separately.

[bwplotka]

@jojohappy brought very nice comparision with other places of various client implementations like pkg/objstore we have (: We have long discussions and long term we want to have them in separate repositories as well. The problem right now is that the interface is just Golang interface and there is no great plugin solution for Golang currently. The other way would be to define Objstorage gRPC/HTTP API but this complexity addition has it's issues.

With StoreAPI contract is clear and cross binary, so it fits into multiple repo ecosystem.

@szalai1 is that ok with you? Separate open source repo for plugin with mention in our docs about details? (: Any concerns?

[szalai1]

@FUSAKLA I have not implemented the downsampling part yet, but I think it can be in the storage layer.

@bwplotka I am totally fine with a separated project and makes more sense for me too.
Just to be clear this means a whole and independent sidecar implementation and it can rely only on the gRPC storage contract. And it also means that it can be done without changing this repo (except if you want the current Prometheus specific sidecar in a separated repo). Is it correct?

[bwplotka]

And it also means that it can be done without changing this repo (except if you want the current Prometheus specific sidecar in a separated repo). Is it correct?

Correct unless there is something that blocks you or you find the Store API too broad/too limited/unclear. Let us know (:

[szalai1]

Thank you for your help. Closing the issue.

[szalai1]

@bwplotka As we agreed, this has been implemented in a separate repo here: https://github.com/G-Research/geras

[bwplotka]

Awesome! As noted in slack discussion:

@szalai1 I would be happy to add integrations.md page to Thanos docs with description how your project intrgrated Thanos with OpenTSDB in details :+1: Also I would be very clear in README of geras that intimplements StoreAPI that then allows PromQL via Thanos querier, which is the case?

[szalai1]

@bwplotka great! geras exposes the StoreAPI so if you want to use it you will need Thanos Querier and specify geras as store. You can even join Prometheus and OpenTSDB data :) .

[bwplotka]

Nice, I would really write some blog post if I were you guys - sounds interesting (:

[bwplotka]

Feel free to propose some integration.md page - if not we will create something at some point

[szalai1]

@bwplotka sure, I'm going to add one. Is that ok if I add the integration.md to the docs directory and a link to it from the main README.md?

[bwplotka]

Yes.

On Mon, Jun 24, 2019, 22:20 Peter Szalai [e-mail] wrote:

@bwplotka https://github.com/bwplotka sure, I'm going to add one. Is
that ok if I add the integration.md to the docs directory and a link to
it from the main README.md?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/improbable-eng/thanos/issues/768?email_source=notifications&email_token=ABVA3OZEMLTVJVRFVQ4BF4TP4E3CPA5CNFSM4GSYVOK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYOIN4A#issuecomment-505186032,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVA3O32V2TLIGJHVQBXVRTP4E3CPANCNFSM4GSYVOKQ
.