Thanos: store/compactor: Improve index-cache logic.

[bwplotka]

AC:

• [X] [Precompute this in compactor
• [x] Move index-cache out of JSON to some flatbuffer/protobuf/binary
• [X] Store should just download them.

---

@xjewer question:
https://improbable-eng.slack.com/archives/CADR0UWS2/p1552931827180200

yep, but store has to have cache component as well, as sidecar sends block with no manipulations with data.
I hope I’ll have time this week to add this, so if you have any other ideas about this, you are welcome.
I personally want to split this work on 2 PRs:

1. Store cache in object storage
2. Implement binary format for cache and optimize store component to use it instead of loading/parsing json.

@bwplotka wdyt?

Fully agree with this :+1:

[xjewer]

What to do with 2 hours block uploaded by sidecar? Store component should get them right after they appear at the object storage.

as a workaround: if there's no index-cache, precompute in on the fly and use, ideally it should work well, but if compactor halts, it could be a problem for store not only work with non-compacted data, but also with calculation cache on each restart.

Thus the best way to solve it is to do cache on both sides: compactor to store cache to objstore and store to do index cache locally on store side.

[bwplotka]

Good point. Alternative is to do that on sidecar, but I think having store gw doing that makes sense more - it can survive lack of compactor AND sidecar being misconfigured.

Also it fixes big problem of upgrade model (backward and forward compatibility) -> we should support system being in partial version. E.g old compactor and new store gw and vice versa.

[bwplotka]

Can we assume you are on it @xjewer ?

[bwplotka]

First part is done, now @GiedriusS ' https://github.com/improbable-eng/thanos/pull/1013

[bwplotka]

To sum up, last item on the list is still index-cache JSON being .. JSON (: @brancz also mentioned as alternative https://github.com/json-iterator/go

[bwplotka]

Just to make sure our goals here is not only size but actually even more:

• data actually allocated in the memory.
• latency of loading index-cache/index-header

[bwplotka]

The current plan is to move index-cache to binary format. The proposal with rationales is in progress.