Thanos: Store GW: Stream chunk fetching; Reduce memory used per query.

Created on 15 Oct 2020 · 4Comments · Source: thanos-io/thanos

Following @pracucci idea from last Thanos Community Sync in order to reduce and predict better memory usage of Thanos Store Gateway per query.

The idea is to move Block index lookup to a separate, first stage and do it concurrently for all blocks that potentially can match the query. We pack all in some kind of merge sort series set.

Then we can have a defined number of workers that get series IDs and fetches chunks for them send to stream (in the right order).

This way we can highly control the overall resources per requests used. This is even more important because we overfetch chunk bytes from object storage to avoid API rate limits. Holding those bytes as short as possible and only for few series / blocks, make a lot of sense. (:

Help wanted, whoever have time first. It's big refactor.

store hard feature request / improvement help wanted

Source