Thanos: Thanos store failed to fetch old data

[satyabhama]

Thanos store component is continuously throwing below error when we are trying to fetch old data from S3 via Thanos query and stroe component
level=warn ts=2020-02-20T08:13:42.014710592Z caller=bucket.go:326 msg="loading block failed" id=01E0X0RPKQ45SC35Y34RSSMDVV err="new bucket block: load index cache: download index file: copy object to file: write =/thanos/store/01E0X0RPKQ45SC35Y34RSSMDVV/index: no space left on device"

Also we are not able to see old data even though it is present in s3.

[kakkoyun]

Hey @satyabhama, As far as I can tell from your logs, you don't have enough space left on your disk where you run your thanos store, it couldn't download the index for the block. Could you please check that?

[satyabhama]

Hey @kakkoyun We increased the memory according to this before only, now memory utilization is less than 75% still we are getting this error.

[bwplotka]

It's not memory, it's disk space.

[kakkoyun]

But from your logs, it seems like a disk issue

download index file: copy object to file: write =/thanos/store/01E0X0RPKQ45SC35Y34RSSMDVV/index: no space left on device"

This happens before building the index in-memory, so try checking the disk.

[satyabhama]

Could you please help us to check disk space, where exactly we need to configure

[bwplotka]

We don't have access to your machine. It's not something you can configure in Thanos. It will use the amount of disk which is needed by amount of your data.

In the next release, we have a new experimental feature called index-header which will use much less disk space.

[satyabhama]

hey @bwplotka how much disk space is needed if 700 GB data is there in s3 bucket

[bwplotka]

It always depends on your cardinality, amount of blocks etc.

For https://github.com/thanos-io/thanos/releases/tag/v0.11.0-rc.1 and with:

1952 Store Gateway: Implemented binary index header. This significantly reduces resource consumption (memory, CPU, net bandwidth) for startup and data loading processes as well as baseline memory. This means that adding more blocks into object storage, without querying them will use almost no resources. This, however, still means that querying large amounts of data will result in high spikes of memory and CPU use as before, due to simply fetching large amounts of metrics data. Since we fixed baseline, we are now focusing on query performance optimizations in separate initiatives. To enable experimental index-header mode run store with hidden experimental.enable-index-header flag.

Disk usage should be marginal, PLUS it does not need to be persistent. E.g. For each block, you have in the system with let's say 0.5-5GB index file, for a reasonable number of symbols and its length, and cardinality (e.g. million of series), I would expect like 100MB index-header file and that's it.

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.