Thanos: compaction: Be clear about storage retention and downsampling.

[bwplotka]

Currently in Thanos compactor, users can set retention for each resolution:

--retention.resolution-raw=...
--retention.resolution-5m=...
--retention.resolution-1h=...

If any is left 0 then it is unlimited.

It is natural for user to think that downsampling is something that magically zooms out the metrics and reduces size, so it's tempting to make raw retention super small (because it seems to be expensive) and rest super long as it is seems to be "just" cheap. However it's not that easy (as we can see for number of tickets related to this), so we need to document it better and make it easier to use.

Facts:

• 5m downsampled data are created only after 40h and there is rather not worth to make it earlier then that.
• 1h downsampled data are created only after 10days, and similar, it is not worth to make it earlier. 1h is created from 5m resolution.
• Downsampled data DOES not need to be smaller really (!) This is because downsampling to stay accurate generates 5 series from single series. It reduces the samples, so for 15s scrape interval, 5m resolution has 20x less samples, 1h has 240x less samples than raw. Depending on scrape interval the gain is smaller or bigger. The problem is that if you have really high cardinally (like several millions), you end up with 5x several millions of series (without overheat from strings size though, as those 4 additional does not have labels). So you might reduce samples (which are highly compressible) in exchange for series (which are non compressible). It might happen that the size reduction is not that great as you expected, so the best is to check your downsampled block sizes on your own. However the whole point of downsampling is not really size. Is ability to cheaply draw accurate graph for long range queries like 1year.
• Due to accuracy aggregations, special usage of downsampled data is recommended. We don't simply remove samples - that could mask meaningful info especially for gauges. This means that best effect you have with <>_over_time aggregations when using downsampled data.
• It is always recommended to leave RAW data with similar retention as others, because it is super valuable to be able to quickly zoom in the past. With just downsampled data you lose that ability, as it is designed only for long range queries.

Additionally:
There is only rare use cases in using Thanos with object storage if you want just short retention (e.g 30d). Usually Prometheus 2.x is capable for really long retentions. I recently talked to person who has 5 year data retention with 2TB SSD and Prometheus deals with it just fine. Using Thanos with object storage is twice more complex than just Thanos sidecars (without uploading) and Thanos Queries for Global View and HA. So make your choice wisely. If you care about 30d of metrics, I really recommend to just use simpler form. Thanos object storage support is designed for longer retentions like years or even unlimited retention. The rare use cases is when you want to use object storage as backup options or you don't want to buy larger/medium SSD disks. (:

Let's dicuss how we can make whole thing easier to use.
Some early acceptance criteria would be:

• Have minmum retention for each flag. So e.g raw min retention would be 40h, for 5m would be 10d etc.
• Document those facts above.
• Write more tests to be sure that this generation across different block sizes and different retentions works.

I will mark all issues/questions related to this a duplicate and forward users to discuss all here in single place.

[Shini31]

Hello,

To make sure I understand, if I set this configuration:
--retention.resolution-raw=30d
--retention.resolution-5m=90d
--retention.resolution-1h=365d

All my samples that are more than 30 days old will be deleted. I'll only have the raw data and never the downsampled data. So it's not possible to mix different downsampling retention. Am i right ?

For my case, the recommended configuration is:
--retention.resolution-raw=365d
--retention.resolution-5m=365d
--retention.resolution-1h=365d

Right ?

[Shini31]

up ?

[matejzero]

If I understand downsampling correctly, the first case will:

• delete raw metrics after 30 days
• delete 5minute metrics after 90d
• delete 1h metrics after 1 year

So you will be able to get raw metrics for queries for the last 30 days, anything older and you'll need to pull from downsampled data.

[Shini31]

Hi @matejzero,

Thanks for your reply.
On the tests I did, I realized that with this configuration all data was deleted after 30 days.

[matejzero]

That should not happen. Only raw data should be deleted after 30d, 5min and 1h downsamples should still be available.

If you do the test again, check what thanos bucket inspect returns. It should show you that longer retention should still be available, because 5min downsamples are calculated after 40h and 1h downsamples are calculated after 10d.

[raffraffraff]

Just got bitten by this. Our configuration was:
--retention.resolution-raw=7d --retention.resolution-5m=14d --retention.resolution-1h=60d

I expected the compactor/downsampler to delete _raw data_ after 7d, but that I would still have downsampled data for longer. We have NO DATA beyond 7d.

[raffraffraff]

The bucket inspect tells a different story. It tells me I have blocks with these attributes:

FROM: 20-05-2019
UNTIL: 30-05-2019
RANGE: 240h0m0s
SAMPLES: 1,079,894,691
CHUNKS: 10,790,261
COMP-FAILED: false
RESOLUTION: 5m0s
SOURCE: compactor

So how come my queries beyond raw retention have no data? I feel like I'm missing something really obvious here.

[bwplotka]

What are you querying for exactly @raffraffraff ? What Thanos version? We fixed one bug on auto downsampling option for querier in v0.5.0

[raffraffraff]

I'm looking at a dashboard that should go back 60 days (we started shipping data to S3 months ago but have been downsampling and retaining raw=7d, 5m=14d, 1h=60d)

Thanos Compactor 0.5.0-rc0
Thanos Store 0.5.0-rc0
Thanos Sidecars 0.4.0
Thanos Query 0.4.0

My dashboard ends at 7d. We've been using v0.5.0-rc0 since it came out, so if it fixed the issue, surely our data wouldn't evaporate exactly 7d ago?

[bwplotka]

You just said that data is there, but you cannot query it, right? https://github.com/improbable-eng/thanos/issues/813#issuecomment-499579588

[raffraffraff]

Basically, yes. Queries give back nothing after 7d but the bucket inspect indicates that there should be data in the buckets. I'm gonna upgrade the query instances to 0.5.0 and report back.

[matejzero]

What is your time range at the end of the query (rate(metric[timerange]))?

[raffraffraff]

Not sure I'm following you @matejzero . My simplest query is mem_used_percent{host=~"$host"} and I see it all the way back to 7 days ago.

Updating to Thanos Query 0.5.0-rc0 didn't help.

[raffraffraff]

And just to confirm, we are using --query.auto-downsampling in our queriers

[raffraffraff]

Interesting: in the Thanos Query interface, the 'Auto downsampling' option returns no data. I go back 2 weeks, and the graph is empty (no data points). I switch 'Max 5m downsampling' or 'Max 1hr downsampling' and I see data. WIth raw, I do not expect to see any because it has been expired. But I do expect 'Auto downsampling' to work.

[raffraffraff]

EDIT: Corrected a few things after further debug logging, reformatted for clarity.

The purpose of --query.auto-downsampling is to allow Thanos Query to return lower resolution data when huge time ranges are queried, but it should specify a "preferred" resolution instead of a strict maximum, which breaks certain queries. Example: auto-downsampling correctly chooses 'raw' resolution for a 1h span, but if that time range starts 10 weeks ago and the raw data expires after 4 weeks, Thanos Query UI throws "No data points found" instead of falling back to 5m data. It works fine the other way around - Thanos Query's getFor() will happily start off with a lower resolution (eg: 1h) and switch to a higher resolution (5m) if 1h downsampled data runs. But it should be able to do this in either direction.

Issues which I'm going to file separately:

1. Log entry MaximumResolution= means that opposite of what you would expect
2. API param max_source_resolution means that opposite of what you would expect
3. getFor() should be able to fall back to _lower resolution data_ if no data is available the resolution given.

TL;DR: Right now, Thanos (all versions, including 0.5.0) downsampled data is ~unusable beyond 'raw' data retention, unless you are using the same retention on all three resolutions.

[wogri]

Is there a plan to fix this brokenness?

[adisrabenh]

Any updates on this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[romoy]

I was just bit by this. Could there be an update of some form to the retention flags to warn of delete in object store?
or confirm backup of object store before attempting next to the flag https://github.com/thanos-io/thanos/blob/master/docs/components/compact.md#downsampling-resolution-and-retention ?

[bwplotka]

Ok this issue went sideways (: Let's keep it focused.

The retention and downsampling was explained in details here https://thanos.io/components/compact.md/#downsampling-resolution-and-retention
Sorry for not closing this issue (:

However, I could see some side requests/questions. Can we start another, separate issue for that?

Especially @romoy Definitely we can discuss to add this feature - we have already code for this, just we need to plumb it (: Can you please add an feature request issue with details What's the use case, what is missing, why you need this? (:

[romoy]

I opened https://github.com/thanos-io/thanos/issues/2290, could I get a review issue? Is it missing any info or a prompt a question?