Thanos: too many compact_{thanos_compactor_pod_name}_runtime_* metrics
Hello
We have one thanos-query and many different clusters behind it. In That clusters we have thanos-stores and thanos-compactors in a pods. Sometimes pods are being recreated with new names. Also we get metrics from these pods. After few month thanos-query has too many metrics with names as compact_{thanos_compactor_pod_name}_runtime_.* and if I try to search any metrics then query works very slowly.
How to delete it?
When I trying to get one of these metric in 1y, query return "No datapoints found."
turbotankist
All 3 comments
Hmm. Weird, let us look this place through.
bwplotka
on 5 Mar 2019
Hi, what version of thanos do you use?
IIRC those unique metrics were removed in the newer version of thanos.
About removing them.. this is unfortunately not possible right now.
For more info about deleting metrics see for example this issue https://github.com/improbable-eng/thanos/issues/415
FUSAKLA
on 23 Mar 2019
removing old bucket with metrics resolve it
turbotankist
on 16 Apr 2019
Related issues
bwplotka
路
4Comments
sevagh
路
3Comments
jdfalk
路
3Comments
wiardvanrij
路
3Comments
barryib
路
4Comments
Most helpful comment
Hi, what version of thanos do you use?
IIRC those unique metrics were removed in the newer version of thanos.
About removing them.. this is unfortunately not possible right now.
For more info about deleting metrics see for example this issue https://github.com/improbable-eng/thanos/issues/415