Thanos: Handle (ignore) partial uploads on downsampling and apply retention cycles

[bwplotka]

AC:

• Meta files sync should ignore not-complete blocks for downsampling and retention cycle.
• Removal of malformed block if any is ONLY on compaction cycle.

We have nice concurrent logic here: https://github.com/improbable-eng/thanos/blob/10f84a7d7f06813fda97e17d609a9969fcda037f/pkg/compact/compact.go#L169

We did not apply it here: https://github.com/improbable-eng/thanos/blob/10f84a7d7f06813fda97e17d609a9969fcda037f/cmd/thanos/downsample.go#L145

And not on retention: https://github.com/improbable-eng/thanos/blob/1d582af96b2cd412ade46dcfa07e6d40ffb3c176/pkg/compact/retention.go#L16

[bwplotka]

It fixes compactor crashes like

level=error ts=2019-07-15T21:09:56.108249354Z caller=main.go:182 msg="running command failed" err="error executing compaction: first pass of downsampling failed: retrieve bucket block metas: get meta for block 01DFVPYR64AGDW009PF99N2YFY: The specified key does not exist."

This is because when doing sync and sidecar uploading block we might hit block with no meta.json.

[lx223]

Hey @bwplotka I can help with this.

[bwplotka]

oh yes please @lx223

Essentially we have ~3~ 4 different sync implementation. I think we should have nice one meta syncer everyone can reuse: compactor for compating/downsampling/retention but also store gateway.

[bwplotka]

Any progress @lx223 ?

[lx223]

sorry for the delay; have been busy with work. should have something up for review soon.

[lx223]

PR: https://github.com/thanos-io/thanos/pull/1394 PTAL

[bwplotka]

Still valid I think, ref: https://cloud-native.slack.com/archives/CK5RSSC10/p1572453579422500

[bwplotka]

It's getting quite important to get this in, another point of failure:

• https://github.com/thanos-io/thanos/issues/1655#issuecomment-559068163
• FIxes like https://github.com/thanos-io/thanos/pull/1807

[bwplotka]

https://github.com/thanos-io/thanos/pull/1394 was our first attempt and can be used as a base (: Will get into this in free time, unless there are other volunteers (: