Thanos: sidecar: Allow to upload old (already compacted blocks).

[bwplotka]

Basically now we upload only blocks with compaction level == 1.

Main blocker for this is explained here: https://github.com/improbable-eng/thanos/issues/206 but TL;DR is that we cannot easily say if you have local compactor enabled or not. We can try with Prometheus 2.2.1, since I added flag endpoint to prometheus, so I guess we can try to mitigate this somehow.

It is not safe to upload stuff when local compactor is turned on, because:

• Race conditions between shipper (sidecar) and Prometheus compaction.
• Overlapping blocks that can happen and lack of vertical compaction abilities for TSDB: https://github.com/prometheus/tsdb/issues/90

First one is potentially easy to solve with snapshotting API.
Second is difficult to solve. Example:
Sidecar see A, B blocks. It uploads A and B. Block C appear. It cannot upload block C for a while (network issue, misconfiguration, etc). Prometheus local compaction compacts A,B,C into ABC. and removes A,B,C We unblock sidecar and it uploads ABC. In storage we have A,B and ABC. This is something that TSDB compactor cannot handle.

[bwplotka]

Maybe some quick command tool will be useful here? (at least will solve some use cases)

[bwplotka]

Workaround for now would be:

• Make sure object store bucket is empty!
• Turn off Prometheus or just disable compactions (tsdb.max-block-size=2h)
• Temporarly turn off global compactor.
• [IMPORTANT] Modify local meta.json and add at the end:

"thanos": {
 "labels": {
   "labelName": "labelValue"
   < all your external labels >
  },
  "downsample": {
  "resolution": 0
 }
}

• Upload manually all blocks to GCS in this format:
  
• Delete all uploaded blocks
• Rerun global compactor
• Rerun Prometheus

[bwplotka]

And easier step - turn off compaction permanently and change in shipper.go to upload all blocks, not only with compaction level 1

[bwplotka]

New proposal arised:

Goals:

• Migrating to Thanos while keeping old metrics generated by non-Thanos setup.
• Thanos gradual setup: Having just Thanos for HA and global view and enabling Thanos long term feature later on.

We propose to add sync command that will grab all the blocks from local filesystem, add thanos meta (the most important: external labels!) and upload all.

Main problem here is the local vs global compaction and lack of vertical compaction. The compaction level could be different (most likely they are). Some solution would be to decompact bigger overlapping blocks to smallest blocks but it would take lots of time and there might be not enough info which parts of index and chunks belongs to each source blocks (or both).

This means that we can safely sync only by uploading from oldest to newest for external labels that has no overlapping time range. Sync command might check against that. We might be able to remove overlaps manually to resolve sync.

[bwplotka]

This was enabled half year ago, but ticket got stale, sorry for that.

It works, but only for new setup (so if Thanos never was added to that Prometheus before):

Just add (hidden) --shipper.upload-compacted on v0.4.0 or older sidecar.

TODO: docs!