Thanos: Feature Request: Add flag to ignore verification of out-of-order label sets

[josephschadlick]

The compactor module does not currently support series beginning with uppercase labels. It catches these series during the verification process with an out-of-order label set error.

For example:

caller=main.go:181 msg="running command failed" err="error executing compaction: compaction failed: compaction: gather index issues for block /data/thanos/compact/0@{monitor=\"gn-development-prometheus\",replica=\"a\"}/01CSK3ETX0VD5KW2FZM3CNBZC3: out-of-order label set {__name__=\"test_gauge\",Address=\"1.2.3.4\",Application=\"TestApp\",Email=\"[email protected]\",Environment=\"Dev\",Name=\"Test\",Stack=\"Test\",Vertical=\"Test\",instance=\"test-instance\",job=\"test-batch-job\",specific_key=\"specific_value\"} for series 2482650"

While I understand that labels should be all lowercase for index efficiency, we are invested in using uppercase characters in our labels. Our monitoring infrastructure (alerts, dashboards, etc) heavily relies around propagating service discovery tags, which are UpperCamelCase, to Prometheus as is. I believe this use case is not uncommon.

As discussed, it would be useful to add a flag (with sufficient warning) to ignore verification of series against this issue.

[bwplotka]

Valid request, PR very welcome (:

[GiedriusS]

I think that instead of having this as an option, we should make compactor accept kinds of labels and their combinations that can be produced by Prometheus. I believe that https://github.com/improbable-eng/thanos/pull/831 and https://github.com/improbable-eng/thanos/pull/848 should solve this without having a separate flag as this sanity check is still definitely useful.

[josephschadlick]

@GiedriusS - I agree that the compactor should be compatible with any valid prometheus format. Those PRs solve my use case more elegantly than my proposed solution. I am happy to close this if those are to be accepted.

[rafaeljesus]

oh great that #831 solves it, I will take a look at it. I am having the same issue

[GiedriusS]

This flag and all of the cases in Thanos and Prometheus were fixed, closing (: please let us know if you continue to have problems