Thanos: query-frontend: populate org_id in slow query logs
The OrgId passed to Cortex is hard-coded to "fake":
https://github.com/thanos-io/thanos/blob/5aac47760359503358191caeb73780f87aa94e38/cmd/thanos/query-frontend.go#L203
When logs are produced to the slow query log, it results in logs which contain org_id=fake, e.g:
level=info ts=2020-09-16T13:33:14.670646Z caller=frontend.go:206 org_id=fake msg="slow query detected" method=GET host=localhost:10902 path=/api/v1/query_range time_taken=890.912705ms ...
When using Cortex Query Frontend (which populates org_id based on the contents of the X-Scope-OrgID header), there was extra value in these logs. The org_id field is useful for identifying sources of problematic queries which take a long time to execute.
Does anyone have any thoughts on populating that org_id log field with the contents of a request header?
neilfordyce
All 5 comments
We have a few different clients which interact with Thanos, which provide various headers which are useful for finding the source of queries (e.g. X-Grafana-User which contains the user name of a logged in Grafana user). Would it be useful if Thanos query-frontend took a list of potential header keys to try to extract an OrgId, then fallback through them until it finds one which isn鈥檛 empty, something like query --query-frontend.org-id-headers=X-Grafana-User,X-Scope-OrgId,X-Another-Service-Header. If none of the headers are set then we could fallback to org_id=fake or anonymous.
It seems like it would be straightforward to add (and I'd be happy to contribute), but a lot depends how valuable other people find the slow query log.
neilfordyce
on 16 Sep 2020
I hardcoded that value to fake because there is no org concept in Thanos. But I think supporting this is definitely a valid feature request. Any thoughts @kakkoyun @bwplotka?
yeya24
on 16 Sep 2020
Sounds good to me !
bwplotka
on 16 Sep 2020
Awesome. I'll have a look at getting a PR together for that.
neilfordyce
on 17 Sep 2020
I think this will work as short term solution, but we should think how to define multitenancy in Thanos long term. (:
bwplotka
on 24 Sep 2020
Related issues
StephenWeber
路
3Comments
bwplotka
路
4Comments
caarlos0
路
3Comments
gburek-fastly
路
4Comments
sevagh
路
3Comments
Most helpful comment
We have a few different clients which interact with Thanos, which provide various headers which are useful for finding the source of queries (e.g. X-Grafana-User which contains the user name of a logged in Grafana user). Would it be useful if Thanos query-frontend took a list of potential header keys to try to extract an OrgId, then fallback through them until it finds one which isn鈥檛 empty, something like query
--query-frontend.org-id-headers=X-Grafana-User,X-Scope-OrgId,X-Another-Service-Header. If none of the headers are set then we could fallback toorg_id=fakeor anonymous.It seems like it would be straightforward to add (and I'd be happy to contribute), but a lot depends how valuable other people find the slow query log.