Thanos: How to reference sidecar in another namespace (same cluster)

[popsonebz]

thanos v0.4.0 prometheus v2.8.1

what happened
i installed created two kubernetes namespaces client-a-prometheus and client-b-prometheus.
i have a complete setup of all component of thanos in client-a-prometheus but only installed prometheus and thanos sidecar in client-b-prometheus.
However, when i configured the thanos querier to pick up the sidecar in client-b-prometheus, it doesn't work because its on a different namespace.

What i expected to happen
I expect thanos querier to pick up sidecar from other namespaces

How to reproduce it in querier config):

      containers:
         - name: thanos
        image: improbable/thanos:v0.4.0
        args:
        - query
        - --log.level=debug
        - --query.replica-label=replica
        - --cluster.disable
        # Discover local store APIs using DNS SRV.
        #- --store=dnssrv+thanos-store-gateway.monitoring.svc
         - --store=dnssrv+thanos-store-gateway.client-a-prometheus.svc

        # Get remote store APIs by IP:Port.
        - --store=sidecar.client-b-prometheus.svc
        - --store=dnssrv+sidecar-1.client-b-prometheus.svc
        - --store=sidecar-1.client-b-prometheus:10901
        - --store=sidecar-1.client-b-prometheus.svc.local:10901
        - --cluster.peers=thanos-peers.default.svc:10900
        ports:
        - name: http
          containerPort: 10902
        - name: grpc
          containerPort: 10901
        - name: cluster
          containerPort: 10900

Full logs to relevant components

Anything else we need to know

[bwplotka]

Hi,

Why not adding similar thanos-store-apis headless Kubernetes service in client-b-prometheus and refer thanos-store-apis.client-b-prometheus.svc ?

I intentionally use thanos-store-apis instead of thanos-store-gateway as I think there is some accdiental naming mistake -> store gateway means rather the component itself that implements browsing of older metrics in object storage. (: We should fix our tutorial manifests...

[popsonebz]

Hi,

Why not adding similar thanos-store-apis headless Kubernetes service in client-b-prometheus and refer thanos-store-apis.client-b-prometheus.svc ?

I intentionally use thanos-store-apis instead of thanos-store-gateway as I think there is some accdiental naming mistake -> store gateway means rather the component itself that implements browsing of older metrics in object storage. (: We should fix our tutorial manifests...

I will give this a try.

[popsonebz]

Hi,

Why not adding similar thanos-store-apis headless Kubernetes service in client-b-prometheus and refer thanos-store-apis.client-b-prometheus.svc ?

I intentionally use thanos-store-apis instead of thanos-store-gateway as I think there is some accdiental naming mistake -> store gateway means rather the component itself that implements browsing of older metrics in object storage. (: We should fix our tutorial manifests...

I gave it a try. I can see the store-api of that namespace but the sidecars of that namespace is not listed.

[vaibhavkhurana2018]

Try using,

--store=<svc_name>.client-b-prometheus.svc.cluster.local.:10901

Replace the svc with the service pointing to the sidecar of prometheus.

[FUSAKLA]

Hi, did you manage to resolve your issue? Can we close this?
Also questions like this are better to ne asked on the slack channel maybe

[popsonebz]

Sorry the I did not give feedback on this.
The issue was solved by using --store=.client-b-prometheus.svc.cluster.local.:10901