Prometheus-operator: ServiceMonitorNamespaceSelector and ServiceMonitorSelector with matchlabels are not working

[ravishankarsrrav]

Removing the pointer to " *metav1.LabelSelector" in "PrometheusSpec" might solve the issue.

https://github.com/coreos/prometheus-operator/blob/9468bf9f1f1219dfd996364f8aa496fce5dbd5fb/pkg/apis/monitoring/v1/types.go#L86

https://github.com/coreos/prometheus-operator/blob/9468bf9f1f1219dfd996364f8aa496fce5dbd5fb/pkg/apis/monitoring/v1/types.go#L83

[metalmatze]

Can you please be a bit more specific about what is not working. I'm missing quite some information and context here. Thanks

[ravishankarsrrav]

Can you please be a bit more specific about what is not working. I'm missing quite some information and context here. Thanks

Giving match selectors for selecting specific servicemonitors for Prometheus is not working because of the pointer in the prometheus spec I think. Please check it out.

[metalmatze]

Can you share your configuration that breaks?

[ravishankarsrrav]

serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
k8s-app: avalanche

• ServicemonitorSelector with label names are not working for me

[brancz]

We are using this and it passes on every run of our end to end tests: https://github.com/coreos/prometheus-operator/blob/5d3c10eb89a637071874bc946396cf77ca322a2b/contrib/kube-prometheus/manifests/prometheus-prometheus.yaml#L30-L31

Could you please post a full example of how to reproduce your issue? Thanks!

[ravishankarsrrav]

Hi, I have uploaded the yaml in the following link which has caused me error while specifying label names in serviceMonitorSelector. Please check out this yaml file in the following link.
https://github.com/ravishankarsrrav/prometheus-yamls/blob/58b9e4d163b8a8c88153caa3c6842b756e608487/prom-svc-monitor-error.yaml#L29

[brancz]

Have you tried the empty selector {}?

[ravishankarsrrav]

I tried empty selector it works well but that selects all the servicemonitors but I want to select specific servicemonitors by giving labels.

[brancz]

In that case you want:

  serviceMonitorSelector:
    matchLabels:
      label: value

[ravishankarsrrav]

Thanks, The above configuration is not working for me.

[brancz]

Could you please be more precise in what you are and what you are not seeing? And share the full Prometheus object (with the matchLabels selector), a ServiceMonitor that you want to be selected, and one that you do not want to be selected, as well as the generated configuration that you can get with:

kubectl -n <namespace> get secret prometheus-<prometheus-object-name> -ojson | jq -r '.data["prometheus.yaml"] | base64 -d

[feli5]

I also hit this problem. With below configure it works well.

serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}

But if I configure something like

  serviceMonitorNamespaceSelector:
    matchLabels:
      namespace: test

And then check Configuration in Prometheus GUI, there is no scrape_configs: at all, below is the entire configure in Prometheus.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: test/k8s
    prometheus_replica: prometheus-k8s-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - test
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager-main
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: web
      replacement: $1
      action: keep
rule_files:
- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml

[feli5]

Forget to mention, version of prometheus-operator is
image: quay.io/coreos/prometheus-operator:v0.29.0

[feli5]

BTW, serviceMonitorSelector works for me.

  serviceMonitorSelector:
    matchLabels:
      namespace: test

[brancz]

Could you share the Namespace object you're trying to select? Does it actually have the namespace: test label?

[feli5]

My object contains label 'namespace: test', otherwise

 serviceMonitorSelector:
    matchLabels:
      namespace: test

should not work either.

[brancz]

If I understand correctly you’re saying that if you configure a namespace selector then you’re not seeing your service monitor picked up. This would make perfect sense if your namespace is not labeled appropriately. Which namespace did you create the servicemonitor in? And does that namespace have the “namespace: test” label? Can you please share the namespace yaml?

[feli5]

That's what I'm using,
namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: redis
  labels:
    app: redis
    namespace: test
  namespace: test
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: redis
  namespaceSelector:
    matchNames:
    - test
  endpoints:
  - port: redis-exporter
    interval: 30s
    honorLabels: true

[brancz]

If you specify a namespace selector, then that namespace must have that label :) you need to add the “namespace: test” label to your namespace.

[feli5]

Cool, I will try it. And I have one more question, if I use serviceMonitorNamespaceSelector, I think prometheus will only pull metrics from exporters in matched Namespace, Is my understand right？

[brancz]

It will only select ServiceMonitor objects from that namespace, which in turn should be pointing at applications/exporters from those namespaces, correct.

[feli5]

Thanks, will try it.

[archyufa]

@feli5 Can you please provide feedback if @brancz recommendation to label namespace worked for you and if this issue can be closed as it's configuration problem. Do you think this is something needs better documentation ?

[DanielShor]

I was experiencing another issue with serviceMonitorSelector.

I'm using Prometheus to scrape applicational metrics (using custom service monitors which select services with specific labels).
For simplicity, let's say that I want Prometheus to select service monitors that match the label prometheus: somelabel, so I'm adding this to serviceMonitorSelector:

serviceMonitorSelector:
  matchLabels:
    prometheus: somelabel

Indeed, I can see the changes in Prometheus using this query: kubectl get prometheus -oyaml, and it works well (meaning, Prometheus select my service monitor).
The issue is that the change doesn't reflect in all the default service monitors (like alertmanager, apiserver, grafana, etc.), meaning they all don't have the corresponding label (prometheus: somelabel), so by definition they won't be selected. I expect that the labels added to serviceMonitorSelector.matchLabels will be added to all default monitors labels.

WDYT?

[Asgoret]

Same error. I tried:

  serviceMonitorNamespaceSelector:
    matchLabels:
      cluster-monitoring: 'true'

And it always return to:

  serviceMonitorNamespaceSelector:
    matchExpressions:
      - key: openshift.io/cluster-monitoring
        operator: Exists

[brancz]

@Asgoret it looks like you're trying to modify the immutable stack shipped on OpenShift. Your changes will always be reverted, this is intentional.

[Asgoret]

@brancz Yeah, i clarified it too) Thanks for help!

[marcosnils]

If you specify a namespace selector, then that namespace must have that label :) you need to add the “namespace: test” label to your namespace.

Which brings me to the next question. What if all the namespaces in my cluser don't have any labels? Is there a reason why it can't be a fieldSelector instead of a label selector?

I.E:

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: 2018-07-24T18:42:00Z
  name: deployments
  resourceVersion: "567"
  selfLink: /api/v1/namespaces/deployments
  uid: 40154a7e-8f71-11e8-9d93-02702fedfe3c
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

[brancz]

We would need to check the API, to see if selection can be done like that. Last time I checked (which admittedly had been a while) field selector was pretty much limited to the pods API.

[marcosnils]

We would need to check the API, to see if selection can be done like that. Last time I checked (which admittedly had been a while) field selector was pretty much limited to the pods API.

You're right. What I'm trying now is to add labels only to the namespaces that I want to exclude and use a matchExpression to scrape everything but the namespaces with that label.

[brancz]

You should be able to model something like that using the “NotExists” and “In” expression operators no?

[marcosnils]

Correct. Worked like a charm. Thx getting back so promptly

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.