Charts: [stable/prometheus-operator] serviceMonitorNamespaceSelector not working

[irizzant]

Describe the bug
serviceMonitorNamespaceSelector does not discover service monitors in namespaces different than the one the chart is installed in.

Version of Helm and Kubernetes:
Helm:
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}

Kubernetes: v1.16.3

Which chart:
stable/prometheus-operator

What happened:
Installed the chart, added the rook-ceph service monitor and additional rules reported here:
https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/monitoring/service-monitor.yaml
https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules.yaml

No metrics are scraped from the service monitor, no additional rules added

What you expected to happen:
Prometheus should have added the service monitor and started scraping the metrics

How to reproduce it (as minimally and precisely as possible):
Install the chart stable/prometheus-operator and the Rook Ceph cluster as reported here, then enable monitoring as reported here.

Anything else we need to know:
Please note that manually adding to the chart the service monitor spec reported here under prometheus.additionalServiceMonitors actually brings up the metrics in Prometheus, meaning that it cannot be a problem in the specs itself.

Also, reading the docs:

prometheus.prometheusSpec.serviceMonitorNamespaceSelector | Namespaces to be selected for ServiceMonitor discovery. See metav1.LabelSelector for usage | {}
-- | -- | --

it looks like the default value for serviceMonitorNamespaceSelector is {} which is not nil so it should be enough to monitor service monitor in any namespace.

[talzion12]

I'm experiencing a similar problem.
Did you try setting prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false?

[irizzant]

It looks like the prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues did the trick, now prometheus picks up the ServiceMonitors in other namespaces too.

I saw that this setting acts on serviceMonitorSelector in the Prometheus CRD so that it's not locked to any specific matchLabel.

Since there could be also additional rules and PodMonitor resources in other namespaces, I assume one has to set to false also:

• ruleSelectorNilUsesHelmValues
• podMonitorSelectorNilUsesHelmValues

correct?

Also, I think the doc should be updated to better describe this behaviour

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[iTaybb]

I also encountered this.
This issue should not be closed until fixed/better documented

[zzhao2010]

It looks like the prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues did the trick, now prometheus picks up the ServiceMonitors in other namespaces too.

I saw that this setting acts on serviceMonitorSelector in the Prometheus CRD so that it's not locked to any specific matchLabel.

Since there could be also additional rules and PodMonitor resources in other namespaces, I assume one has to set to false also:

• ruleSelectorNilUsesHelmValues
• podMonitorSelectorNilUsesHelmValues

correct?

Also, I think the doc should be updated to better describe this behaviour

So the fix was to set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false in the values.yaml file?

[zzhao2010]

@irizzant @talzion12 ,
Could you please share what was the fix? I tried to set serviceMonitorSelectorNilUsesHelmValues to false, however it still didn't work.

[zzhao2010]

I also encountered this.
This issue should not be closed until fixed/better documented

@iTaybb Did you get a fix for this issue after all?

[vaishali-prophecy]

I am also facing the same issue. Setting serviceMonitorSelectorNilUsesHelmValues to false only enables selecting all service monitors from the namespace where prometheus is deployed. It is not selecting service monitors from different namespaces. Also matchLabels construct on serviceMonitorNamespaceSelector doesn't seem to work