Kube-prometheus: Prometheus Operator does not observe PrometheusRule CRDs outside the monitoring namespace

[teralype]

What happened?
Prometheus Operator, as deployed with Kube Prometheus, does not observe PrometheusRule objects outside the monitoring namespace. When I create a PrometheusRule object inside a namespace named ops, the Prometheus ConfigMap is not properly regenerated by the Operator. However, deploying this same PrometheusRule object inside the monitoring namespace properly triggers the Operator to regenerate the ConfigMap for Prometheus.

This is the jb fragment for Kube Prometheus:

        {
            "name": "kube-prometheus",
            "source": {
                "git": {
                    "remote": "https://github.com/coreos/kube-prometheus",
                    "subdir": "jsonnet/kube-prometheus"
                }
            },
            "version": "3d5fc3e38d8d8fdc6699f113908dd04f6e026c3d"
        },

Did you expect to see some different?
Yes, I expect that as well as ServiceMonitor objects are observed in all the additional namespaces configured, PrometheusRule should behave the same.

How to reproduce it (as minimally and precisely as possible):
I can provide the Jsonnet code that I'm using.

Environment
GKE + Kube Prometheus + ClickHouse

• Prometheus Operator version:
  quay.io/coreos/prometheus-operator:v0.33.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:42:50Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-gke.9", GitCommit:"6c1e92d07f5717440f751666d4aad6943015d3cb", GitTreeState:"clean", BuildDate:"2019-10-11T23:14:17Z", GoVersion:"go1.11.13b4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  GKE (deployed via Terraform)

• Manifests:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: prometheus-k8s-rules
  namespace: monitoring
spec:
  groups:
  - name: clickhouse.rules
    rules:
    - alert: ClickHouseDiskUsageTooHigh
      annotations:
        description: This is an indication that the disk usage ratio for {{ $labels.hostname
          }} in namespace {{ $labels.namespace }} is {{ $value }} > 0.8
        summary: ClickHouse disk usage too high.
        value: '{{ $value }}'
      expr: |
        (
          chi_clickhouse_metric_DiskDataBytes
        /
          (
            chi_clickhouse_metric_DiskDataBytes
          +
            chi_clickhouse_metric_DiskFreeBytes
          )
        ) > 0.8
      for: 10m
      labels:
        severity: critical

• Prometheus Operator Logs:

Nothing relevant, but I still lots of:

W1108 08:50:24.613129       1 reflector.go:302] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:452: watch of *v1.ConfigMap ended with: too old resource version: 11405801 (11407503)

• Prometheus Logs:

Nothing relevant as Prometheus is not touched at all, not reloaded, its configuration unchanged, etc.

Anything else we need to know?:

[brancz]

The servicemonitorNamespaceSelector and ruleNamespaceSelector need to select the namespace in question and the Prometheus serviceaccount needs to have a role binding bound to it in order to discover pods as specified in the servicemonitor.

[teralype]

serviceMonitors are working properly, and Pods are discovered. However, the PromethesRule CRDs are not being observed or are ignored. I'll see whether the ruleNamespaceSelector is correct.

[rverma-jm]

@teralype Did you find the root cause for this, facing same issue.
Few things which I checked

1. PrometheusRule has same label as promethues
2. RuleNameSelector has matchLables | matchNames for corresponding namespace (labels/name).

Still my prometheus cluster couldn't discover prometheusRules.

[brancz]

matchNames does not exist for PrometheusRule selector.

[tfluehmann]

I had the same problem.
After some time I figured out setting the ruleNamespaceSelector to an empty map {} fixes the problem and all prometheusrules in all namespaces are selected.

[paulfantom]

Closing as this appears to be answered. Please reopen if you think this is still relevant.