Prometheus-operator: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope

[Shini31]

What happened?
There is an unexpected flood of error message for Thanos Ruler CRDS

E0301 17:04:11.624026       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:12.626151       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:13.653165       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:14.655977       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:15.658397       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope

Did you expect to see something different?
No error log message

How to reproduce it (as minimally and precisely as possible):
Deploy Helm prometheus-operator in 8.10.0 version

Environment
GKE

• Prometheus Operator version:
  0.36

• Kubernetes version information:

Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.17", GitCommit:"bdceba0734835c6cb1acbd1c447caf17d8613b44", GitTreeState:"clean", BuildDate:"2020-01-17T23:10:13Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  GKE

• Manifests:
  Helm prometheus-operator

• Prometheus Operator Logs:

E0301 17:04:11.624026       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:12.626151       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:13.653165       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:14.655977       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
E0301 17:04:15.658397       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:monitoring:sandbox-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope

Anything else we need to know?:

[Arkaniad]

For what it's worth I am also able to reproduce this issue on DOKS 1.16.6-do.0 with Prometheus Operator 8.10.0 (chart version 0.36.0), same exact error as @Shini31

stern output:

prometheus-operator-operator-649876c46d-shpss prometheus-operator E0301 18:42:22.614961       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:prometheus-operator:prometheus-operator-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
prometheus-operator-operator-649876c46d-shpss prometheus-operator E0301 18:42:23.620745       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:prometheus-operator:prometheus-operator-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope
prometheus-operator-operator-649876c46d-shpss prometheus-operator E0301 18:42:24.623309       1 reflector.go:123] github.com/coreos/prometheus-operator/pkg/thanos/operator.go:315: Failed to list *v1.ThanosRuler: thanosrulers.monitoring.coreos.com is forbidden: User "system:serviceaccount:prometheus-operator:prometheus-operator-operator" cannot list resource "thanosrulers" in API group "monitoring.coreos.com" at the cluster scope

One message every second

[brancz]

This sounds like a problem with the helm chart(s).

The helm charts are neither used nor maintained by the prometheus-operator maintainers. Therefore we don't know the setup, so we're unlikely to be able to help you. I recommend creating an issue on the upstream helm charts repo where these helm charts are maintained.

[Shini31]

I hesitated when I created the issue.

FYI: https://github.com/helm/charts/issues/21171