Kube-prometheus: "Failed to list" errors logged using default ClusterRole "prometheus-k8s"

[miff2000]

Prometheus is spamming our logs with messages about it not having permissions to list the resources services, endpoints and pods in our cluster. The error messages look like this:

level=error ts=2020-04-06T11:17:49.793Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:264: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"fluentd\""
level=error ts=2020-04-06T11:17:49.756Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"fluentd\""
level=error ts=2020-04-06T11:17:49.753Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:265: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"nginx-ingress\""

Amending the included ClusterRole to include the resources and the list verb stops the errors. However, I notice that in all of the releases in this repo, none of them have these permissions included on the ClusterRole:

• https://github.com/coreos/kube-prometheus/blob/master/manifests/prometheus-clusterRole.yaml

Looking at the Prometheus docs, their ClusterRole does include these permissions in their docs.

What am I missing? I notice that in this issue the same errors are reported, but the answer is that they should be using the release-0.1 branch.

We're running Kubernetes 1.16.5 with the release-0.3 branch of kube-prometheus.

Thanks in advance!

[miff2000]

The reason for this, it seems, is that those resources and verbs are granted on a per-namespace basis, not as part of the ClusterRole.

You can therefore update the array of namespaces which these permissions should be extended to, or amend the ClusterRole to add in those permissions. I'm going to do the latter, and I'll add a comment to this to demonstrate how I did it

[miff2000]

I updated the ClusterRole to include the extra roles by adding this to my base jsonnet file

local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local clusterRole = k.rbac.v1.clusterRole;
local policyRule = clusterRole.rulesType;
local extra_cluster_role_resources = policyRule.new() +
                                     policyRule.withApiGroups(['']) +
                                     policyRule.withResources(['services','pods','endpoints']) +
                                     policyRule.withVerbs(['get','list','watch']);

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    prometheus+:: {
      clusterRole+: {
        rules+: [extra_cluster_role_resources],
      },
    },
  };

These changes resulted in my ClusterRole looking like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  - pods
  - endpoints
  verbs:
  - get
  - list
  - watch