Charts: [stable/prometheus-operator] KubeletTooManyPods rule evaluation error

[Angivare]

Describe the bug
When installing the chart with kubelet.serviceMonitor.https=false, I get a PrometheusRuleFailures alert in the prometheus interface. Logs show the failing rule is KubeletTooManyPods

Version of Helm and Kubernetes:
helm v3.1.1
kubernetes v1.14.8 (deployed on AKS)

Which chart: stable/prometheus-operator 8.10.0

What happened: The default alert rule KubeletTooManyPods fails to be evaluated

What you expected to happen: Rule is evaluated correctly, no errors in logs and no RuleFailure alert

How to reproduce it (as minimally and precisely as possible):

helm install -n monitoring prometheus stable/prometheus-operator \
    --set kubelet.serviceMonitor.https=false

Anything else we need to know:
The incriminating log, found in the prometheus container within the prometheus pod:

level=warn ts=2020-02-28T20:34:35.340Z caller=manager.go:525
component="rule manager" group=kubernetes-system-kubelet
msg="Evaluating rule failed" rule="
alert: KubeletTooManyPods
expr: max by(node) (max by(instance) (kubelet_running_pod_count{job=\"kubelet\",metrics_path=\"/metrics\"})
  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})
  / max by(node) (kube_node_status_capacity_pods{job=\"kube-state-metrics\"}) > 0.95
for: 15m
labels:
  severity: warning
annotations:
  message: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage
    }} of its Pod capacity.
  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubelettoomanypods
"
err="
found duplicate series for the match group {instance=\"10.0.0.4:10255\"} on the right hand-side of the operation:
[
    {
        __name__=\"kubelet_node_name\",
        endpoint=\"http-metrics\",
        instance=\"10.0.0.4:10255\",
        job=\"kubelet\",
        metrics_path=\"/metrics\",
        namespace=\"kube-system\",
        node=\"aks-agentpool-32262492-vmss000000\",
        service=\"prometheus-prometheus-oper-kubelet\"
    },
    {
        __name__=\"kubelet_node_name\",
        endpoint=\"http-metrics\",
        instance=\"10.0.0.4:10255\",
        job=\"kubelet\",
        metrics_path=\"/metrics\",
        namespace=\"kube-system\",
        node=\"aks-agentpool-32262492-vmss000000\",
        service=\"prometheus-operator-kubelet\"
    }
];
many-to-many matching not allowed: matching labels must be unique on one side
"

[jasonsattler]

Hit the same issue. Sorry new to prometheus. I got it working by changing the expression to

max by(node) (kubelet_running_pod_count{job="kubelet",metrics_path="/metrics"}) / max by(node) (kube_node_status_capacity_pods{job="kube-state-metrics"}) > 0.95

I only run 1 kubelet per node anyways.

Update:
Ignore the above. The issue was that I renamed and redeployed and some services did not get cleaned up. So I did have duplicates that caused the issue.

[Sovietaced]

@jasonsattler where exactly do you change this?

[jasonsattler]

@jasonsattler where exactly do you change this?

https://github.com/helm/charts/blob/master/stable/prometheus-operator/templates/prometheus/rules-1.14/kubernetes-system-kubelet.yaml#L46

I am using argocd to deploy and downloaded the chart. So I just replaced the expr

[Sovietaced]

I may have found out what was going on in my cluster. I think I had some old resources laying around in kube-system from an old partial installation of prometheus-operator

[retr0h]

@Sovietaced can you expand on this?

What resources did you find laying around? I'm seeing this same error. Recently upgraded prom operator chart to 8.9.3.

[Sovietaced]

@Sovietaced can you expand on this?

What resources did you find laying around? I'm seeing this same error. Recently upgraded prom operator chart to 8.9.3.

I can't recall exactly at this moment. I may be able to reproduce the issue in a few days though.

It was a service under the kube-system namespace though. For whatever reason it doesn't get correctly cleaned up when the helm release is deleted.

[retr0h]

If i look at the query kubelet_node_name I see I get back duplicate metrics,where the service differs.

kubelet_node_name{endpoint="https-metrics",instance="x.x.196.11:10250",job="kubelet",metrics_path="/metrics",namespace="kube-system",node="csg-nscg-0008",service="prometheus-operator-kubelet"} | 1

kubelet_node_name{endpoint="https-metrics",instance="x.x.196.11:10250",job="kubelet",metrics_path="/metrics",namespace="kube-system",node="csg-nscg-0008",service="prometheus-prometheus-oper-kubelet"} | 1

So you are deff onto something. I do see duplicate

prometheus-prometheus-oper-etcd                                1      323d
prometheus-prometheus-oper-k8s-coredns                         1      323d
prometheus-prometheus-oper-k8s-node-rsrc-use                   1      323d
prometheus-prometheus-oper-k8s-resources-cluster               1      323d
prometheus-prometheus-oper-k8s-resources-namespace             1      323d
prometheus-prometheus-oper-k8s-resources-pod                   1      323d
prometheus-prometheus-oper-k8s-resources-workload              1      323d
prometheus-prometheus-oper-k8s-resources-workloads-namespace   1      323d
prometheus-prometheus-oper-nodes                               1      323d
prometheus-prometheus-oper-persistentvolumesusage              1      323d
prometheus-prometheus-oper-pods                                1      323d
prometheus-prometheus-oper-statefulset                         1      323d

prometheus-operator-etcd                                       1      238d
prometheus-operator-k8s-coredns                                1      238d
prometheus-operator-node-rsrc-use                              1      22d
prometheus-operator-k8s-resources-cluster                      1      238d
prometheus-operator-k8s-resources-namespace                    1      238d
prometheus-operator-k8s-resources-pod                          1      238d
prometheus-operator-k8s-resources-workload                     1      238d
prometheus-operator-k8s-resources-workloads-namespace          1      238d
prometheus-operator-nodes                                      1      238d
prometheus-operator-persistentvolumesusage                     1      238d
prometheus-operator-pods                                       1      238d
prometheus-operator-statefulset                                1      238d

Not sure if this is my problem tho.

[retr0h]

Ah, I have duplicate prom endpoints in kube-system!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[verstrae]

seems like the kubelet service is not cleaned up properly when destroying the chart

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[Baryczka]

I think this one is related to this ticket (https://github.com/helm/charts/issues/18761) as service resource not being cleaned up and metrics can be duplicated or even triplicated as I had the case

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.