Helm-charts: [prometheus-kube-stack] High memory pressure

[shibumi]

Describe the bug

I have deployed a standard kube-prometheus-stack on a cluster with 9 worker nodes.
This lead to massive memory consumption by prometheus and an outage of the worker node running prometheus.

In the prometheus targets section I see 44+44+44+44 kubelet targets and per kubelet target multiple identical metric endpoints. Is it possible that this is a helm chart issue?

Also the helm chart deployment takes up to 20min for me (over LTE, I have no stable internet connection right now.. but blackbox exporter for example is significantly faster (just a few seconds).

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.15.3"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"archive", BuildDate:"2020-10-16T20:58:56Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6", GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean", BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Which chart: kube-prometheus-stack
Which version of the chart: 11.0.0
App version: 0.43.0

What happened:

Prometheus consumed way too much memory and started swapping generating a huge disk I/O on the node, thus crashing the kubelet and kubeproxy.

What you expected to happen:

Prometheus should need way less memory for such a small cluster (9 worker nodes)

How to reproduce it (as minimally and precisely as possible):

diff --git a/default.yaml b/kube-prometheus-stack.yaml
index 47202be..760080a 100644
--- a/default.yaml
+++ b/kube-prometheus-stack.yaml
@@ -46,7 +46,7 @@ defaultRules:
     kubernetesResources: true
     kubernetesStorage: true
     kubernetesSystem: true
-    kubeScheduler: true
+    kubeScheduler: false
     kubeStateMetrics: true
     network: true
     node: true
@@ -146,18 +146,17 @@ alertmanager:
   config:
     global:
       resolve_timeout: 5m
+    receivers:
+      - name: prometheus-msteams
+        webhook_configs:
+          - url: "http://prometheus-msteams:2000/alertmanager"
+            send_resolved: true
     route:
-      group_by: ['job']
+      group_by: [...]
       group_wait: 30s
       group_interval: 5m
       repeat_interval: 12h
-      receiver: 'null'
-      routes:
-      - match:
-          alertname: Watchdog
-        receiver: 'null'
-    receivers:
-    - name: 'null'
+      receiver: prometheus-msteams

   ## Pass the Alertmanager configuration directives through Helm's templating
   ## engine. If the Alertmanager configuration contains Alertmanager templates,
@@ -782,14 +781,14 @@ kubelet:
 ## Component scraping the kube controller manager
 ##
 kubeControllerManager:
-  enabled: true
+  enabled: false

   ## If your kube controller manager is not deployed as a pod, specify IPs it can be found on
   ##
-  endpoints: []
-  # - 10.141.4.22
-  # - 10.141.4.23
-  # - 10.141.4.24
+  endpoints:
+   - 10.101.0.9
+   - 10.101.0.10
+   - 10.101.0.11

   ## If using kubeControllerManager.endpoints only the port and targetPort are used
   ##
@@ -915,14 +914,14 @@ kubeDns:
 ## Component scraping etcd
 ##
 kubeEtcd:
-  enabled: true
+  enabled: false

   ## If your etcd is not deployed as a pod, specify IPs it can be found on
   ##
-  endpoints: []
-  # - 10.141.4.22
-  # - 10.141.4.23
-  # - 10.141.4.24
+  endpoints:
+   - 10.101.0.9
+   - 10.101.0.10
+   - 10.101.0.11

   ## Etcd service. If using kubeEtcd.endpoints only the port and targetPort are used
   ##
@@ -975,14 +974,14 @@ kubeEtcd:
 ## Component scraping kube scheduler
 ##
 kubeScheduler:
-  enabled: true
+  enabled: false

   ## If your kube scheduler is not deployed as a pod, specify IPs it can be found on
   ##
-  endpoints: []
-  # - 10.141.4.22
-  # - 10.141.4.23
-  # - 10.141.4.24
+  endpoints:
+   - 10.101.0.9
+   - 10.101.0.10
+   - 10.101.0.11

   ## If using kubeScheduler.endpoints only the port and targetPort are used
   ##
@@ -1432,11 +1431,11 @@ prometheus:

     ## Loadbalancer IP
     ## Only use if service.type is "loadbalancer"
-    loadBalancerIP: ""
+    loadBalancerIP: "10.101.0.207"
     loadBalancerSourceRanges: []
     ## Service type
     ##
-    type: ClusterIP
+    type: LoadBalancer

     sessionAffinity: ""

@@ -1882,15 +1881,14 @@ prometheus:
     ## Prometheus StorageSpec for persistent data
     ## ref: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md
     ##
-    storageSpec: {}
-    #  volumeClaimTemplate:
-    #    spec:
-    #      storageClassName: gluster
-    #      accessModes: ["ReadWriteOnce"]
-    #      resources:
-    #        requests:
-    #          storage: 50Gi
-    #    selector: {}
+    storageSpec:
+      volumeClaimTemplate:
+        spec:
+          storageClassName: managed-nfs-storage
+          accessModes: ["ReadWriteOnce"]
+          resources:
+            requests:
+              storage: 50Gi

     # Additional volumes on the output StatefulSet definition.
     volumes: []

Anything else we need to know:

[shibumi]

Why do we have multiple identical endpoints?

[shibumi]

The number of kubelet scrape targets is abnormal high, too (I think):

[mashun4ek]

I believe you need to review the services in your kube-system namespace. ServiceMonitor for kubelet is picking up the services from kube-system, and it looks like you have "extra", so delete unnecessary services or review the labels.

[shibumi]

@mashun4ek Hi Mariia, thanks a lot. There were indeed multiple different kubelet exporter services in the kube-system namespace. I wonder how they came up there.

This is what happened:

I installed one helm release of "prometheus-operator" called "monitoring" half a year ago... then I delete that later and installed a new one called "prometheus-operator".. then I deleted that, because the old helm chart got deprecated so I moved to kube-prometheus-stack helm char. The release was called "kube-prometheus-stack". Later on, I had trouble with this one, too. So I removed that and reinstalled as "prometheus-stack".

All other resources seems to get cleaned up after an helm uninstall,but the kubelet exporter service gets not properly uninstalled. Is this a known issue, a bug or just something that "should" work, but didn't work for me? :D

[mashun4ek]

I'm glad it worked! My guess is helm uninstall didn't delete the previous service from kube-system, not sure why though. Needs a further investigation 🕵️‍♀️

[monotek]

Crds are not deleted by helm. See: https://helm.sh/docs/topics/charts/#limitations-on-crds

[shibumi]

@mashun4ek if you need any debugging information, I am happy to help out.

@monotek correct me if I am wrong, but the issue wasn't the CRDs. The issue was a remaining service resource in the kube-system namespace.

[monotek]

As far as i understood the remaining resources have been of kind "ServiceMonitor" which is a custom resource created by prometheus operator. So if the prometheus operator is helm deleted the crd and the custom resources might have been kept.

To be honest it was just a guess as i'm not a user of the prometheus-kube-stack chart yet but as helms crd handling is not complete i thought it might be related.

[shibumi]

@monotek no. The remaining resource has been of kind "Service" and that should definitely been deleted by helm.