Kube-state-metrics: Applying CPU limits to kube-state-metrics causes rapid memory consumption

[while1malloc0]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
Applying CPU limits to a pod running kube-state-metrics with no namespace flag causes unbounded memory allocation. This continues until the pod is OOM-killed by Kubernetes. This seems to happen reliably whenever CPU limits are applied. With no CPU limits, or with the --namespace flag supplied, the pod stays well under its memory limit. I observed this by applying a limit, running htop, and watching the process memory grow until it is killed.

The particular cluster is fairly large. Running kubectl get all --all-namespaces | wc -l returns 1375, so it seems reasonable that quite a bit memory would have to be allocated, as explained in this issue, but I can't seem to figure out why applying CPU limits would cause such rapid and unbounded memory allocation.

What you expected to happen:
Memory allocation does not grow as rapidly.

How to reproduce it (as minimally and precisely as possible):
Deploy a Kubernetes Deployment containing CPU resource limits to a cluster. Here's the deployment file that I'm seeing this behavior with:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.2.0
        ports:
        - name: http-metrics
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          limits:
            # removing this CPU limit reliably solves memory limit issues
            cpu: 200m
            memory: 4Gi
          requests:
            cpu: 200m
            memory: 4Gi

Anything else we need to know?

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-
27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kube-state-metrics image version: I've reproduced this issue with v1.2.0 and v1.3.0

[brancz]

In some sense kube-state-metrics has various queues internally, which work off incoming events, therefore when CPU is limited memory can increase more or less infinitely as the queue keeps growing.

My feeling is that we should remove the addon-resizer and only put requests, or no resources at all, until the next larger scalability test, so we can make better resource recommendations.

cc @piosz @DirectXMan12 what are your thoughts on what I suggested above?

[while1malloc0]

Bumping the CPU limits to 800m fixed the issue, which seems to confirm the internal-queue-length hypothesis. In case it's helpful, I created a small doc PR here letting other users know about the issue: https://github.com/kubernetes/kube-state-metrics/pull/462

[andyxning]

The output of kube-state-metrics heap pprof should give us more insight.

[andyxning]

In some sense kube-state-metrics has various queues internally, which work off incoming events, therefore when CPU is limited memory can increase more or less infinitely as the queue keeps growing.

Does this related to Golang GC?

[while1malloc0]

The output of kube-state-metrics heap pprof should give us more insight.

I took a profile while trying to diagnose the issue yesterday. Here's the output for top:

(pprof) top
Showing nodes accounting for 2566.26kB, 100% of 2566.26kB total
Showing top 10 nodes out of 63
      flat  flat%   sum%        cum   cum%
  518.02kB 20.19% 20.19%   518.02kB 20.19%  k8s.io/kube-state-metrics/vendor/github.com/beorn7/perks/quantile.newStream (inline)
  512.14kB 19.96% 40.14%   512.14kB 19.96%  reflect.mapassign
  512.05kB 19.95% 60.10%   512.05kB 19.95%  k8s.io/kube-state-metrics/vendor/github.com/json-iterator/go.describeStruct
  512.02kB 19.95% 80.05%   512.02kB 19.95%  k8s.io/kube-state-metrics/vendor/github.com/json-iterator/go.(*Iterator).ReadString
  512.02kB 19.95%   100%   512.02kB 19.95%  k8s.io/kube-state-metrics/vendor/k8s.io/client-go/tools/cache.MetaNamespaceKeyFunc
         0     0%   100%   518.02kB 20.19%  k8s.io/kube-state-metrics/collectors.(*cronJobCollector).Collect
         0     0%   100%   518.02kB 20.19%  k8s.io/kube-state-metrics/vendor/github.com/beorn7/perks/quantile.NewTargeted
         0     0%   100%   512.02kB 19.95%  k8s.io/kube-state-metrics/vendor/github.com/json-iterator/go.(*Iterator).ReadArrayCB
         0     0%   100%   512.14kB 19.96%  k8s.io/kube-state-metrics/vendor/github.com/json-iterator/go.(*Iterator).ReadMapCB
         0     0%   100%  1536.22kB 59.86%  k8s.io/kube-state-metrics/vendor/github.com/json-iterator/go.(*Iterator).ReadVal

[andyxning]

github.com/json-iterator is imported in k8s.io/apimachinery. Seems that this is related to the serializer of api objects.

./vendor/k8s.io/apimachinery/pkg/apis/meta/v1/BUILD:        "//vendor/github.com/json-iterator/go:go_default_library",
./vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/BUILD:        "//vendor/github.com/json-iterator/go:go_default_library",
./vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:   jsoniter "github.com/json-iterator/go"

/cc @kubernetes/sig-api-machinery-bugs @kubernetes/sig-api-machinery-api-reviews

[liggitt]

json iterator is simply what decodes objects, so it is where memory from incoming objects is allocated

the particular profile shows 2.5MB of memory, which is not particular concerning/interesting

perhaps a better question would be to look at what kube-state-metrics does with its queue of incoming objects to process. does it hold them all in memory if its processing queue is slow? does it dedupe repeated events for the same objects? if not, rapid changes to a few objects could easily cause unbounded memory growth if the processing speed was slower than the rate of change.

[brancz]

I suspect that the relation is that decoding performance is throttled when cpu limit is reached, therefore queueing up objects from the watch to be decoded therefore building up memory. kube-state-metrics watches pretty much every object in a kubernetes cluster, therefore I can definitely see this being impactful. I'm not 100% sure on the mechanics of how a watch works, but naively analyzing the situation the above seems plausible to me.

The problem is going to need resources, if the above is true, then now one can decide whether performance + non-staleness is important using high cpu or low/bounded cpu, possibly stale data and memory usage. A rather classic cpu/memory tradeoff I feel.

[andyxning]

perhaps a better question would be to look at what kube-state-metrics does with its queue of incoming objects to process. does it hold them all in memory if its processing queue is slow? does it dedupe repeated events for the same objects? if not, rapid changes to a few objects could easily cause unbounded memory growth if the processing speed was slower than the rate of change.

It would be pretty much good for client-go maintainers to add some detailed docs to the mechanism about watch. And some best practices are needed to make client-go well used or tuned to different kind of applications.

I suspect that the relation is that decoding performance is throttled when cpu limit is reached, therefore queueing up objects from the watch to be decoded therefore building up memory. kube-state-metrics watches pretty much every object in a kubernetes cluster, therefore I can definitely see this being impactful. I'm not 100% sure on the mechanics of how a watch works, but naively analyzing the situation the above seems plausible to me.

Sounds reasonable. We can just add a tip about this phenomenon to end users. Just like what @while1malloc0 does in #462 .