Kube-state-metrics: Observing More Than One Ns Leads to Inconsistent Metrics Being Exported

[paxbit]

/kind bug

What happened:

Note: I'm going to name only the kube_pod_container_resource_requests metric in my following explanation. The problem affects all exported metrics though, just with different counts and numbers than the ones I wrote down for kube_pod_container_resource_requests. To verify the amount of metrics being exported I directly fetch the kube-state-metrics metrics endpoint with a browser and count the number of series occurrences using the find feature.

I'm running kube-state-metrics trying to observe a total of 148 PODs over 17 namespaces running 175 containers. Therefore I'm expecting to see >at least< 175x2=350 kube_pod_container_resource_requests metrics being exported. Two for each container - resource="cpu" and resource="memory". I'm saying "at least", since kube-state-metrics also exports completed POD resource metrics while I'm only interested and counting running PODs.
However I only see around 20-25 of those metrics when observing all 17 namespaces. As soon as I start lowering the amount of observed namespaces to the first 8 of those 17, I see more metrics. In those first 8 NS 129 containers are running - and I can see 218 of the expected 129x2=258 resource metrics.

As long as I observe only one of those namespaces everything seems fine and I can find both memory and cpu metrics for each container in the metrics endpoint. The same applies for all the other exported metrics.

Also, over time the amount of exported metrics seems to decline. Right after start-up the kube-state-metrics instance might export the above stated 218 resource series. However when I let it sit for an hour or so that amount might have dropped to e.g. 83. Then a couple minutes later it might go up again to 102 or so but never reach the expected amount relating to the actual container count.

What I also find quite weird is the fact that when I keep reloading the metrics endpoint over and over again I may always see say only 10 metrics when I might expect 20. But every time I reload those 10 metrics being exported seem to randomly window over an internally larger known subset of metrics as I always get a different set of 10.

There is nothing in terms of errors in the logs. Only the to be expected watch expiration warnings.

What you expected to happen:

Always export all metrics of all observed resources.

How to reproduce it (as minimally and precisely as possible):

IDK. Come to my desk? o_0
After reading on a couple of other issues here I cannot believe kube-state-metrics is hitting a resource barrier here, especially since there is nothing in the logs pointing to any sort of failure. Also I don't think I'm doing anything fancy. Just trying to consistently observe the PODs of more than one namespace.

Anything else we need to know?:

Since we're running this metrics instance specifically for a sub-project it is part of the specific application monitoring infrastructure and not deployed at the cluster level. So I set it up like noted here: https://github.com/kubernetes/kube-state-metrics#limited-privileges-environment

When testing I have all PODs under my control. So all of this cannot be caused by POD life-cycling caused by someone else.

Environment:

Args:

• --collectors=pods
• --v=5
• --namespace=[1..17]

Tried:

• quay.io/coreos/kube-state-metrics:v1.7.2
• quay.io/coreos/kube-state-metrics:v1.8.0

Kubernetes version:

• openshift v3.11.135
• kubernetes v1.11.0+d4cacc0

Requests:

• 1core/800M (I used the recommended defaults initially)

Limits:

• 2cores/2G

Thank you for your time! I appreciate any help. Maybe some of you have ideas how to debug this, where to look and what to check.

[lilic]

@paxbit Thanks for your issue. Sounds like a problem, do you mind providing a few more details on this.

When you say:

As soon as I start lowering the amount of observed namespaces to the first 8 of those 17,

How do you lower the amount of the observed namespaces?

[paxbit]

@LiliC Of course. Thank you for your reply!

How do you lower the amount of the observed namespaces?

I change the --namespaces config flag and redeploy the kube-state-metrics POD. I then waited up to a couple of hours for the state to develop over time. But mostly I was changing the namespace list, restarted and took a sample of the amount of metrics minutes after the kube-state-metrics POD became ready.

[brancz]

Could you setup:

• one instance of kube-state-metrics with just namespace A
• one instance of kube-state-metrics with just namespace B
• one instance of kube-state-metrics with just namespace A, B

and compare them in the same moment of time, and see if these inconsistencies still apply? Metrics can vary over time generally, that's totally possible, but in the same moment of time these should be consistent.

[paxbit]

Ok I did that and also hacked together the following script to find differences.

#!/usr/bin/env bash

AB_INSTANCE=https://[ab-instance]/metrics
A_INSTANCE=https://[a-instance]/metrics
B_INSTANCE=https://[b-instance]/metrics

OUT_FOLDER=temp

mkdir -p ${OUT_FOLDER}

for iteration in {1..1000}
do
  SAMPLE_TS=$(date)

  wget -q ${AB_INSTANCE} -O ${OUT_FOLDER}/AB_${iteration}.out &
  wget -q ${A_INSTANCE} -O ${OUT_FOLDER}/A_${iteration}.out &
  wget -q ${B_INSTANCE} -O ${OUT_FOLDER}/B_${iteration}.out &

  sleep 3

  if [[ ! -f ${OUT_FOLDER}/AB_${iteration}.out ]] || [[ ! -f ${OUT_FOLDER}/A_${iteration}.out ]] || [[ ! -f ${OUT_FOLDER}/B_${iteration}.out ]]; then
    echo "Expectation failed! Missing output file."
    exit 1
  fi

  for subset in A B; do

    TOTAL=0
    MISSING_IN_AB=0
    FOUND_IN_AB=0

    for line in $(grep -oP '^kube_pod[^\}]+\}' ${OUT_FOLDER}/${subset}_${iteration}.out); do
      ((TOTAL++))
      if grep -q "$line" ${OUT_FOLDER}/AB_${iteration}.out; then
        ((FOUND_IN_AB++))
      else
        ((MISSING_IN_AB++))
      fi
    done

    echo
    echo "# Looking for $subset in AB stats at $SAMPLE_TS"
    echo "  Total metrics # in $subset: $TOTAL"
    echo "  Total metrics # in AB: $(grep -P '^kube_pod' ${OUT_FOLDER}/AB_${iteration}.out | wc -l)"
    echo "  Found in AB: $FOUND_IN_AB"
    echo "  Missing in AB: $MISSING_IN_AB"
  done

done

1st Trial - While I Was Hacking the Script

The script output after running all three instances for an hour and a bit. I don't have the output from the time directly after starting all three instances b/c there was no script then.

After an hour:

• A is fully contained within AB.
• B is nowhere to be found in AB.

(excerpt)

# Looking for B in AB stats at Thu Oct 10 16:09:36 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:10:00 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:10:00 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:10:22 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:10:22 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:10:45 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:10:45 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:11:08 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:11:08 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:11:31 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:11:31 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:11:54 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:11:54 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 16:12:20 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 16:12:20 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

I manually verified this by looking into those output files and taking some samples looking for A and B in AB.

2nd Trial

After redeploying all three instances I get those results right after they become ready.

# Looking for A in AB stats at Thu Oct 10 16:15:29 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 16:15:29 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 16:15:53 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 16:15:53 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 16:16:16 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 16:16:16 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 16:16:40 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 16:16:40 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 16:17:05 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 16:17:05 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 16:17:28 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892
...

Then I let it run for around 1.5h without observing any change to those numbers.

At 15:38:16 I see the first watch expiration for the AB instance in its logs:

W1010 15:38:16.539229       1 reflector.go:302] k8s.io/kube-state-metrics/internal/store/builder.go:314: watch of *v1.Pod ended with: too old resource version: 191092426 (191099236)

At the same time I see a change in the available metrics in AB:

# Looking for B in AB stats at Thu Oct 10 17:36:55 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 17:37:23 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 17:37:23 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 17:37:51 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2589
  Found in AB: 474
  Missing in AB: 892

# Looking for B in AB stats at Thu Oct 10 17:37:51 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2589
  Found in AB: 2115
  Missing in AB: 0

# Looking for A in AB stats at Thu Oct 10 17:38:20 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 2115
  Found in AB: 0
  Missing in AB: 1366

# Looking for B in AB stats at Thu Oct 10 17:38:20 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 2115
  Found in AB: 2115
  Missing in AB: 0

Minutes later at 15:44:39 the watches again expired for AB and also the first time for A and B at 15:43:40 and 15:43:43 respectively.

AB Instance Log

W1010 15:38:16.539229       1 reflector.go:302] k8s.io/kube-state-metrics/internal/store/builder.go:314: watch of *v1.Pod ended with: too old resource version: 191092426 (191099236)
W1010 15:44:39.557192       1 reflector.go:302] k8s.io/kube-state-metrics/internal/store/builder.go:314: watch of *v1.Pod ended with: too old resource version: 191092621 (191106754)

A Instance Log

W1010 15:43:40.698716       1 reflector.go:302] k8s.io/kube-state-metrics/internal/store/builder.go:314: watch of *v1.Pod ended with: too old resource version: 191092621 (191105377)

B Instance Log

W1010 15:43:43.512716       1 reflector.go:302] k8s.io/kube-state-metrics/internal/store/builder.go:314: watch of *v1.Pod ended with: too old resource version: 191092426 (191105476)

After those events the numbers changed to:

# Looking for A in AB stats at Thu Oct 10 17:44:34 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 17:44:34 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

# Looking for A in AB stats at Thu Oct 10 17:44:59 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Thu Oct 10 17:44:59 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

Again I verified manually there is no instance of any B metric in AB. A is still fully covered by AB though.

[paxbit]

...
To me it looks like the watch expirations (which I understand are normal in a Kubernetes cluster) seem to make those kube-state-metrics instances "forget" even more things - after not picking up the full AB state initially.

[paxbit]

...
it's the next day, I kept the instances running overnight. Still shows the same picture after ~19h.

Somehow it looks like namespace B is being completely discarded from observation by the AB instance after some time.

# Looking for A in AB stats at Fri Oct 11 11:18:19 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 1366
  Found in AB: 1366
  Missing in AB: 0

# Looking for B in AB stats at Fri Oct 11 11:18:19 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 1366
  Found in AB: 0
  Missing in AB: 2115

[paxbit]

... I flipped the --namespaces flag of the AB instance from --namespaces=A,B to --namespaces=B,A

Right after start it seem that now all B metrics are covered by the BA instance while some of the A metrics are missing from it. Also the instance observing two namespaces this time discovered 3338 metrics for A and B combined while in the previous run it only saw 2589.
The two instances only observing one NS so far always discovered the same amount of metrics for A and B (1366 and 2115).

# Looking for A in AB stats at Fri Oct 11 11:25:36 CEST 2019
  Total metrics # in A: 1366
  Total metrics # in AB: 3338
  Found in AB: 1223
  Missing in AB: 143

# Looking for B in AB stats at Fri Oct 11 11:25:36 CEST 2019
  Total metrics # in B: 2115
  Total metrics # in AB: 3338
  Found in AB: 2115
  Missing in AB: 0

[lilic]

@paxbit Sounds very interesting, thanks for your results and the work!!

I am curious if this is the problem with our namespace flag, or a general problem. For the AB experiment, did you also pass the namespaces AB via the args to kube-state-metircs. Are any of the above results for when the --namespace flag was not used?

[paxbit]

@LiliC No I always used the --namespace flag. So far I'm not aware of any other way to run in an unprivileged environment without a cluster role.

[lilic]

@paxbit So I tried to reproduce, and I noticed that in the namespaced metrics there are also cluster wide resources so that accounted for the difference for me. Can you double-check that is not also causing the problems for you as well? Thanks again!

[paxbit]

@LiliC Thanks for checking! I'm not sure I understand you correctly. Are you saying that when you configure your instance with --namespace=A you also see resources from a namespace like kube-system?

In any case I checked the output of my A, B and AB instances and find that:
All metrics relate to a resource in the namespaces A and B. I see nothing cluster wide in A, B and AB.

Also how would that explain that when I configure three instances with:

1. --namespace=A
2. --namespace=B
3. --namespace=A,B

... that AB does not contain a complete aggregate of A and B? And also that after a watch expires (minutes up to 1.5h after restart in my trials), the AB instance "forgets" everything from B?

[brancz]

Could you share the exact flags that you are using? Maybe even the entire collection of manifests?

[paxbit]

@brancz Sure, here's the template with only the route host names redacted and actual namespace names anonymized.

---
kind: Template
apiVersion: v1
metadata:
  name: kube-state-metrics
objects:
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: kube-state-metrics
      namespace: ns-4A68G9
      labels:
        app: kube-state-metrics
  - apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
      name: kube-state-metrics
      namespace: ns-4A68G9
      labels:
        app: kube-state-metrics
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: view
    subjects:
      - kind: ServiceAccount
        name: kube-state-metrics
        namespace: ns-4A68G9
  - apiVersion: v1
    kind: Service
    metadata:
      name: kube-state-metrics
      namespace: ns-4A68G9
      labels:
        app: kube-state-metrics
        k8s-app: kube-state-metrics
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http-metrics
          port: 8080
          targetPort: http-metrics
          protocol: TCP
        - name: telemetry
          port: 8081
          targetPort: telemetry
          protocol: TCP
        - name: http-metrics-2
          port: 8180
          targetPort: http-metrics-2
          protocol: TCP
        - name: telemetry-2
          port: 8181
          targetPort: telemetry-2
          protocol: TCP
        - name: http-metrics-3
          port: 8280
          targetPort: http-metrics-3
          protocol: TCP
        - name: telemetry-3
          port: 8281
          targetPort: telemetry-3
          protocol: TCP
      selector:
        k8s-app: kube-state-metrics
  - apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      labels:
        app: kube-state-metrics
        k8s-app: kube-state-metrics
      name: kube-state-metrics-1
      namespace: ns-4A68G9
    spec:
      host: [redacted]
      port:
        targetPort: http-metrics
      tls:
        termination: edge
      to:
        kind: Service
        name: kube-state-metrics
        weight: 100
      wildcardPolicy: None
  - apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      labels:
        app: kube-state-metrics-2
        k8s-app: kube-state-metrics
      name: kube-state-metrics-2
      namespace: ns-4A68G9
    spec:
      host: [redacted]
      port:
        targetPort: http-metrics-2
      tls:
        termination: edge
      to:
        kind: Service
        name: kube-state-metrics
        weight: 100
      wildcardPolicy: None
  - apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      labels:
        app: kube-state-metrics-3
        k8s-app: kube-state-metrics
      name: kube-state-metrics-3
      namespace: ns-4A68G9
    spec:
      host: [redacted]
      port:
        targetPort: http-metrics-3
      tls:
        termination: edge
      to:
        kind: Service
        name: kube-state-metrics
        weight: 100
      wildcardPolicy: None
  - apiVersion: apps.openshift.io/v1
    kind: DeploymentConfig
    metadata:
      labels:
        app: kube-state-metrics
        k8s-app: kube-state-metrics
      name: kube-state-metrics
    spec:
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        app: kube-state-metrics
        deploymentconfig: kube-state-metrics
        k8s-app: kube-state-metrics
      strategy:
        resources:
          limits:
            cpu: '1'
            memory: 1G
          requests:
            cpu: '0.4'
            memory: 400M
        rollingParams:
          intervalSeconds: 1
          maxSurge: 25%
          maxUnavailable: 25%
          timeoutSeconds: 600
          updatePeriodSeconds: 1
        type: Rolling
      template:
        metadata:
          labels:
            app: kube-state-metrics
            k8s-app: kube-state-metrics
            deploymentconfig: kube-state-metrics
        spec:
          serviceAccountName: kube-state-metrics
          containers:
            - env:
              image: quay.io/coreos/kube-state-metrics:v1.8.0
              imagePullPolicy: Always
              name: kube-state-metrics-1
              args:
                - '--collectors=pods'
                - '--v=10'
                - '--namespace=ns-4A68G9,ns-2B384T'
              ports:
                - name: http-metrics
                  containerPort: 8080
                - name: telemetry
                  containerPort: 8081
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 8080
                initialDelaySeconds: 5
                timeoutSeconds: 5
              readinessProbe:
                httpGet:
                  path: /
                  port: 8080
                initialDelaySeconds: 5
                timeoutSeconds: 5
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            - image: quay.io/coreos/kube-state-metrics:v1.8.0
              imagePullPolicy: Always
              name: kube-state-metrics-2
              args:
                - '--collectors=pods'
                - '--v=10'
                - '--port=8180'
                - '--telemetry-port=8181'
                - '--namespace=ns-4A68G9'
              ports:
                - name: http-metrics-2
                  containerPort: 8180
                - name: telemetry-2
                  containerPort: 8181
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 8180
                initialDelaySeconds: 5
                timeoutSeconds: 5
              readinessProbe:
                httpGet:
                  path: /
                  port: 8180
                initialDelaySeconds: 5
                timeoutSeconds: 5
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            - image: quay.io/coreos/kube-state-metrics:v1.8.0
              imagePullPolicy: Always
              name: kube-state-metrics-3
              args:
                - '--collectors=pods'
                - '--v=10'
                - '--port=8280'
                - '--telemetry-port=8281'
                - '--namespace=ns-2B384T'
              ports:
                - name: http-metrics-3
                  containerPort: 8280
                - name: telemetry-3
                  containerPort: 8281
              livenessProbe:
                httpGet:
                  path: /healthz
                  port: 8280
                initialDelaySeconds: 5
                timeoutSeconds: 5
              readinessProbe:
                httpGet:
                  path: /
                  port: 8280
                initialDelaySeconds: 5
                timeoutSeconds: 5
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30

[lilic]

I could reproduce this once, but cannot reproduce it consistently. 🤔

[lilic]

Just an update: I will try to take some time next week to dig a bit deeper, a bit busy right now.

[paxbit]

@LiliC Thanks! Just let me know if I can provide additional data.

[paxbit]

@LiliC Hi. Have you had a chance to take a look at the problem so far?

[lilic]

@paxbit Not yet I apologize, but its on the top of my todo list for this week!

[lilic]

@paxbit So I finally got a chance to reproduce this. And I have managed to reproduce it! But only when the namespaces were limited. So @tariq1890 @brancz we have a bug in the namespaces options. No clue as of now where this bug might be.

My results, I deployed 3 kube-state-metrics running in different namespaces:

Scenario 1:

A ksm -> namespace A, B ksm -> namespace B, AB ksm -> all namespaces - no restrictions

Conclusion for scenario 1:

No missing metrics were found, not restricting namespaces works as expected.

# Looking for B in AB stats at Fri Nov  1 14:40:35 CET 2019
  Total metrics # in B: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for A in AB stats at Fri Nov  1 14:40:39 CET 2019
  Total metrics # in A: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for B in AB stats at Fri Nov  1 14:40:39 CET 2019
  Total metrics # in B: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for A in AB stats at Fri Nov  1 14:40:42 CET 2019
  Total metrics # in A: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for B in AB stats at Fri Nov  1 14:40:42 CET 2019
  Total metrics # in B: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for A in AB stats at Fri Nov  1 14:40:46 CET 2019
  Total metrics # in A: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

# Looking for B in AB stats at Fri Nov  1 14:40:46 CET 2019
  Total metrics # in B: 41
  Total metrics # in AB:      514
  Found in AB: 41
  Missing in AB: 0

Scenario 2:

A ksm -> namespace A, B ksm -> namespace B, AB ksm -> used flag --namespace=a,b

Conclusion for scenario 2:

Missing metrics printed out, I confirmed this also by manually querying the /metrics endpoint and indeed the metrics were missing in AB that were there in B.
One thing that might help us pin down the problem that I found, was it was only missing metrics per namespace per resource, we never had missing metrics for one resource from both namespaces.

I also restarted AB kube-state-metrics and after restart, we got different metrics missing.

missing: kube_pod_info{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",host_ip="10.0.2.15",pod_ip="172.17.0.6",uid="e621204f-d008-4ca6-8d9e-8efe1ef54ecc",node="minikube",created_by_kind="ReplicaSet",created_by_name="kube-state-metrics-696dd8696b",priority_class=""}
missing: kube_pod_start_time{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r"}
missing: kube_pod_owner{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",owner_kind="ReplicaSet",owner_name="kube-state-metrics-696dd8696b",owner_is_controller="true"}
missing: kube_pod_labels{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",label_app_kubernetes_io_name="kube-state-metrics",label_app_kubernetes_io_version="v1.8.0",label_pod_template_hash="696dd8696b"}
missing: kube_pod_created{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r"}
missing: kube_pod_restart_policy{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",type="Always"}
missing: kube_pod_status_scheduled_time{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r"}
missing: kube_pod_status_phase{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",phase="Pending"}
missing: kube_pod_status_phase{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",phase="Succeeded"}
missing: kube_pod_status_phase{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",phase="Failed"}
missing: kube_pod_status_phase{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",phase="Running"}
missing: kube_pod_status_phase{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",phase="Unknown"}
missing: kube_pod_status_ready{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="true"}
missing: kube_pod_status_ready{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="false"}
missing: kube_pod_status_ready{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="unknown"}
missing: kube_pod_status_scheduled{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="true"}
missing: kube_pod_status_scheduled{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="false"}
missing: kube_pod_status_scheduled{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",condition="unknown"}
missing: kube_pod_container_info{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",image="quay.io/coreos/kube-state-metrics:v1.8.0",image_id="docker-pullable://quay.io/coreos/kube-state-metrics@sha256:f75c3e5c5c7f65846ddd6883d6187b38f77721a3938f241c9e5d0ebe7beb8e19",container_id="docker://d9a2306f186463a500ef637ae6e70d0c38ac5dc0ae66a95e24fb23764cdd1ad8"}
missing: kube_pod_container_status_waiting{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="ContainerCreating"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="CrashLoopBackOff"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="CreateContainerConfigError"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="ErrImagePull"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="ImagePullBackOff"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="CreateContainerError"}
missing: kube_pod_container_status_waiting_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="InvalidImageName"}
missing: kube_pod_container_status_running{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics"}
missing: kube_pod_container_status_terminated{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics"}
missing: kube_pod_container_status_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="OOMKilled"}
missing: kube_pod_container_status_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="Completed"}
missing: kube_pod_container_status_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="Error"}
missing: kube_pod_container_status_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="ContainerCannotRun"}
missing: kube_pod_container_status_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="DeadlineExceeded"}
missing: kube_pod_container_status_last_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="OOMKilled"}
missing: kube_pod_container_status_last_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="Completed"}
missing: kube_pod_container_status_last_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="Error"}
missing: kube_pod_container_status_last_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="ContainerCannotRun"}
missing: kube_pod_container_status_last_terminated_reason{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics",reason="DeadlineExceeded"}
missing: kube_pod_container_status_ready{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics"}
missing: kube_pod_container_status_restarts_total{namespace="standard-b",pod="kube-state-metrics-696dd8696b-cgs6r",container="kube-state-metrics"}

# Looking for A in AB stats at Fri Nov  1 14:52:31 CET 2019
  Total metrics # in A: 41
  Total metrics # in AB:       41
  Found in AB: 0
  Missing in AB: 41

# Looking for A in AB stats at Fri Nov  1 15:10:32 CET 2019
  Total metrics # in A: 167
  Total metrics # in AB:       41
  Found in AB: 121
  Missing in AB: 46
missing: kube_deployment_created{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_status_replicas{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_status_replicas_available{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_status_replicas_unavailable{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_status_replicas_updated{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_status_observed_generation{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_spec_replicas{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_spec_paused{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_spec_strategy_rollingupdate_max_unavailable{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_spec_strategy_rollingupdate_max_surge{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_metadata_generation{namespace="standard-a",deployment="kube-state-metrics"}
missing: kube_deployment_labels{namespace="standard-a",deployment="kube-state-metrics",label_app_kubernetes_io_name="kube-state-metrics",label_app_kubernetes_io_version="v1.8.0"}
missing: kube_replicaset_created{namespace="standard-a",replicaset="kube-state-metrics-cdfc69d4c"}
missing: kube_replicaset_created{namespace="standard-a",replicaset="kube-state-metrics-7c7bdd8ffb"}                                                                               missing: kube_replicaset_created{namespace="standard-a",replicaset="kube-state-metrics-7f8fbfb5f8"}
missing: kube_replicaset_status_replicas{namespace="standard-a",replicaset="kube-state-metrics-cdfc69d4c"}
missing: kube_replicaset_status_replicas{namespace="standard-a",replicaset="kube-state-metrics-7c7bdd8ffb"}
missing: kube_replicaset_status_replicas{namespace="standard-a",replicaset="kube-state-metrics-7f8fbfb5f8"}
missing: kube_replicaset_status_fully_labeled_replicas{namespace="standard-a",replicaset="kube-state-metrics-cdfc69d4c"}
missing: kube_replicaset_status_fully_labeled_replicas{namespace="standard-a",replicaset="kube-state-metrics-7c7bdd8ffb"}
missing: kube_replicaset_status_fully_labeled_replicas{namespace="standard-a",replicaset="kube-state-metrics-7f8fbfb5f8"}
missing: kube_replicaset_status_ready_replicas{namespace="standard-a",replicaset="kube-state-metrics-cdfc69d4c"}
missing: kube_replicaset_status_ready_replicas{namespace="standard-a",replicaset="kube-state-metrics-7c7bdd8ffb"}
missing: kube_replicaset_status_ready_replicas{namespace="standard-a",replicaset="kube-state-metrics-7f8fbfb5f8"}

[brancz]

I have a theory. I think what's happening is that we are creating multiple reflectors that act on a single store, resulting in Replace calls that replace the entire store with items from a reflector that is namespace + type scoped.

If this is true, then we need to ensure that we have a 1:1 relationship between our store and listwatch. Best would be that we create a synthetic multi-list-watch, that internally starts a listwatch per namespace and acts as if it was a single listwatch. This should be relatively easy to create a proof of concept for, as we've already done something very similar in the Prometheus Operator.

[lilic]

Forgot to say, but thanks @paxbit for reporting this and sorry for being so late to respond, busy times! Much appreciated!

[paxbit]

@LiliC @brancz Thanks for your looking into it! I appreciate it. For the moment I fell back to deploying one instance per namespace. 17 pods atm. It works but would be a waste of resources for going live next year where we'll have to monitor north of 100 NS.
One thing I'm curious about is why any how does this specific bug come up just now at 1.4(at least) through 1.8. If you assumption about the cause lying in the relationship between the store and the listwatch is correct @brancz to me it seems it should affect everyone always, as soon as more than one NS is being observed. ksm seems like a widely used component. So why does it seem to be OK for most people, which I assume it is because it the problem is very obvious once you start looking at the recorded series' which simply end at some point.
Or put differently, could I do anything to put myself into the same situation where I don't hit the metric loss when observing more than one NS?

[lilic]

@paxbit I believe more people just don't restrict the namespaces and use it as is, to watch all namespaces.

[brancz]

Yes I think it's exactly as @LiliC said. We at least never noticed this as we run it with watching all namespaces.

[paxbit]

@LiliC @brancz I see. So the reason is that with --all-namespaces ksm will establish a single, cluster-wide watch using it's cluster role - which I cannot use - instead of iterating all NS and watch them discretely.

[brancz]

Precisely.

[geekofalltrades]

We're suffering this issue as well. In our case, we're monitoring five namespaces, like so: --namespace=A,B,C,D,E

Here's a metric query:

count(kube_pod_labels) by (namespace)

| | |
| --- | --- |
| {namespace="B"} | 229 |
| {namespace="C"} | 1 |

All of the namespaces definitely have pods with labels on them, and namespace C definitely has more than 1.

In an unrelated question, it looks like using --namespace also discards metrics about non-namespaced resources like Nodes. Is there a way to get the non-namespaced resources and also limit namespaced resources?

[lilic]

@geekofalltrades the PR to fix this will be landing in the next release and is linked above, feel free to test it. :)

In an unrelated question, it looks like using --namespace also discards metrics about non-namespaced resources like Nodes. Is there a way to get the non-namespaced resources and also limit namespaced resources?

You can start ksm with all namespaces but filter out most resources and keep only non-namespaced resources.