Kube-prometheus: kube-state-metrics deployment fails - invalid memory address or nil pointer dereference

Created on 2 Apr 2020 · 3Comments · Source: prometheus-operator/kube-prometheus

What happened?
kube-state-metrics deployment fails, here are the logs for the kube-state-metrics container:

I0402 15:41:42.111277       1 main.go:86] Using default collectors
I0402 15:41:42.111356       1 main.go:98] Using all namespace
I0402 15:41:42.111370       1 main.go:139] metric white-blacklisting: blacklisting the following items: 
W0402 15:41:42.111391       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0402 15:41:42.112906       1 main.go:184] Testing communication with server
I0402 15:41:42.119869       1 main.go:189] Running with Kubernetes cluster version: v1.14+. git version: v1.14.10-gke.17. git tree state: clean. commit: bdceba0734835c6cb1acbd1c447caf17d8613b44. platform: linux/amd64
I0402 15:41:42.119897       1 main.go:191] Communication with server successful
I0402 15:41:42.120015       1 main.go:225] Starting metrics server: 127.0.0.1:8081
I0402 15:41:42.120149       1 metrics_handler.go:96] Autosharding disabled
I0402 15:41:42.207350       1 main.go:200] Starting kube-state-metrics self metrics server: 127.0.0.1:8082
I0402 15:41:42.207909       1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
E0402 15:41:42.317406       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 72 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x13c4de0, 0x2178810)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82
panic(0x13c4de0, 0x2178810)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.glob..func81(0xc0000c8bb8, 0xc00066c620)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:284 +0x1d4
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0000c8bb8, 0xc000448930)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:300 +0x5c
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0000c8bb8, 0x7f36ff3ed2f8, 0xc0000c8bb8, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003bcc00, 0x153f360, 0xc0000c8bb8, 0x0, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003bcc00, 0xc0000aac40, 0x4, 0x4, 0xc0000c0ad0, 0x8, 0x125574a0, 0x219a560)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc00039bc80, 0xc0000aac00, 0x4, 0x4, 0xc0000c0ad0, 0x8, 0x0, 0xc00044ea80)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc00039bc80, 0xc00048c480, 0xc0001a0c00, 0xc00065fbb8, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00039bc80, 0xc0001a0c00, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000431f78)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00065ff78, 0x3b9aca00, 0x0, 0x1, 0xc0001a0c00)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00039bc80, 0xc0001a0c00)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
    /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12601d4]

goroutine 72 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105
panic(0x13c4de0, 0x2178810)
    /usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kube-state-metrics/internal/store.glob..func81(0xc0000c8bb8, 0xc00066c620)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:284 +0x1d4
k8s.io/kube-state-metrics/internal/store.wrapHPAFunc.func1(0x153f360, 0xc0000c8bb8, 0xc000448930)
    /go/src/k8s.io/kube-state-metrics/internal/store/hpa.go:300 +0x5c
k8s.io/kube-state-metrics/pkg/metric.(*FamilyGenerator).Generate(...)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:39
k8s.io/kube-state-metrics/pkg/metric.ComposeMetricGenFuncs.func1(0x153f360, 0xc0000c8bb8, 0x7f36ff3ed2f8, 0xc0000c8bb8, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metric/generator.go:78 +0x122
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Add(0xc0003bcc00, 0x153f360, 0xc0000c8bb8, 0x0, 0x0)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:76 +0xf6
k8s.io/kube-state-metrics/pkg/metrics_store.(*MetricsStore).Replace(0xc0003bcc00, 0xc0000aac40, 0x4, 0x4, 0xc0000c0ad0, 0x8, 0x125574a0, 0x219a560)
    /go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:138 +0xa4
k8s.io/client-go/tools/cache.(*Reflector).syncWith(0xc00039bc80, 0xc0000aac00, 0x4, 0x4, 0xc0000c0ad0, 0x8, 0x0, 0xc00044ea80)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:354 +0xf8
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc00039bc80, 0xc00048c480, 0xc0001a0c00, 0xc00065fbb8, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:250 +0x8fa
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc00039bc80, 0xc0001a0c00, 0x0, 0x0)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:257 +0x1a9
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:155 +0x33
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000431f78)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00065ff78, 0x3b9aca00, 0x0, 0x1, 0xc0001a0c00)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc00039bc80, 0xc0001a0c00)
    /go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:154 +0x16b
created by k8s.io/kube-state-metrics/internal/store.(*Builder).reflectorPerNamespace
    /go/src/k8s.io/kube-state-metrics/internal/store/builder.go:337 +0x265

Did you expect to see some different?
It worked pretty well, it might have happened after jb update I did a while ago and didn't notice.

How to reproduce it (as minimally and precisely as possible):
Pretty minimal deployment of the kube-prometheus stack with no major tweaks or edits.

Environment

Prometheus Operator version:

v0.34.1
Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"archive", BuildDate:"2020-02-29T16:37:45Z", GoVersion:"go1.14", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.17", GitCommit:"bdceba0734835c6cb1acbd1c447caf17d8613b44", GitTreeState:"clean", BuildDate:"2020-01-17T23:10:13Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

Managed (GCP)
Manifests:

kube-state-metrics-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - args:
        - --logtostderr
        - --secure-listen-address=:8443
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        - --upstream=http://127.0.0.1:8081/
        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
        name: kube-rbac-proxy-main
        ports:
        - containerPort: 8443
          name: https-main
        resources:
          limits:
            cpu: 20m
            memory: 40Mi
          requests:
            cpu: 10m
            memory: 20Mi
      - args:
        - --logtostderr
        - --secure-listen-address=:9443
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        - --upstream=http://127.0.0.1:8082/
        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
        name: kube-rbac-proxy-self
        ports:
        - containerPort: 9443
          name: https-self
        resources:
          limits:
            cpu: 20m
            memory: 40Mi
          requests:
            cpu: 10m
            memory: 20Mi
      - args:
        - --host=127.0.0.1
        - --port=8081
        - --telemetry-host=127.0.0.1
        - --telemetry-port=8082
        image: quay.io/coreos/kube-state-metrics:v1.9.2
        name: kube-state-metrics
        resources:
          limits:
            cpu: 100m
            memory: 150Mi
          requests:
            cpu: 100m
            memory: 150Mi
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: kube-state-metrics

Prometheus Operator Logs:

ts=2020-04-02T15:24:47.955126738Z caller=main.go:199 msg="Starting Prometheus Operator version '0.34.1'."
ts=2020-04-02T15:24:48.046544796Z caller=main.go:96 msg="Staring insecure server on :8080"
level=info ts=2020-04-02T15:24:48.053594553Z caller=operator.go:219 component=alertmanageroperator msg="connection established" cluster-version=v1.14.10-gke.17
level=info ts=2020-04-02T15:24:48.053704862Z caller=operator.go:441 component=prometheusoperator msg="connection established" cluster-version=v1.14.10-gke.17
level=info ts=2020-04-02T15:24:48.845646704Z caller=operator.go:641 component=alertmanageroperator msg="CRD updated" crd=Alertmanager
level=info ts=2020-04-02T15:24:48.947172907Z caller=operator.go:1880 component=prometheusoperator msg="CRD updated" crd=Prometheus
level=info ts=2020-04-02T15:24:48.961361717Z caller=operator.go:1880 component=prometheusoperator msg="CRD updated" crd=ServiceMonitor
level=info ts=2020-04-02T15:24:48.972301171Z caller=operator.go:1880 component=prometheusoperator msg="CRD updated" crd=PodMonitor
level=info ts=2020-04-02T15:24:48.98201882Z caller=operator.go:1880 component=prometheusoperator msg="CRD updated" crd=PrometheusRule
level=info ts=2020-04-02T15:24:51.948017653Z caller=operator.go:235 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2020-04-02T15:24:52.348570535Z caller=operator.go:190 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2020-04-02T15:24:52.348840574Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:24:52.545295117Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:24:52.885585262Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:25:01.061379327Z caller=operator.go:457 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2020-04-02T15:25:03.374274676Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:25:03.759379616Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:25:04.164521873Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:25:07.84499562Z caller=operator.go:387 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2020-04-02T15:25:07.845227048Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:25:08.37366645Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:25:08.503553621Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:25:08.677858043Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:25:12.899258468Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:25:18.044220686Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
W0402 15:33:27.064164       1 reflector.go:299] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:468: watch of *v1.ConfigMap ended with: too old resource version: 45866978 (45874838)
level=info ts=2020-04-02T15:34:51.950893169Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:36:19.186407142Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:49.256844739Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:49.366374839Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:49.476518835Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:54.124168591Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:54.168062796Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:36:58.721641629Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
W0402 15:39:33.061624       1 reflector.go:299] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:468: watch of *v1.ConfigMap ended with: too old resource version: 45876337 (45881818)
level=info ts=2020-04-02T15:39:51.951098821Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:43:52.488847894Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:53.335027771Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:53.457828035Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:53.578944046Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:53.760596964Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:54.059246964Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:43:56.202143719Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:44:51.951361332Z caller=operator.go:462 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
level=info ts=2020-04-02T15:45:24.550346486Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:41.563286026Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:46.270598373Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:46.459118482Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:46.593671467Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:46.761768473Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2020-04-02T15:47:47.79404395Z caller=operator.go:1072 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s

Prometheus Logs:

level=info ts=2020-04-02T15:25:15.225Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.11.0, branch=HEAD, revision=4ef66003d9855ed2b7a41e987b33828ec36db34d)"
level=info ts=2020-04-02T15:25:15.225Z caller=main.go:330 build_context="(go=go1.12.7, user=root@0dc27cf95f36, date=20190709-09:54:35)"
level=info ts=2020-04-02T15:25:15.225Z caller=main.go:331 host_details="(Linux 4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT 2019 x86_64 prometheus-k8s-0 (none))"
level=info ts=2020-04-02T15:25:15.225Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-04-02T15:25:15.225Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-04-02T15:25:15.227Z caller=web.go:448 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-04-02T15:25:15.227Z caller=main.go:652 msg="Starting TSDB ..."
level=info ts=2020-04-02T15:25:15.240Z caller=main.go:667 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-04-02T15:25:15.240Z caller=main.go:668 msg="TSDB started"
level=info ts=2020-04-02T15:25:15.240Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-02T15:25:15.244Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:15.245Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:15.246Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:15.248Z caller=kubernetes.go:192 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:15.314Z caller=main.go:766 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-02T15:25:15.314Z caller=main.go:621 msg="Server is ready to receive web requests."
level=info ts=2020-04-02T15:25:19.033Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-02T15:25:19.042Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.043Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.045Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.047Z caller=kubernetes.go:192 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.105Z caller=main.go:766 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=info ts=2020-04-02T15:25:19.106Z caller=main.go:738 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=error ts=2020-04-02T15:25:19.114Z caller=endpoints.go:131 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-02T15:25:19.114Z caller=endpoints.go:131 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=error ts=2020-04-02T15:25:19.115Z caller=endpoints.go:131 component="discovery manager scrape" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2020-04-02T15:25:19.115Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.116Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.117Z caller=kubernetes.go:192 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2020-04-02T15:25:19.118Z caller=kubernetes.go:192 component="discovery manager notify" discovery=k8s msg="Using pod service account via in-cluster config"
level=error ts=2020-04-02T15:25:19.120Z caller=endpoints.go:131 component="discovery manager notify" discovery=k8s role=endpoint msg="endpoints informer unable to sync cache"
level=info ts=2020-04-02T15:25:19.164Z caller=main.go:766 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
level=warn ts=2020-04-02T15:32:03.129Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 45867264 (45868707)"
level=warn ts=2020-04-02T15:39:12.136Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 45874719 (45877109)"
level=warn ts=2020-04-02T15:45:35.143Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 45883024 (45884361)"

Anything else we need to know?:
Got to this kube-state-metrics issue which seems like it's supposed to be fixed already (as you can see i'm using v1.9.2).

Source

argamanza

Most helpful comment

Like Damien said this is fixed in kube-state-metrics, what version/commit of kube-prometheus are you using? In latest commit/master we are at 1.9.5 version of kube-state-metrics https://github.com/coreos/kube-prometheus/blob/master/manifests/kube-state-metrics-deployment.yaml#L26

lilic on 2 Apr 2020

❤2

All 3 comments

Hi @argamanza, you might want to look into this PR: https://github.com/kubernetes/kube-state-metrics/pull/1030

This error was recurring in kube-state-metric, thus they dropped the related metrics in v1.9.3. Upgrading kube-state-metric to v1.9.3 should solve your problem.