Kube-prometheus: Target Kubelet 0/0 up and others are down

[littlechicks]

Hello,

I have deployed kube-prometheus stack over Kubernetes.
Almost things are working except:

• Targets kubelet are not configured : it's showing 0/0 up
• Targets kube-prometheus-stack-kube-controller-manager are down
• Targets kube-prometheus-stack-kube-etcd are down
• Targets kube-prometheus-stack-kube-proxy are down
• Targets kube-prometheus-stack-kube-scheduler are down

For the last four error it seem like prometheus is using the node IP instead of ClusterIP.
The service for this four entries are correctly created but there is no Cluster IP assigned (see below)

kube-dns                                        ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP         5d22h
kube-prometheus-stack-coredns                   ClusterIP   None             <none>        9153/TCP                       13m
kube-prometheus-stack-kube-controller-manager   ClusterIP   None             <none>        10252/TCP                      13m
kube-prometheus-stack-kube-etcd                 ClusterIP   None             <none>        2379/TCP                       13m
kube-prometheus-stack-kube-proxy                ClusterIP   None             <none>        10249/TCP                      13m
kube-prometheus-stack-kube-scheduler            ClusterIP   None             <none>        10251/TCP                      13m
kube-prometheus-stack-kubelet                   ClusterIP   10.110.99.227    <none>        10250/TCP,10255/TCP,4194/TCP   3d23h
metrics-server                                  ClusterIP   10.100.147.217   <none>        443/TCP                        5d20h
tiller-deploy                                   ClusterIP   10.100.200.176   <none>        44134/TCP                      5d22h

Although the values.ymlare correctly set up for using Kubernetes Services as below:

kubeControllerManager:
  enabled: true
  service:
    port: 10252
    targetPort: 10252
    selector:
      component: kube-controller-manager
  serviceMonitor:
    interval: ""
    https: false
    insecureSkipVerify: null
    serverName: null
    metricRelabelings: []
    relabelings: []

kubeEtcd:
  enabled: true
  service:
    port: 2379
    targetPort: 2379
    selector:
      component: etcd
  serviceMonitor:
    interval: ""
    scheme: http
    insecureSkipVerify: false
    serverName: ""
    caFile: ""
    certFile: ""
    keyFile: ""
    metricRelabelings: []
    relabelings: []

kubeScheduler:
  enabled: true
  service:
    port: 10251
    targetPort: 10251
    selector:
      component: kube-scheduler
  serviceMonitor:
    interval: ""
    https: false
    insecureSkipVerify: null
    serverName: null
    metricRelabelings: []
    relabelings: []

kubeProxy:
  enabled: true
  service:
    port: 10249
    targetPort: 10249
    selector:
      k8s-app: kube-proxy
  serviceMonitor:
    interval: ""
    https: false
    metricRelabelings: []
    relabelings: []

The pod Labels are correctly matching:

ansible@master1:~$ kubectl get pods --show-labels -n=kube-system
NAME                              READY   STATUS    RESTARTS   AGE     LABELS
coredns-f9fd979d6-mk56n           1/1     Running   0          5d22h   k8s-app=kube-dns,pod-template-hash=f9fd979d6
coredns-f9fd979d6-z5tcm           1/1     Running   0          5d22h   k8s-app=kube-dns,pod-template-hash=f9fd979d6
etcd-master1                      1/1     Running   0          5d22h   component=etcd,tier=control-plane
etcd-master2                      1/1     Running   0          5d22h   component=etcd,tier=control-plane
etcd-master3                      1/1     Running   0          5d22h   component=etcd,tier=control-plane
kube-apiserver-master1            1/1     Running   0          5d22h   component=kube-apiserver,tier=control-plane
kube-apiserver-master2            1/1     Running   0          5d22h   component=kube-apiserver,tier=control-plane
kube-apiserver-master3            1/1     Running   0          5d22h   component=kube-apiserver,tier=control-plane
kube-controller-manager-master1   1/1     Running   1          5d22h   component=kube-controller-manager,tier=control-plane
kube-controller-manager-master2   1/1     Running   0          5d22h   component=kube-controller-manager,tier=control-plane
kube-controller-manager-master3   1/1     Running   0          5d22h   component=kube-controller-manager,tier=control-plane
kube-flannel-ds-2ldz8             1/1     Running   0          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-flannel-ds-6rgzp             1/1     Running   1          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-flannel-ds-8xv2f             1/1     Running   0          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-flannel-ds-b7mhz             1/1     Running   0          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-flannel-ds-lsvrg             1/1     Running   0          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-flannel-ds-xnlgl             1/1     Running   1          5d22h   app=flannel,controller-revision-hash=787f4dfddc,pod-template-generation=1,tier=node
kube-proxy-bkp7b                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-drzmv                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-gd9bk                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-kscx7                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-x6qwl                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-proxy-xjrtt                  1/1     Running   0          5d22h   controller-revision-hash=744c595cb,k8s-app=kube-proxy,pod-template-generation=1
kube-scheduler-master1            1/1     Running   1          5d22h   component=kube-scheduler,tier=control-plane
kube-scheduler-master2            1/1     Running   1          5d22h   component=kube-scheduler,tier=control-plane
kube-scheduler-master3            1/1     Running   0          5d22h   component=kube-scheduler,tier=control-plane
metrics-server-554df5cbcf-t5j5c   1/1     Running   0          4d23h   k8s-app=metrics-server,pod-template-hash=554df5cbcf
tiller-deploy-565984b594-22svl    1/1     Running   0          5d22h   app=helm,name=tiller,pod-template-hash=565984b594

I have no idea of why Prometheus want scrapping on the node IP instead of ClusterIP Services....
And for Kubelet Targets it's look like there is no service for kubelet ? Am I wrong ?

Thanks...

Configuration:

• CNI : Flannel
• KubeVersion : 1.19

[littlechicks]

For the Proxy I've solved it by updating configmap:

$ kubectl edit cm/kube-proxy -n kube-system

...
kind: KubeProxyConfiguration
metricsBindAddress: 0.0.0.0:10249
...

$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

[littlechicks]

Moreover for the making metrics available to prometheus i have to edit /etc/kubernetes/manifests/ files by changing binding address to 0.0.0.0 and comment the --port:0.

But that's no a good thing because Scheduler is now exposed outside cluster on non Secure port.

So again, how can I achieve to make the servicemonitor working on ClusterIP like others wokring targets.... ?

[abelal83]

any luck with this? We too have the same problem.

prom-operator/prometheus-operator-kubelet/0 (0/20 up)
prom-operator/prometheus-operator-kubelet/1 (0/20 up)
prom-operator/prometheus-operator-kubelet/2 (0/20 up)

we use AKS and disabled most of the other jobs as not fully supported but I can't work out whats needed to get kublet monitor up.

[littlechicks]

Not yet, I'm still facing the same problems

[abelal83]

I managed to fix this by enabling

kubelet:
  serviceMonitor:
    https: true # was false

I hope this helps someone else.

[esteban1983cl]

Hello. I have the same problem in AWS EKS 1.18 with the kubelet.serviceMonitor.https=true parameter.
Ideas ?

[dcs3spp]

I am also experiencing this, see issue