metrics-server : ServiceUnavailable

Created on 9 Oct 2020 · 13Comments · Source: kubernetes-sigs/metrics-server

Trying to enable __metrics-server__ on kubernetes clustrer (1.19), by running

$ kubectl create -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

To check if everything is alright :

kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
...
kube-system   kube-proxy-m26g7                               1/1     Running   1          5d6h
kube-system   kube-proxy-wdnbv                                1/1     Running   0          5d6h
kube-system   kube-scheduler-master                         1/1     Running   2          5d6h
kube-system   metrics-server-75f98fdbd5-xrlzt            1/1     Running   0          6m3s

Knowing that I :

Enabled beforehand the __aggregator-routing__ on kube-apiserver via
yaml <ul> <li>--enable-aggregator-routing=true<br />

Edited the metrics-server deployment resource :
shell k edit deploy -n kube-system metrics-server
```yaml
containers:
- args:
  - --cert-dir=/tmp
  - --secure-port=4443
  - --kubelet-insecure-tls
  - --kubelet-preferred-address-types=InternalIP

```

What happened:

So far so good, but when I try to run

$ kubectl top po -A
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

I've got same error.

What you expected to happen:

To have an output like this :

$ kubectl top pod --all-namespaces
NAMESPACE     NAME                                                    CPU(cores)   MEMORY(bytes)
kube-system   calico-kube-controllers-7b9dcdcc5-qg6zd          2m           6Mi
kube-system   calico-node-dr279                                             23m          22Mi
kube-system   calico-node-xtvfd                                               21m          22Mi
kube-system   coredns-5644d7b6d9-k7kts                                 2m           6Mi
kube-system   coredns-5644d7b6d9-rnr2v                                 3m           6Mi

Anything else we need to know?:

Environment:

Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.):
- GCE
- kubeadm
Container Network Setup (flannel, calico, etc.): calico
Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Metrics Server manifest : https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

spoiler for Metrics Server manifest:

Kubelet Systemd unit file:

spoiler for Kubelet config:

#### systemd service file
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

kubelet.conf

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ....
    server: https://kubernetesmaster:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:master
  name: system:node:master@kubernetes
current-context: system:node:master@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:master
  user:
    client-certificate: /var/lib/kubelet/pki/XX.pem
    client-key: /var/lib/kubelet/pki/XXX.pem

Metrics server logs:

spoiler for Metrics Server logs:

/kind bug

kinsupport

Source

devlifealways

Most helpful comment

Had exactly the same issue from a clean cluster on docker for desktop and doing

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

the args --kubelet-insecure-tls did fix it.

bklooste on 20 Jan 2021

👍2

All 13 comments

Please provide:

Metrics Server Logs
results from kubectl describe apiservice v1beta1.metrics.k8s.io

serathius on 31 Oct 2020

@serathius I'm running into a similar issue as well, the results from kubectl describe apiservice v1beta1.metrics.k8s.io are:

kubectl describe apiservice v1beta1.metrics.k8s.io Name: v1beta1.metrics.k8s.io Namespace: Labels: app.kubernetes.io/instance=metrics-server app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=metrics-server helm.sh/chart=metrics-server-4.3.0 k8s-app=metrics-server Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"v... meta.helm.sh/release-name: metrics-server meta.helm.sh/release-namespace: kube-system API Version: apiregistration.k8s.io/v1 Kind: APIService Metadata: Creation Timestamp: 2020-11-04T04:04:24Z Resource Version: 1200626 Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io UID: 5df6284e-4a8a-4372-89e6-ec0afa443df4 Spec: Group: metrics.k8s.io Group Priority Minimum: 100 Insecure Skip TLS Verify: true Service: Name: metrics-server Namespace: kube-system Port: 443 Version: v1beta1 Version Priority: 100 Status: Conditions: Last Transition Time: 2020-11-12T23:09:17Z Message: endpoints for service/metrics-server in "kube-system" have no addresses with port name "https" Reason: MissingEndpoints Status: False Type: Available Events: <none>

elijahomolo on 13 Nov 2020

Define service/metrics-server properly,

---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https

Maybe you miss .spec.selector or misconfigration about deployment.[]containers.[]ports.port name

DingGGu on 13 Nov 2020

I was able to resolve this by removing the unnecessary selectors:

Previously:

  selector:
    app.kubernetes.io/instance: metrics-server
    app.kubernetes.io/name: metrics-server
    k8s-app: metrics-server

Currently:

  selector:
    k8s-app: metrics-server

Is there any specific reason why this would cause the service issue?

elijahomolo on 22 Nov 2020

For functioning Metrics API you need a service with at least one pod/endpoint to handle traffic. With wrong selectors (like above) metrics-server pod will not be considered to serve the API. Selectors need to match labels on pod.

What's wierd for me is that labels app.kubernetes.io was never part of MS manifests, so the bug could not come from incompatibility with previous version, but from merging with different distribution (kubectl apply merges fields).

Please make sure that you install metrics server from clean status. You can do that by first running delete, like:

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

serathius on 22 Nov 2020

👍1

Hi, I'm also running into a similar issue here, and the result of kubectl describe apiservice v1beta1.metrics.k8s.io is the same with @elijahomolo, and I'm using the kubeadm1.19.4 and metrics-server v0.4.1
The results from kubectl describe pod metrics-server-8545bf4d7-ssczf -n kube-system are:

  Normal   Killing    59s                 kubelet            Container metrics-server failed liveness probe, will be restarted
  Normal   Created    58s (x2 over 117s)  kubelet            Created container metrics-server
  Normal   Started    58s (x2 over 117s)  kubelet            Started container metrics-server
  Warning  Unhealthy  19s (x5 over 99s)   kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  16s (x6 over 116s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

Anyone can have a look? I don't know how to deal with the probe failed issue.

sawyerwu on 11 Dec 2020

Failing probes can be caused by multiple different factors, for example in case of @elijahomolo it was conflict with previous installation. Please let's focus this issue on resolving problem of original author @devlifealways.

@sawyerwu To understand cause for your case we need much more debug information. Please file a separate bug and fill the bug template there. It should give you list of things that are needed to debug Metrics Server and how to get them.

If original doesn't provide previously requested information for debugging:

Metrics Server Logs
results from kubectl describe apiservice v1beta1.metrics.k8s.io

I will be inclined to close this issue, to avoid it drifting from original problem more.

serathius on 11 Dec 2020

Hi, I'm also running into a similar issue here, and the result of kubectl describe apiservice v1beta1.metrics.k8s.io is the same with @elijahomolo, and I'm using the kubeadm1.19.4 and metrics-server v0.4.1
The results from kubectl describe pod metrics-server-8545bf4d7-ssczf -n kube-system are:
  Normal   Killing    59s                 kubelet            Container metrics-server failed liveness probe, will be restarted
  Normal   Created    58s (x2 over 117s)  kubelet            Created container metrics-server
  Normal   Started    58s (x2 over 117s)  kubelet            Started container metrics-server
  Warning  Unhealthy  19s (x5 over 99s)   kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  16s (x6 over 116s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
Anyone can have a look? I don't know how to deal with the probe failed issue.

Same issue here, here is the result of kubectl describe apiservice v1beta1.metrics.k8s.io:

Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       k8s-app=metrics-server
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"v...
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2020-12-09T07:57:32Z
  Resource Version:    238149
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 227739eb-7887-4146-b75b-b963592d2cc4
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-12-09T07:57:32Z
    Message:               endpoints for service/metrics-server in "kube-system" have no addresses with port name "https"
    Reason:                MissingEndpoints
    Status:                False
    Type:                  Available
Events:                    <none>

@serathius Something wrong with the port name, but I do not know how to modify the yaml file, need your help.

KeithTt on 11 Dec 2020

@KeithTt I solved this problem by adding - --kubelet-insecure-tls in the args, you can have a try.

sawyerwu on 11 Dec 2020

👍1

@KeithTt I solved this problem by adding - --kubelet-insecure-tls in the args, you can have a try.

It works, thanx a lot.

But I guess it has not been resolved completely...

KeithTt on 11 Dec 2020

Had exactly the same issue from a clean cluster on docker for desktop and doing

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

the args --kubelet-insecure-tls did fix it.

bklooste on 20 Jan 2021

👍2

I ran into the same issue and the solution from @bklooste did work for me:

delete the metrics-server deployment
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

chevdor on 15 Feb 2021

Closing issue as original author didn't respond and issue seems to be resolved in other cases. If problem was not resolved, please create separate issue and provide all nesesery information for debugging.

serathius on 21 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings