Metrics-server: FailedDiscoveryCheck Get https://<metrics-server-ip>/apis/metrics.k8s.io/v1beta1: Forbidden

[floryut]

What happened:

> kubectl top node
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

Because of this

> kubectl get apiservice
v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   33m

There is also this error

> kubectl api-resources
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

I've tried --enable-aggregator-routing=true on apiserver, hostNetwork: true on metricsServer manifest, I'm even starting the pod with --kubelet-insecure-tls
but nothing change my apiservice (and so my api-server log) still display

failing or missing response from https://10.254.0.226:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.0.226:443/apis/metrics.k8s.io/v1beta1: Forbidden

I've even allow unauth access to be sure RBAC was not the cause...

$ kubectl get --raw /api/v1/namespaces/kube-system/services/https:metrics-server:443/proxy
{
  "paths": [
    "/apis",
    "/apis/metrics.k8s.io",
    "/apis/metrics.k8s.io/v1beta1",
    "/healthz",
    "/healthz/healthz",
    "/healthz/log",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/metrics",
    "/openapi/v2",
    "/version"
  ]
}

My startup commands are bellow, if needed I can post the entire deploy.yml

      - command:
        - /metrics-server
        - --logtostderr
        - --cert-dir=/tmp
        - --secure-port=8443
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-insecure-tls
        - --metric-resolution=90s

I really don't know what to check anymore and what to tweak...

What you expected to happen:
Metrics displayed

Anything else we need to know?:
I don't know

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): Kubespray
• Container Network Setup (flannel, calico, etc.): Flannel
• Kubernetes version (use kubectl version): 1.17.6
• Metrics Server manifest: 0.3.6
• Kubelet config:
• Metrics Server logs:

I0615 14:25:37.490637       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0615 14:26:00.593741       1 secure_serving.go:116] Serving securely on [::]:8443
I0615 14:26:00.788617       1 log.go:172] http: TLS handshake error from 172.16.31.3:45588: EOF
I0615 14:26:00.889390       1 log.go:172] http: TLS handshake error from 172.16.31.3:45576: EOF
I0615 14:26:01.188762       1 log.go:172] http: TLS handshake error from 172.16.31.3:45612: EOF

/king bug

[serathius]

I see two potential problems:

Network/TLS problem when connecting to Metrics Server:

I0615 14:26:00.788617       1 log.go:172] http: TLS handshake error from 172.16.31.3:45588: EOF

RBAC permission problem

Get https://10.254.0.226:443/apis/metrics.k8s.io/v1beta1: Forbidden

I would start from looking into the first one. I would look into directly connecting to Metrics Server without apiserver proxy.

[jensenity]

same issue here as well.

Followed all the guides and tutorials on AWS EKS metric server + Kubernetes Dashboard + metric server deployment and i'm still getting this kind of error kubectl get --raw /apis/metrics.k8s.io/v1beta1 Error from server (ServiceUnavailable): the server is currently unable to handle the request

[ravigude]

I am having the same issue as well. AWS EKS metric server + Kubernetes Dashboard + metric server + Cilium.
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

[ravigude]

The following worked. Let us know if it helped for you as well. EKS + cilium + metrics-server.
helm install metrics-server stable/metrics-server --set hostNetwork.enabled=true

[serathius]

/kind support

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[serathius]

Closing due to lack of response from author.

[evgf]

The EKS clusters using the CNI networking may experience this issue when the metrics server pod runs in the secondary network. As @ravigude pointed out, you can enable host networking for the metrics server that always brings it to the main network. If you already have an installation (for example deployed without Helm based on the AWS docs), you can simply edit the metrics server deployment kubectl edit deployment metrics-server -n kube-system and set the new pod property hostNetwork: true.