Metrics-server: Failed to get kubernetes address: No kubernetes source found.

[Aleks3050]

Hi!
I try to deploy metrics-server to my kubernetes cluster(bare metal):
[root@node1 metrics-server]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[root@node1 metrics-server]# kubectl get pods --namespace=kube-system --selector=k8s-app=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-6cf87b9c8d-nvm9j 0/1 Error 8 16m
[root@node1 metrics-server]# kubectl logs metrics-server-6cf87b9c8d-nvm9j -n kube-system
I0814 08:12:57.748260 1 heapster.go:71] /metrics-server
I0814 08:12:57.748292 1 heapster.go:72] Metrics Server version v0.2.1
F0814 08:12:57.748300 1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

[root@node1 metrics-server]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
creationTimestamp: 2018-08-14T07:40:59Z
name: v1beta1.metrics.k8s.io
resourceVersion: "13893882"
selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io
uid: 63107df3-9f95-11e8-a53f-00259047efc9
spec:
caBundle: null
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
status:
conditions:

• lastTransitionTime: 2018-08-14T07:40:59Z
  message: endpoints for service/metrics-server in "kube-system" have no addresses
  reason: MissingEndpoints
  status: "False"
  type: Available

Please helm with this issue!
Thank you!

[henley-regatta]

EDIT: This issue and the solution below is a duplicate of #97

When I pulled the project yesterday, I got an updated deploy/1.8+/metrics-server-deployment.yaml which is missing a command: stanza for the container: definition. I was able to fix by deleting the deployment, editing the file so the container definition reads:

containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

..then re-deploying. I suspect change a823af8 ("Refactor Makefile and update Dockerfile") is the cause, but I can't explain why (as I'd pulled the project onto another cluster after that change and had it work OK).

[Aleks3050]

Ok! work fine after this changes but please tell me:
E0814 14:46:05.004234 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.85.94:10255): Get http://89.108.x.x:10255/stats/summary/: dial tcp 89.108.x.x:10255: getsockopt: connection refused

my kubelet on port 10250.
kubelet 1477 root 19u IPv4 26755 0t0 TCP 89.108.x,x:10250 (LISTEN)
how to change port for metrics-server?

maybe documentation? could not find

[henley-regatta]

Aha, there's been a recent change and the read-only port used for metric gathering is disabled by default now. The quick hack to fix this is to add:

readOnlyPort: 10255

...to the bottom of /var/lib/kubelet/config.yaml and re-start kubelet (systemctl stop kubelet; sleep 10; systemctl start kubelet).

[DirectXMan12]

please don't do that. You should be using the authenticated port when possible. There's a reason the unauthenticated (read-only) port was disabled.

[henley-regatta]

@DirectXMan12 Instructions for configuring metrics-server to do so would be _greatly_ appreciated, because I looked but I did not find.

Nevermind. I see there's an example in #77 from @nasuku -

i.e
$ kubectl -n kube-system edit deploy metrics-server

look in section spec/template/spec/command for something like:
- --source=kubernetes
and replace that line with
- --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

[DirectXMan12]

yep. That'll be the default (and only) option in the next released version (master today). Speaking of which, @kawych can we get a beta release pushed so people can try out the new version more?

[Aleks3050]

hi!
@henley-regatta thank you for your answer.
I tried to install - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true

but see in logs:
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

service account has been created:
[root@node1 metrics-server]# kubectl --namespace=kube-system describe serviceaccount metrics-server
Name: metrics-server
Namespace: kube-system
Labels:
Annotations:
Image pull secrets:
Mountable secrets: metrics-server-token-r2snk
Tokens: metrics-server-token-r2snk
Events:

[Aleks3050]

Hi!
Any updates?
Thank you!

[henley-regatta]

Sorry, can't help you. At a guess - and it is only a guess - you might need to re-run metrics-server/deploy/1.8+/auth-reader.yaml to (re-)create the role?

[Aleks3050]

yes. i recreate role, but ... no effect.

[michelgokan]

I have the same issue.

[cristianocasella]

same troubles

[danilorsilva]

Same problem here.

```
1 heapster.go:71] /metrics-server
1 heapster.go:72] Metrics Server version v0.2.1
1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

[nikhilkhanna]

I have the same issue as well, seems that deploy/1.8+/ is broken

[s-irvine]

FYI anyone getting:

heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

@unclok found a solution that seems to be working for a lot of people over in: https://github.com/kubernetes-incubator/metrics-server/issues/97#issuecomment-412394545

[Aleks3050]

Hi!
At this moment error in pod metrics-server
E0814 19:14:05.017995 1 summary.go:97] error while getting metrics summary from Kubelet node1(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.018016 1 summary.go:97] error while getting metrics summary from Kubelet node2(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0814 19:14:05.027822 1 summary.go:97] error while getting metrics summary from Kubelet node3(89.108.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

[ggulati2]

I am also getting unauthorized 401 error

[Aleks3050]

Hi!
I pull code and recreate metrics-server but error is same:
I0830 12:22:30.541833 1 serve.go:85] Serving securely on 0.0.0.0:443
E0830 12:23:05.017203 1 summary.go:97] error while getting metrics summary from Kubelet node3(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.018103 1 summary.go:97] error while getting metrics summary from Kubelet node1(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"
E0830 12:23:05.023943 1 summary.go:97] error while getting metrics summary from Kubelet node2(x.x.x.x:10250): request failed - "401 Unauthorized", response: "Unauthorized"

[iakko]

I don't know if it's the same for minikube, but I've just installed it and cloned this repo.

Running the creation of Metrics Server, I got this error:

I0830 14:55:46.240497       1 heapster.go:71] /metrics-server
I0830 14:55:46.240548       1 heapster.go:72] Metrics Server version v0.2.1
F0830 14:55:46.240555       1 heapster.go:79] Failed to get kubernetes address: No kubernetes source found.

Any hint?

[ggulati2]

Has anyone found a solution to fix it?

[mabushey]

I wish someone would fix this

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[serathius]

Can you try to reproduce it on newer version of metrics-server v0.3+?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.