Kubespray: Error while getting metrics summary from Kubelet, after install heapster

[johnzheng1975]

Error while getting metrics summary from Kubelet, after install heapster

Here is heapster log:

I0329 02:40:09.751756 1 heapster.go:72] /heapster --source=kubernetes.summary_api:''
I0329 02:40:09.751809 1 heapster.go:73] Heapster version v1.4.0
I0329 02:40:09.752053 1 configs.go:61] Using Kubernetes client with master "https://10.233.0.1:443" and version v1
I0329 02:40:09.752078 1 configs.go:62] Using kubelet port 10255
I0329 02:40:09.752788 1 heapster.go:196] Starting with Metric Sink
I0329 02:40:10.150100 1 heapster.go:106] Starting heapster on port 8082
E0329 02:41:05.049166 1 summary.go:97] error while getting metrics summary from Kubelet kubernetes-devtest-c-worker1(10.250.87.195:10255): Get http://10.250.87.195:10255/stats/summary/: dial tcp 10.250.87.195:10255: getsockopt: connection refused
E0329 02:41:05.049166 1 summary.go:97] error while getting metrics summary from Kubelet kubernetes-devtest-c-master2(10.250.69.31:10255): Get http://10.250.69.31:10255/stats/summary/: dial tcp 10.250.69.31:10255: getsockopt: connection refused

This issue was not raised on v2.4.0 release, git version: f7d5256

[woopstar]

This is due to the fact that the kublet read only port was disabled by default.

You can enable the read-only port in all.yml with: kube_read_only_port: 10255

The only other way I have found to work yet, is to update the deployment of heapster and set the source to:

--source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true

But then you need to make sure kubelet_authentication_token_webhook is enabled too

[Data-drone]

@woopstar how do I enable kubelet_authentication_token_webhook?

[woopstar]

Add kubelet_authentication_token_webhook=true to k8s-cluster.yml

[arslanbekov]

How to solve it is possible to use a variable kubelet_custom_flags: ["--authentication-token-webhook"] in k8s-cluster.yaml

@woopstar this yaml :) use correctly:
kubelet_authentication_token_webhook: true

[botzill]

Hi.

I did enable this option but I still have the same issue

E0517 12:16:05.013890       1 summary.go:97] error while getting metrics summary from Kubelet 
node2(x.x.x.x:10255): Get http://x.x.x.x:10255/stats/summary/: dial tcp x.x.x.x:10255: getsockopt:
 connection refused

Any ideas on what could be wrong?

[johnzheng1975]

@woopstar, I enabled kube_read_only_port: 10255, everything works for me.

However, I did not enable kubelet_authentication_token_webhook, kube_api_anonymous_auth is still true
Are they necessary to be changed? will it bring security issue if I did not enable it?
I heard there is a serious security concern if I enable kubelet 10255 port.

[johnzheng1975]

Close after enable kube_read_only_port: 10255

[DeamonMV]

Can some one write how to apply those changes

Add kubelet_authentication_token_webhook=true to k8s-cluster.yml

what kind of ansible command i should run?

[arslanbekov]

@DeamonMV you can upgrade cluster or install new.

ansible-playbook -i inventory/sample/hosts.ini -e kubelet_authentication_token_webhook='true' cluster.yml --tags kubelet

[DeamonMV]

@Arslanbekov thanks)
i have tried but tags kubelet do not work properly
without tags work as expected