Kubeadm: kubadm init v1.6.1 fails -> api-server uses wrong interface

[Mikey032]

Is this a request for help? No

What keywords did you search in Kubernetes issues before filing this one? kubeadm init, api-server crash, 6443

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Kubernetes version (use kubectl version): v1.6.1 and v1.6.0

Environment:

• Cloud provider or hardware configuration: Local (native & virtualbox)
• OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS
• Kernel (e.g. uname -a): Linux ubuntu 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Others: NAT + host only adapter
• Install tools: kubeadm

What happened:
I am experimenting with the best way to install Kubernetes. After a few days of trying to get Kubeadm work, I still was getting stuck with the error:
[apiclient] Temporarily unable to list nodes (will retry)
After a bit of searching, I found the api-server going up and down with Docker ps -a. I went looking for the logs but I couldn't find a solution on the internet.
UPDATE -> I found out that the apiserver uses the wrong interface

OUTPUT of the log of the api-server

root@ubuntu:~# tail -n 500 /var/log/containers/kube-
kube-apiserver-ubuntu_kube-system_kube-apiserver-31989c0a06decf19ef99557658d1a07df8c5be12786e7029f3f549715e5b2452.log
kube-controller-manager-ubuntu_kube-system_kube-controller-manager-1c6bd5ecb13318145b031bef792b157458917a13432ba62c67ab48269622bf09.log
root@ubuntu:~# tail -n 500 /var/log/containers/kube-apiserver-ubuntu_kube-system_kube-apiserver-31989c0a06decf19ef99557658d1a07df8c5be12786e7029f3f549715e5b2452.log
{"log":"[restful] 2017/04/04 12:18:59 log.go:30: [restful/swagger] listing is available at https://192.168.57.111:6443/swaggerapi/\n","stream":"stderr","time":"2017-04-04T12:18:59.712716159Z"}
{"log":"[restful] 2017/04/04 12:18:59 log.go:30: [restful/swagger] https://192.168.57.111:6443/swaggerui/ is mapped to folder /swagger-ui/\n","stream":"stderr","time":"2017-04-04T12:18:59.71274958Z"}
{"log":"I0404 12:18:59.791426       1 serve.go:79] Serving securely on 0.0.0.0:6443\n","stream":"stderr","time":"2017-04-04T12:18:59.79177779Z"}
{"log":"E0404 12:19:02.698462       1 storage_rbac.go:140] unable to initialize clusterroles: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/clusterroles: dial tcp 10.0.2.150:6443: getsockopt: no route to host\n","stream":"stderr","time":"2017-04-04T12:19:02.703156447Z"}
{"log":"F0404 12:19:02.698519       1 controller.go:128] Unable to perform initial IP allocation check: unable to refresh the service IP block: Get https://localhost:6443/api/v1/services: dial tcp 10.0.2.150:6443: getsockopt: no route to host\n","stream":"stderr","time":"2017-04-04T12:19:02.703181402Z"}
root@ubuntu:~# nable to initialize clusterroles: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/clusterroles: dial tcp 10.0.2.150:6443: getsockopt: no route to host\n","stream":"stderr","time":"2017-04-04T12:19:02.703156447Z"}

Docker ps -a OUTPUT

docker logs api-server
unable to initialize clusterroles: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/clusterroles: dial tcp 10.0.2.150:6443: getsockopt: no route to host

What you expected to happen:
Kubernetes cluster to be up.

How to reproduce it (as minimally and precisely as possible):
follewed this guide:
https://kubernetes.io/docs/getting-started-guides/kubeadm/

Anything else we need to know:
I did a few tries with Kubadm reset, service kubelet start, kubeadm init
kube init command
kubeadm init --apiserver-advertise-address 192.168.57.111 --pod-network-cidr 10.244.0.0/16 --kubernetes-version="v1.6.1"

[sachinna]

I am also facing same issue on Centos 7.3 with kubernetes 1.6 and 1.6.1

[Mikey032]

The temporary fix is that you should use only 1 network interface and that inferface needs to be connected to the internet.

[luxas]

Generally, I think kubernetes assumes some things about networking interfaces that don't always play well with reality when using multiple NICs.

Maybe @thockin, @bboreham, @jbeda or @errordeveloper know more about what we could do to better support multiple NICs

[bboreham]

Per the kubeadm docs the flag is --api-advertise-address

[bboreham]

But see also https://github.com/kubernetes/kubeadm/issues/102, wherein Linux needs guidance to pick the right source address.

[Mikey032]

I have used the flag, but the apiserver still uses the wrong interface. Is there another way to tell the apiserver what interface to use?

[ajayaram1]

any workaround? @KubernetesIntership you mentioned abt temp fi with single interface what did you mean? can you elaborate?

[bboreham]

Can you post a transcript of it going wrong, including evidence that the correct flag was used.

[Mikey032]

@ajayaram1 well I was testing it with virtualbox, I have removed the host-only network and I port forwarded the nat-network, so I could reach the dashboard.

[Mikey032]

@bboreham I have normally put everything in my post. The command I have used is at anything else we should know and I have posted all the logs. If you need something, let me know.

[bboreham]

@KubernetesIntership I am asking for a full transcript because your original post describes a number of attempts and different options, and it is much easier to debug from one single run.
Do kubeadm reset then start from the top, with the correct 'advertise' option (it changed name between versions of kubeadm)

[sdurrheimer]

@bboreham I'm also facing the same issue. Would my transcript help you ?

[bboreham]

@sdurrheimer sure

[sdurrheimer]

Setup

• 1 physical server with ESXI 6.5
• 4 virtual machines with ubuntu 16.04
  
  
  
  • 1 master
  
  
  • 3 worker
  
  
• each VM has 3 network interfaces
  
  
  
  • docker bridge
  
  
  • public ip with external connectivity
  
  
  • private network for internal connectivity between the 4 VMs
  
  
• docker 1.12.6-0~ubuntu-xenial
• kubeadm/kubectl/kubelet 1.6.1-00

kubeadm init

root@kube-master-1:~# kubeadm init --apiserver-advertise-address 192.168.0.10 --kubernetes-version v1.6.1 --skip-preflight-checks --token 5d7679.61654b383aced888
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.1
[init] Using Authorization mode: RBAC
[preflight] Skipping pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [kube-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.10]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 18.021708 seconds
[apiclient] Waiting for at least one node to register
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
[apiclient] Temporarily unable to list nodes (will retry)
...

apiserver logs

root@kube-master-1:~# docker logs -f k8s_kube-apiserver_kube-apiserver-kube-master-1_kube-system_b3ec8ab8eec7b63e0dce596477317af4_1
[restful] 2017/04/13 09:27:18 log.go:30: [restful/swagger] listing is available at https://192.168.0.10:6443/swaggerapi/
[restful] 2017/04/13 09:27:18 log.go:30: [restful/swagger] https://192.168.0.10:6443/swaggerui/ is mapped to folder /swagger-ui/
I0413 09:27:18.643179       1 serve.go:79] Serving securely on 0.0.0.0:6443
I0413 09:27:19.655302       1 trace.go:61] Trace "List /api/v1/componentstatuses" (started 2017-04-13 09:27:19.153957206 +0000 UTC):
[8.523µs] [8.523µs] About to List from storage
[501.108873ms] [501.10035ms] Listing from storage done
[501.134242ms] [25.369µs] Self-linking done
[501.322968ms] [188.726µs] Writing http response done (3 items)
"List /api/v1/componentstatuses" [501.323291ms] [323ns] END
E0413 09:27:48.573836       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *rbac.Role: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/roles?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574021       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.LimitRange: Get https://localhost:6443/api/v1/limitranges?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574182       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.Secret: Get https://localhost:6443/api/v1/secrets?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574371       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.ResourceQuota: Get https://localhost:6443/api/v1/resourcequotas?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574519       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *rbac.ClusterRole: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/clusterroles?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574670       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.Namespace: Get https://localhost:6443/api/v1/namespaces?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.574895       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *rbac.ClusterRoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.575049       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *rbac.RoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1beta1/rolebindings?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.575292       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *api.ServiceAccount: Get https://localhost:6443/api/v1/serviceaccounts?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.575518       1 reflector.go:201] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: Failed to list *storage.StorageClass: Get https://localhost:6443/apis/storage.k8s.io/v1beta1/storageclasses?resourceVersion=0: dial tcp 195.154.241.115:6443: i/o timeout
W0413 09:27:48.643538       1 storage_extensions.go:127] third party resource sync failed: Get https://localhost:6443/apis/extensions/v1beta1/thirdpartyresources: dial tcp 195.154.241.115:6443: i/o timeout
E0413 09:27:48.643769       1 client_ca_hook.go:58] Post https://localhost:6443/api/v1/namespaces: dial tcp 195.154.241.115:6443: i/o timeout
F0413 09:27:48.644121       1 controller.go:128] Unable to perform initial IP allocation check: unable to refresh the service IP block: Get https://localhost:6443/api/v1/services: dial tcp 195.154.241.115:6443: i/o timeout

[cheinig]

I had the same problem in my environment.
The problem was, that "localhost" had not been resolved to the loopback address 127.0.0.1 by the nameserver, but to a different one. This was checked with "nslookup localhost".
In my case, the root cause was a wrong "search" entry in my /etc/resolv.conf file. This added the wrong domainname to my localhost name entry (eg. localhost.powersrv.de) which results in the wrong address.

Perhaps this will help some of you.

[sdurrheimer]

@cheinig Thank you, that was exactly the issue !

[renhuailin]

@cheinig Thank you, I had the same problem. and I installed the dnsmasq ,then add 127.0.0.1 to '/etc/resolv.conf', now it works.

$ sudo apt-get install dnsmasq

My '/etc/resolv.conf':

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
nameserver 8.8.8.8

[optlin]

@cheinig @renhuailin Thank you for the notes. I had exact same problem and resolved it after fixing /etc/resolv.conf file.

[vascofg]

Bump, this seems like a bug to me...
curl localhost -v shows it trying to connect to 127.0.0.1 but kubernetes fails to start because localhost.mylocaldomain.com is mapped to a machine on the network.
Maybe in the way that kubernetes opens sockets that's using DNS server directly, ignoring prority in /etc/nsswitch.conf

[surlypants]

encountered same issue.

applications should observe /etc/nsswitch.conf (and subsequently /etc/hosts). a broken upstream nameserver shouldn't bring an application to its knees.

[bboreham]

Since you are commenting on a closed issue, and the last few comments are all talking about resolving localhost which is a different subject to this issue, I suggest you open a new issue.

[luxas]

@surlypants Please comment on the issues that are relevant to what you're describing... there are a couple that touch DNS misconfigurations which makes Kubernetes fail. As an aside, I don't think Kubernetes is responsible for fixing invalid /etc/hosts and /etc/resolv.conf files.

[elaron]

@bboreham Can you help me with the kubelet confusing NIC interface problem? Thanks

I have two NIC interface on my VM.

2: enp0s3: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:19:83:14 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 74933sec preferred_lft 74933sec
inet6 fe80::a00:27ff:fe19:8314/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ec:09:b4 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.141/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:feec:9b4/64 scope link
valid_lft forever preferred_lft forever

Even I config the api-server’s ip in /etc/kubernetes/kubelet as:
KUBELET_API_SERVER=“--api-servers=http://${MASTER_ADDRESS}:8080”
And MASTER_ADDRESS=192.168.56.141

but, kubelet is still trying to visit api-server on another NIC interface, and report Error below:

11月 17 17:14:38 k8s-node1 kubelet[3375]: E1117 17:14:38.890137 3375 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-node1&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused