Kubeadm: kube-dns cannot connect to apiserver

[dl00]

What keywords did you search in kubeadm issues before filing this one?

Possibly related to #193, but as one pointed out, this issue probably deserves its own thread now.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

Currently attempting to do this with a single CentOS server for testing, hosted at my residence. There is an OpenVPN connection associated with the server.

• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

Linux <REDACTED> 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

When attaching a network manager to the network, kube-dns reports errors and fails to start:

E1020 15:29:15.988144       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
E1020 15:29:15.988200       1 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host
I1020 15:29:16.362664       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I1020 15:29:16.862707       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I1020 15:29:17.362954       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I1020 15:29:17.862703       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I1020 15:29:18.362673       1 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
F1020 15:29:18.862674       1 dns.go:168] Timeout waiting for initialization

This issue has been tested by me as occuring with both the flannel or kube-router network drivers. Canal does not seem to have this issue specifically, but it has other pod connection issues which are probably out of the scope of this issue. Long story short, I have not been able to get any of the network drivers to work.

What you expected to happen?

I expect that kube-dns will start up successfully as documented and for packets to be routed properly.

How to reproduce it (as minimally and precisely as possible)?

On my setup, all I have done is the following:

kubeadm init --apiserver-advertise-address=192.168.240.1 --pod-network-cidr=10.244.0.0/16

After waiting for pods to init and copying over the new keyfile I run one of these:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.0/Documentation/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

Anything else we need to know?

@murali-reddy and I spent a ton of time trying to figure this out. We determined:

• The issue does not seem to have to do with iptables firewall blocking anything
• When pinging from any pod to 10.96.0.1, this is the output:

$ kubectl exec -it -n kube-system kube-proxy-fvfc8 ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1): 56 data bytes
36 bytes from 96.120.60.109: Packet Filtered
36 bytes from 96.120.60.109: Packet Filtered
^C--- 10.96.0.1 ping statistics ---
17 packets transmitted, 0 packets received, 100% packet loss

The IP address shown is not associated with my network, it is a WWW IP address associated with my ISP.

[dl00]

Found this problem, turns out for CentOS, firewalld is completely incompatible with kubernetes. So disable and restart:

systemctl disable firewalld
systemctl reboot

[bamb00]

What do you mean firewalld is incompatible with kubernetes? Is there Kubernetes documentation mention kubernetes does not support enable firewalld?

Thanks.

[dl00]

@bamb00 yes, it specifically says on this page: https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/

[bithavoc]

I was having the same issue with ubuntu xenial (in vagrant), sudo ufw disable did the trick.

[sa-shukla]

master # ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1) 56(84) bytes of data.
^C
--- 10.96.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

Should this work?

[anjuls]

I was facing the same issue, i did following to allow all traffic.

iptables -P FORWARD ACCEPT
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT

[stefaneg]

Did all of the above. Strangely, flannel started to work, but kubedns is still refused to connect on 10.96.0.1

Nodes host OS is Ubuntu 16.04. ufw is disabled, have applied the changes above, rebooted the nodes and recreated the KubeDNS pods. Still no luck. Any bright ideas?

[sa-shukla]

What do the kube-dns logs report? Any errors that you see? Which services are failing?
Regards,Sagar

Sent from Yahoo Mail on Android

On Tue, May 29, 2018 at 11:22 PM, Guðlaugur Stefán [e-mail] wrote:
Did all of the above. Strangely, flannel started to work, but kubedns is still refused to connect on 10.96.0.1

Nodes host OS is Ubuntu 16.04. ufw is disabled, have applied the changes above, rebooted the nodes and recreated the KubeDNS pods. Still no luck. Any bright ideas?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[anjuls]

I had such issues in past and then I switched to calico with following done on all nodes in cluster. Post this you can restart kube-proxy pods, dns pods.

systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /etc/cni/net.d
rm -rf /run/flannel

iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT


ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X
ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT

iptables -L -v -n 
iptables -L -v -n -t nat
conntrack -L 
ipset list

lsmod|grep br_netfilter
modprobe br_netfilter
systemctl start docker
systemctl start kubelet

[rayburgemeestre]

Found this ticket twice now troubleshooting this error, I remember your previous snippet worked for me the last time, the one with the 3 iptables commands (a Month ago or so), but now they didn't, and this time your above snippet saved me, thanks!

It fixes the endless loop kubedns produces in my case:

I0725 15:14:56.447479       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0725 15:14:56.947408       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0725 15:14:57.447588       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...
I0725 15:14:57.947595       1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...