K3s: Network break down on host after cni0 or veth up

Created on 7 Mar 2019  路  8Comments  路  Source: k3s-io/k3s

Issue

after a container running, the network can't be used on host, likes "apt-get update"

$ ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:95:0b:64:fa  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:95ff:fe0b:64fa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:939 errors:0 dropped:0 overruns:0 frame:0
          TX packets:576 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:66707 (66.7 KB)  TX bytes:72699 (72.6 KB)

enxb827ebe8728f Link encap:Ethernet  HWaddr b8:27:eb:e8:72:8f  
          inet addr:43.4.1.45  Bcast:43.4.3.255  Mask:255.255.252.0
          inet6 addr: fe80::ba27:ebff:fee8:728f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST DYNAMIC  MTU:1500  Metric:1
          RX packets:561899 errors:9 dropped:234029 overruns:0 frame:8
          TX packets:85095 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:311195553 (311.1 MB)  TX bytes:10504846 (10.5 MB)

flannel.1 Link encap:Ethernet  HWaddr 12:f6:fa:1c:fb:54  
          inet addr:10.42.1.0  Bcast:10.42.1.0  Mask:255.255.255.255
          inet6 addr: fe80::10f6:faff:fe1c:fb54/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:99 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:30226 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30226 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:20255374 (20.2 MB)  TX bytes:20255374 (20.2 MB)

vetha11889e Link encap:Ethernet  HWaddr 66:cc:ab:fb:8a:eb  
          inet addr:43.4.1.45  Bcast:43.4.3.255  Mask:255.255.252.0
          inet6 addr: fe80::64cc:abff:fefb:8aeb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23 errors:0 dropped:0 overruns:0 frame:0
          TX packets:271 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1875 (1.8 KB)  TX bytes:29094 (29.0 KB)

vethe888a78 Link encap:Ethernet  HWaddr 1a:17:02:41:32:2e  
          inet addr:43.4.1.45  Bcast:43.4.3.255  Mask:255.255.252.0
          inet6 addr: fe80::1817:2ff:fe41:322e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST DYNAMIC  MTU:1500  Metric:1
          RX packets:23 errors:0 dropped:0 overruns:0 frame:0
          TX packets:170 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1907 (1.9 KB)  TX bytes:17563 (17.5 KB)

analysis

after "ip link delete vetha11889e vethe888a78", network work well on host, likes "apt-get update"
it seems that container get a ip(43.4.1.45) same as host(43.4.1.45), which cause the network break down on host.

Can you give me some advises about how to resolve this issue?

thanks

help wanted
Source
picture yechenglin-dev

All 8 comments

When the system is in a broken state, can you please give the output of ip route and iptables-save ?

ibuildthecloud picture ibuildthecloud on 7 Mar 2019

thanks for your feedback

the issue machine that network is in a broken state

ip route:

$ ip route
default dev enxb827ebe8728f  scope link 
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink 
10.42.1.0/24 dev cni0  proto kernel  scope link  src 10.42.1.1 
43.4.0.0/22 dev enxb827ebe8728f  proto kernel  scope link  src 43.4.1.45 
43.4.0.0/22 dev veth5254848b  proto kernel  scope link  src 43.4.1.45 
43.4.0.0/22 dev veth8124a4c5  proto kernel  scope link  src 43.4.1.45 
137.153.66.28 dev veth8124a4c5  scope link 
137.153.66.28 dev veth5254848b  scope link 
137.153.66.28 dev enxb827ebe8728f  scope link 
137.153.112.2 dev veth8124a4c5  scope link 
137.153.112.2 dev veth5254848b  scope link 
137.153.112.2 dev enxb827ebe8728f  scope link 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown

iptables-save :

$ iptables-save
# Generated by iptables-save v1.4.21 on Fri Mar  8 10:52:24 2019
*nat
:PREROUTING ACCEPT [33:3865]
:INPUT ACCEPT [22:2590]
:OUTPUT ACCEPT [1:77]
:POSTROUTING ACCEPT [1:77]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-QUSWRCCKNVSH5RXX - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
-A POSTROUTING -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-QUSWRCCKNVSH5RXX -s 127.0.0.1/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-QUSWRCCKNVSH5RXX -p tcp -m tcp -j DNAT --to-destination 127.0.0.1:6445
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-SEP-QUSWRCCKNVSH5RXX
COMMIT
# Completed on Fri Mar  8 10:52:24 2019
# Generated by iptables-save v1.4.21 on Fri Mar  8 10:52:24 2019
*filter
:INPUT ACCEPT [313:75166]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [183:53840]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics has no endpoints" -m tcp --dport 9153 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Fri Mar  8 10:52:24 2019

another machine that k3s work well

ip route:

$ ip route
default via 192.168.0.1 dev enxb827eb4a2651 
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink 
10.42.1.0/24 dev cni0  proto kernel  scope link  src 10.42.1.1 
43.82.217.60 via 192.168.0.1 dev enxb827eb4a2651 
43.82.217.61 via 192.168.0.1 dev enxb827eb4a2651 
192.168.0.0/24 dev enxb827eb4a2651  proto kernel  scope link  src 192.168.0.47 
192.168.0.1 dev enxb827eb4a2651  scope link

iptables-save:

$ iptables-save
# Generated by iptables-save v1.4.21 on Fri Mar  8 09:44:11 2019
*filter
:INPUT ACCEPT [135:57135]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [90:32167]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.42.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.43.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.43.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics has no endpoints" -m tcp --dport 9153 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Fri Mar  8 09:44:11 2019
# Generated by iptables-save v1.4.21 on Fri Mar  8 09:44:11 2019
*nat
:PREROUTING ACCEPT [5:958]
:INPUT ACCEPT [5:958]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-QUSWRCCKNVSH5RXX - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
-A POSTROUTING -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN
-A POSTROUTING ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-QUSWRCCKNVSH5RXX -s 127.0.0.1/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-QUSWRCCKNVSH5RXX -p tcp -m tcp -j DNAT --to-destination 127.0.0.1:6445
-A KUBE-SERVICES ! -s 10.42.0.0/16 -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.43.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -j KUBE-SEP-QUSWRCCKNVSH5RXX
COMMIT
# Completed on Fri Mar  8 09:44:11 2019
yechenglin-dev picture yechenglin-dev on 8 Mar 2019

I am facing the same sort of issue.

As soon as I start the k3s server I loose the network. Same results on both my raspberry pi 3 with raspian stretch light and intel pc with Ubuntu 16.04.

I'm thinking it might be related to some network forwarding configuration needed maybe?

I see in the logs that there is a difference in iptables in the working and broken nodes:
:FORWARD DROP [0:0]
and
:FORWARD ACCEPT [0:0]

My networking skills are poor and rusty, but I will try and spend a few hours more on solving it this weekend...

jonasfugedi picture jonasfugedi on 24 May 2019

@jonasfugedi on my host PC, "connman" used to manager network by default, i just try replace "connman" with "network-manager", network works well. you can try "apt-get install network-manager" or "apt-get install connman", then reboot your system.

yechenglin-dev picture yechenglin-dev on 27 May 2019

On my cluster at home networking is fine. On my cluster in the office networking always fails and I do not know why... so not able to resolve this yet, but will report back when I get it sorted.

Maybe it is because I used 10.42.0.xxx addresses for my static ip's... At home it is 10.11.12.xxx ...

jonasfugedi picture jonasfugedi on 28 May 2019

Ok, I have reconfigured to use another ip range (10.11.10.1/24) on my raspberry cluster and now k3s works. So, collision with the 10.42.0.1 range seems to be my root cause of the networking problems.

jonasfugedi picture jonasfugedi on 30 May 2019
👍1

Ok, I have reconfigured to use another ip range (10.11.10.1/24) on my raspberry cluster and now k3s works. So, collision with the 10.42.0.1 range seems to be my root cause of the networking problems.

~Thanks! This fixed my issues as well: everything would start to crawl on my raspberry Pi after a day or so, and the network would drop as well.~

I also had to make sure the hostname is resolvable (ie. put myhostname into /etc/hosts), because otherwise k3s would lookup the hostname from DNS server provided by DHCP, and that server is my ISP's, so obviously it'll fail to resolve myhostname.

My home network is 192.168.1.0/24, and I have no idea what is conflicting with 10.42.0.1/24. I do have Docker installed if that matters, but other than that - no idea. My Raspberry Pi is running Arch Linux ARM.

EDIT: nevermind, my wlan0 was once again without an IP address. But at least the system wasn't crawling.
EDIT2: ~I think I found a solution to my issue. It was never k3s, k3s only increased i/o load on my system and thus exhibiting the symptoms.~
EDIT3: Still not solved, here's what's in the systemd-networkd log after it has failed:

Jul 26 16:39:16 raspberry systemd-networkd[229]: wlan0: DHCP lease lost
Jul 26 16:39:33 raspberry systemd-networkd[229]: wlan0: DHCPv4 address 192.168.1.124/24 via 192.168.1.1
Jul 26 16:40:22 raspberry systemd-networkd[229]: wlan0: Could not drop route: Connection timed out
Jul 26 16:41:01 raspberry systemd-networkd[229]: wlan0: Could not drop route: Connection timed out
Jul 26 16:41:30 raspberry systemd-networkd[229]: wlan0: Could not drop address: Connection timed out
Jul 26 16:41:57 raspberry systemd-networkd[229]: wlan0: Could not set DHCPv4 address: Connection timed out
Jul 26 16:42:05 raspberry systemd-networkd[229]: wlan0: Failed
Jul 26 16:45:14 raspberry systemd-networkd[229]: wlan0: Could not set DHCPv4 route: Connection timed out
Jul 26 16:45:42 raspberry systemd-networkd[229]: wlan0: Could not set DHCPv4 route: Connection timed out
ljani picture ljani on 22 Jun 2019
👍1

Closing due to age.

brandond picture brandond on 5 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Moep90 picture Moep90  路  3Comments
VictorRobellini picture VictorRobellini  路  3Comments
e-nikolov picture e-nikolov  路  3Comments
giminni picture giminni  路  3Comments
davidnuzik picture davidnuzik  路  3Comments