Flannel: cant connect to pods from external k8s nodes , k8s nodes can connect only to local pods, load balancing doesnt work properly
Your Environment
3 servers with CentOS 7.3.1611 (Core) / 3.10.0-514.el7.x86_64
flannel - v0.8.0
kubernetes - v 1.7.5
3 node cluster with 1 master (cent501) and 2 workers (cent502, cent503)
[root@cent501 ~]# kubectl get nodes
NAME STATUS AGE VERSION
cent501 Ready 39m v1.7.5
cent502 Ready 24m v1.7.5
cent503 Ready 14m v1.7.5
Expected Behavior
1.k8s nodes should serve my service (type=nodePort) on servicePort on each k8s node
2.pods should communicate with others, nodes should communicate with pods
Current Behavior
run simple deployment based on 6 replicas of nginx:
[root@cent501 ~]# kubectl run deployment01 --image=nginx --replicas=6 --port=80
deployment "deployment01" created
my pods:
[root@cent501 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
deployment01-554722758-7lfdg 1/1 Running 0 48s 10.244.1.29 cent502
deployment01-554722758-9gvp5 1/1 Running 0 48s 10.244.2.13 cent503
deployment01-554722758-hdgw2 1/1 Running 0 48s 10.244.0.10 cent501
deployment01-554722758-p5hzs 1/1 Running 0 48s 10.244.2.12 cent503
deployment01-554722758-sxqr2 1/1 Running 0 48s 10.244.2.10 cent503
deployment01-554722758-vxp7j 1/1 Running 0 48s 10.244.2.11 cent503
expose deployment as a service with type "NodePort"
[root@cent501 ~]# kubectl expose deployment/deployment01 --type="NodePort" --port 80
service "deployment01" exposed
my services and my endpoints (nginx service works on 31326):
[root@cent501 ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deployment01 10.110.20.86 <nodes> 80:31326/TCP 4s
kubernetes 10.96.0.1 <none> 443/TCP 44m
[root@cent501 ~]# kubectl get ep
NAME ENDPOINTS AGE
deployment01 10.244.0.10:80,10.244.1.29:80,10.244.2.10:80 + 3 more... 6s
kubernetes 192.168.43.148:6443 44m
[root@cent501 tmp]# kubectl describe svc deployment01
Name: deployment01
Namespace: default
Labels: run=deployment01
Annotations: <none>
Selector: run=deployment01
Type: NodePort
IP: 10.110.20.86
Port: <unset> 80/TCP
NodePort: <unset> 31326/TCP
Endpoints: 10.244.0.10:80,10.244.1.29:80,10.244.2.10:80 + 3 more...
Session Affinity: None
Events: <none>
my flannel subnet.env files on 3 nodes:
# ssh cent501 cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
# ssh cent502 cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
# ssh cent503 cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.2.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
I can't curl (on service port = 31326) my hardware-nodes, load balancing doesnt work properly
I can only curl and ping pods from nodes where these pods are running , no way to communicate with pods from other host
Once again my pods:
[root@cent501 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
deployment01-554722758-7lfdg 1/1 Running 0 14m 10.244.1.29 cent502
deployment01-554722758-9gvp5 1/1 Running 0 14m 10.244.2.13 cent503
deployment01-554722758-hdgw2 1/1 Running 0 14m 10.244.0.10 cent501
deployment01-554722758-p5hzs 1/1 Running 0 14m 10.244.2.12 cent503
deployment01-554722758-sxqr2 1/1 Running 0 14m 10.244.2.10 cent503
deployment01-554722758-vxp7j 1/1 Running 0 14m 10.244.2.11 cent503
on node cent501 I can only ping these one pod with 10.244.0.10
on node cent502 I can only ping these one pod with 10.244.1.29
on node cent503 I have 4 pods so I can ping them but only them:
10.244.2.13
10.244.2.12
10.244.2.10
10.244.2.11
with curling 80 the same effect
my routing tables:
# ssh cent501 ip route show
default via 192.168.43.1 dev enp0s3 proto static metric 100
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.0.0/16 dev flannel.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.43.0/24 dev enp0s3 proto kernel scope link src 192.168.43.148 metric 100
# ssh cent502 ip route show
default via 192.168.43.1 dev enp0s3 proto static metric 100
10.244.0.0/16 dev flannel.1
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.43.0/24 dev enp0s3 proto kernel scope link src 192.168.43.116 metric 100
# ssh cent503 ip route show
default via 192.168.43.1 dev enp0s3 proto static metric 100
10.244.0.0/16 dev flannel.1
10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.43.0/24 dev enp0s3 proto kernel scope link src 192.168.43.163 metric 100
slawekgh
All 17 comments
type=vxlan
[root@cent501 ~]# cat /tmp/kube-flannel.yml | grep -i type
"type": "flannel",
"Type": "vxlan"
I installed k8s via kubeadm and after that apply 2 flannel yml files:
kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f /tmp/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
the only difference between my local kube-flannel.yml and "official" file:
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
the only diff is:
< command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
---
> command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=enp0s3" ]
I made this change according to:
https://github.com/coreos/flannel/issues/535
without or without these small change I still have problems
slawekgh
on 20 Sep 2017
I also tested solution from other issues:
ip route add 10.244.0.0/16 dev enp0s3
with no success :-(
slawekgh
on 20 Sep 2017
Finally I found important checkpoint during installation:
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
if it is not ok (empty output) my problem still persists, if its ok (output with 10.244.1.0/24) there is no problem any more - and I have correct inter-node communication between pods
slawekgh
on 23 Sep 2017
and 2nd milestone - after reboot of kubernetes slave node we must execute
iptables -P FORWARD ACCEPT
I'm leaving this open to track the iptables -P FORWARD ACCEPT part
tomdee
on 4 Nov 2017
news? @slawekgh
acomisario
on 10 Nov 2017
still (sometimes) have to use "iptables -P FORWARD ACCEPT"
imho it's docker oriented problem , it is not a bug in k8s/Flannel
we can close the issue or leave it open for future users
slawekgh
on 10 Nov 2017
I've run iptables -P FORWARD ACCEPT on all nodes and pods still can't communicate across hosts. One of the effects of this is that I can't get cluster DNS working (they can't reach the dns pods)
# Generated by iptables-save v1.6.0 on Thu Nov 16 02:50:17 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-IT2ZTR26TO4XFPTO - [0:0]
:KUBE-SEP-S5EQFLAXMXOTYOQY - [0:0]
:KUBE-SEP-YIL6JZP7A3QYXJU2 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-IT2ZTR26TO4XFPTO -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-IT2ZTR26TO4XFPTO -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.244.0.2:53
-A KUBE-SEP-S5EQFLAXMXOTYOQY -s 10.0.1.2/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-S5EQFLAXMXOTYOQY -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-S5EQFLAXMXOTYOQY --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.0.1.2:6443
-A KUBE-SEP-YIL6JZP7A3QYXJU2 -s 10.244.0.2/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-YIL6JZP7A3QYXJU2 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.244.0.2:53
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-IT2ZTR26TO4XFPTO
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-S5EQFLAXMXOTYOQY --mask 255.255.255.255 --rsource -j KUBE-SEP-S5EQFLAXMXOTYOQY
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-S5EQFLAXMXOTYOQY
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-YIL6JZP7A3QYXJU2
COMMIT
# Completed on Thu Nov 16 02:50:17 2017
# Generated by iptables-save v1.6.0 on Thu Nov 16 02:50:17 2017
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1302:324500]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -s 192.168.168.141/32 -p tcp -j ACCEPT
-A INPUT -s 192.168.168.141/32 -p udp -j ACCEPT
-A INPUT -s 192.168.151.236/32 -p tcp -j ACCEPT
-A INPUT -s 192.168.151.236/32 -p udp -j ACCEPT
-A INPUT -i wg0 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p udp -m udp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p udp -m udp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
-A INPUT -p udp -m udp --dport 6443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m udp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -m comment --comment "Allow all loopback traffic" -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "Drop all traffic to 127 that doesn\'t use lo" -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Thu Nov 16 02:50:17 2017
camflan
on 16 Nov 2017
I have a similar issue.
My setup:
k8s version: v1.8.5
I have two nodes (Oracle Linux 7.4)
Network plugin: flannel
node1 -> pod1 (nginx)
node2 -> pod2 (nginx)
node1 to pod1 - works
node2 to pod2 - works
node1 to pod2 -> ping works. curl does not work.
node2 to pod1 -> ping works. curl does not work.
I have added the iptables rules to FORWARD chain. Still does not work. I am really not sure why ping works but not http. Any help is much appreciated.
manojmeda
on 13 Feb 2018
Using tcpdump, I can see that the traffic reaches the pod but the I do not see anything going out from the pod.
manojmeda
on 13 Feb 2018
Having the same issue but with weave. (5 Nodes CoreOS)
I have to agree with @slawekgh ... maybe its linked to docker somehow.
If I ping from other nodes, it works. If I curl from other nodes, itdoes not work.
If I ping from the slave node, it works. if I curl from the slave node (where that specific pod is running), it works.
daveamit
on 19 Feb 2018
Hello,
i've the same problem in my cluster...
Did someone resolved this issue ?
Regards
Julien
jverger
on 12 Apr 2018
facing the same for weave as well
helpp??
RahulG115
on 8 May 2018
Hello,
this post helped me to solve my issue:
It seems that some forwarding rules needs to be set manualy since docker v1.13
Maybe earlier version of flanneld can solve this issue (actually can't have a more recent version than 0.7.0)
Regards
jverger
on 14 May 2018
jverger
on 14 May 2018
I am seeing the exact same symptoms as @manojmeda and @daveamit :
- ping to pods running on other nodes works, but curl does not
- curl to a pod only works from the node running the pod
- with a tcpdump running within a pod, I can see traffic coming in (TCP SYNs), but no reply traffic going out
iptables -P FORWARD ACCEPT is set.
Did anyone manage to get to the bottom of this issue?
bogd
on 21 Jan 2019
Related issues
d11wtq
路
9Comments
outcoldman
路
31Comments
sshukun
路
25Comments
cpg1111
路
26Comments
ljfranklin
路
12Comments
Most helpful comment
and 2nd milestone - after reboot of kubernetes slave node we must execute