flannel network issue --- cannot ping pod on other nodes

[sd1587745]

Hi there,

I have a kubernetes cluster with version 1.3.10, now pod on one node cannot ping pod on different node, but can ping other's docker port.

for example:

root@pcnbj-sto024:/etc/default# kubectl get pods -o wide
NAME                        READY     STATUS    RESTARTS   AGE       IP            NODE
my-nginx-2395715568-4ai66   1/1       Running   2          1d        172.31.11.4   10.148.13.236
my-nginx-2395715568-dyr5w   1/1       Running   1          7h        172.31.15.3   10.140.140.29
my-nginx-2395715568-kpd19   1/1       Running   2          1d        172.31.15.2   10.140.140.29
my-nginx-2395715568-z3k3q   1/1       Running   0          7h        172.31.9.2    10.140.35.98

u see IP of pod on 10.148.13.236 is 172.31.11.4, I cannot ping it from 10.140.140.29:

root@pcnbj-cp027:/var/lib# ping 172.31.11.4
PING 172.31.11.4 (172.31.11.4) 56(84) bytes of data.
^C

but docker0 port on 10.148.13.236 is reachable from 10.140.140.29:

IP address on 10.148.13.236:

root@pcnbj-sto024:/etc/default# ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 02:42:61:43:c7:9a  
          inet addr:172.31.11.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:61ff:fe43:c79a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:57466 errors:0 dropped:0 overruns:0 frame:0
          TX packets:57847 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:11560525 (11.5 MB)  TX bytes:6530376 (6.5 MB)

root@pcnbj-sto024:/etc/default# ifconfig flannel.1
flannel.1 Link encap:Ethernet  HWaddr e6:e0:c3:ee:7a:ac  
          inet addr:172.31.11.0  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::e4e0:c3ff:feee:7aac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7930 errors:0 dropped:0 overruns:0 frame:0
          TX packets:94 errors:0 dropped:11 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:631824 (631.8 KB)  TX bytes:8844 (8.8 KB)

root@pcnbj-cp027:/var/lib# ping 172.31.11.1
PING 172.31.11.1 (172.31.11.1) 56(84) bytes of data.
64 bytes from 172.31.11.1: icmp_seq=1 ttl=64 time=0.267 ms
64 bytes from 172.31.11.1: icmp_seq=2 ttl=64 time=0.234 ms
64 bytes from 172.31.11.1: icmp_seq=3 ttl=64 time=0.231 ms
^C

I also checked flannel ip forward feature are enabled on both nodes, please help me to fingure out what happend, thanks all.

root@pcnbj-cp027:/var/lib# sysctl -a | grep 'net.ipv4.conf.flannel/1.forwarding'
net.ipv4.conf.flannel/1.forwarding = 1

[sd1587745]

SW versions:

KUBE_VERSION=1.3.10
FLANNEL_VERSION=0.5.5
ETCD_VERSION=3.1.6

and using flannel vxlan mode.

root@pcnbj-sto024:/etc/default# etcdctl get /coreos.com/network/subnets/172.31.15.0-24
{"PublicIP":"10.140.140.29","BackendType":"vxlan","BackendData":{"VtepMAC":"16:f3:7d:8c:04:ab"}}

[tomdee]

Was this working previously? Did you do anything that could have stopped it from working? Are you able to retry with the latest flannel release?

[sd1587745]

when I use flannel 0.7.1 and a bug as this link occurred: [https://github.com/coreos/flannel/issues/196]

my etcd has new subnet profile for these hosts, docker0 port is set to correct subnet, but flannel.1 port remains old config.

root@pcnbj-sto024:/var/run# etcdctl get /coreos.com/network/subnets/172.31.92.0-24
{"PublicIP":"10.148.13.236","BackendType":"vxlan","BackendData":{"VtepMAC":"e6:e0:c3:ee:7a:ac"}}

docker0   Link encap:Ethernet  HWaddr 02:42:f9:8f:51:2b  
          inet addr:172.31.92.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:f9ff:fe8f:512b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:536 (536.0 B)  TX bytes:648 (648.0 B)

root@pcnbj-sto024:/var/run# ifconfig flannel.1
flannel.1 Link encap:Ethernet  HWaddr e6:e0:c3:ee:7a:ac  
          inet addr:172.31.11.0  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::e4e0:c3ff:feee:7aac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7941 errors:0 dropped:0 overruns:0 frame:0
          TX packets:145 errors:0 dropped:27 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:632524 (632.5 KB)  TX bytes:14266 (14.2 KB)

acturally subnet.env is correct for flannel, but I restart flanneld many times it still keeps older IP subnet:

root@pcnbj-sto024:/var/run# cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.31.0.0/16
FLANNEL_SUBNET=172.31.92.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

how to fix it.

[sd1587745]

also pid and log file doesn't created, I'm using ubuntu14.04.3.

config file in /etc/init.d/flanneld

FLANNEL_PIDFILE=/var/run/$BASE.pid
FLANNEL_LOGFILE=/var/log/$BASE.log

FLANNEL_START="start-stop-daemon \
--start \
--background \
--quiet \
--exec $FLANNEL \
--make-pidfile --pidfile $FLANNEL_PIDFILE \
-- $FLANNEL_OPTS \
>> $FLANNEL_LOGFILE 2>&1"

no files created:

root@pcnbj-sto024:~# cat /var/run/flanneld.pid cat: /var/run/flanneld.pid: No such file or directory root@pcnbj-sto024:~# cat /var/log/flanneld.log cat: /var/log/flanneld.log: No such file or directory

[sd1587745]

anyway, when change back to flannel version 0.5.5, flannel.1 auto updated, not version 0.7.1.
pid and log files still doesn't exist.

[Sherler]

same issue, can ping the crossed node docker0, but not worked when pinging crossed node pods

[tomdee]

Closing this stale support issue

[jnickc]

sd1587745, did you solved this issue? I have exactly the same problem

[phhutter]

@sd1587745 did you finally solve it ?

[azalio]

Today I have faced this problem and have solved it by running the command:
# iptables --policy FORWARD ACCEPT

[lvthillo]

same issue, iptables does not seem to help.

[tqs0117]

Today I have faced this problem and have solved it by running the command:
# iptables --policy FORWARD ACCEPT

it helps in my case, thanks

[sxyandapp]

Today I have faced this problem and have solved it by running the command:
# iptables --policy FORWARD ACCEPT

牛逼啊，这个好使

[masaeedu]

@azalio Running this command where? Could you expand on what the problem is and how this fixes it?

[sxyandapp]

@azalio Running this command where? Could you expand on what the problem is and how this fixes it?

long time , i have forget the detail, u can try to run that command on every node