Kube-router: Routing / NAT issue on multiple network interfaces host
Context
Given the following scenario:
A kubernetes node called FE with two network interfaces:
- (front-end interface) eth0 ip 172.24.34.201 net 172.24.34.192/27
- (back-end interface) eth1 ip 10.224.162.173 net 10.224.162.160/27
A kubernetes node called BE with a single network interface:
- (back-end interface) eth0 ip 10.224.139.21 net 10.224.139.0/27
ps: both nodes communicate through a switch/router.
The BE is running some services that need to be accessed from FE.
The issue we will describe here happens with any kind of service but we're going to use the DNS as a concrete example.
The pod_cidr is 10.244.0.0/16 and the service_cluster_ip_range is 10.255.2.0/24.
The issue, tests, and outputs
The issue
In the FE node when we do a dig command to the DNS service that have its PODs running at the BE node, it does not work.
It does not work only when this command is run while in the host itself or in a POD's container with host_network=true.
[root@FE]# dig @10.255.2.2 kubernetes.default.svc.cluster.local
; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.255.2.2 kubernetes.default.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
When running the same command from a POD's container with
host_network=falseit does work.
The tests and outputs
What we observed was:
- the ip
10.255.2.2is not routable (judging by theip routeoutput) but it's bound to akube-dummy-if - this ip is then NAT'ed by IPVS
[root@FE]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
...
UDP 10.255.2.2:53 rr
-> 10.244.2.135:53 Masq 1 0 0
-> 10.244.8.16:53 Masq 1 0 0
- Let's say the IPVS picked the real
10.244.2.135:53 - The real ip is then tunneled (tun-1022413921)
[root@FE]# ip route
default via 172.24.34.193 dev eth0
10.0.0.0/8 via 10.224.162.161 dev eth1
10.224.162.160/27 dev eth1 proto kernel scope link src 10.224.162.173
10.244.2.0/24 dev tun-1022413921 proto 17 src 10.224.162.173
10.244.4.0/24 dev kube-bridge proto kernel scope link src 10.244.4.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.24.34.192/27 dev eth0 proto kernel scope link src 172.24.34.201
- this tunnel will send packets to BE
10.224.139.21(as desired)
[root@FE]# ip tunnel show tun-1022413921
tun-1022413921: ip/ip remote 10.224.139.21 local 10.224.162.173 dev eth1 ttl inherit
- Since it didn't work we tried to observe from
tcpdumpany packets going or coming to BE host from FE host.
[root@FE]# tcpdump -i any host 10.224.139.21 -n -l -vvnneSs 0
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:30:57.258529 Out 06:25:20:00:6b:2e ethertype IPv4 (0x0800), length 129: (tos 0x0, ttl 64, id 42897, offset 0, flags [DF], proto IPIP (4), length 113)
10.224.162.173 > 10.224.139.21: (tos 0x0, ttl 64, id 17500, offset 0, flags [none], proto UDP (17), length 93)
172.24.34.201.12397 > 10.244.2.135.53: [udp sum ok] 45008+ [1au] A? kubernetes.default.svc.cluster.local. ar: . OPT UDPsize=4096 (65)
- And we also did the
tcpdumpfrom BE node as well.
[root@BE]# tcpdump -i any host 10.224.162.173 -n -l -vvnneSs 0
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:41:04.248968 In d8:9e:f3:d5:0f:e2 ethertype IPv4 (0x0800), length 129: (tos 0x0, ttl 63, id 28179, offset 0, flags [DF], proto IPIP (4), length 113)
10.224.162.173 > 10.224.139.21: (tos 0x0, ttl 64, id 38639, offset 0, flags [none], proto UDP (17), length 93)
172.24.34.201.31474 > 10.244.2.135.53: [udp sum ok] 15559+ [1au] A? kubernetes.default.svc.cluster.local. ar: . OPT UDPsize=4096 (65)
# in case you want to know BE can't return from "172.24.34.201"
[root@BE]# ip route
default via 10.224.139.1 dev eth0
10.224.139.0/27 dev eth0 proto kernel scope link src 10.224.139.21
10.244.2.0/24 dev kube-bridge proto kernel scope link src 10.244.2.1
10.244.4.0/24 dev tun-10224162173 proto 17 src 10.224.139.21
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
Partial conclusion
It seems that the IPVS from FE is doing the NAT partially correctly, it does forward to the right real but it uses as source the wrong source IP (eth0).
The tunnel packet (outer ipip header) is correct while the tunneled packet (inner ip) is wrong, it should use as SNAT the ip from the interface eth1 but it's using the ip from the first interface, in this specific case the eth0.
10.224.162.173 > 10.224.139.21: (tos 0x0, ttl 64, id 17500, offset 0, flags [none], proto UDP (17), length 93)
172.24.34.201.12397 > 10.244.2.135.53: [udp sum ok] 45008+ [1au]
Points that support our conclusions
When we do the dig to the real ip, thus skipping the IPVS, it works:
[root@FE]# dig @10.244.2.135 kubernetes.default.svc.cluster.local
; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> @10.244.2.135 kubernetes.default.svc.cluster.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59764
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;kubernetes.default.svc.cluster.local. IN A
;; ANSWER SECTION:
kubernetes.default.svc.cluster.local. 5 IN A 10.255.2.1
;; Query time: 388 msec
;; SERVER: 10.244.2.135#53(10.244.2.135)
;; WHEN: Wed Oct 03 20:45:25 UTC 2018
;; MSG SIZE rcvd: 117
When we inverted the interface order using eth0 as the back-end interface it works (without no additional route or any other modification).
- (back-end interface) eth0 ip 10.224.162.173 net 10.224.162.160/27
- (front-end interface) eth1 ip 172.24.34.201 net 172.24.34.192/27
Questions
- Can we instruct kube-router/ipvs to use a given interface as source ip?
leandromoreira
All 9 comments
thanks @leandromoreira for the very detailed report
The way Linux choose the source IP address has been a problem before as well. Please see https://github.com/cloudnativelabs/kube-router/issues/376 which is extreme case where same service VIP is selected as source IP.
So to avoid that in current implementation of kube-router an explicit rule is added to use node IP as source when accessing the cluster IP.
If you see ip route list table local you see the source IP used would be Kubernetes node IP of that node when accessing the cluster IP.
On FE is 172.24.34.201 is the node IP?
murali-reddy
on 4 Oct 2018
Thank you so much @murali-reddy, when I run the command ip route list table local I see these info:
[root@FE]# ip route list table local
...
local 10.255.2.2 dev kube-dummy-if proto kernel scope host src 10.224.162.173
broadcast 10.255.2.2 dev kube-dummy-if proto kernel scope link src 10.255.2.2
...
broadcast 172.24.34.192 dev eth0 proto kernel scope link src 172.24.34.201
local 172.24.34.201 dev eth0 proto kernel scope host src 172.24.34.201
broadcast 172.24.34.223 dev eth0 proto kernel scope link src 172.24.34.201
I took off some of them and left the ones related to the ips: 10.255.2.2 and 172.24.34.201.
On FE is 172.24.34.201 is the node IP?
Is node IP the same as the internal node (InternalIP)? I've pasted some information from the FE node, please let me know if you need anything else.
root@desktop:~$ kubectl get nodes -o wide
NAME STATUS AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION
...
BE.cp.domain.com Ready 316d v1.11.3 <none> Red Hat Enterprise Linux Server 7.4 (Maipo) 3.10.0-693.2.2.el7.x86_64
FE.cp.domain.com Ready 15d v1.11.3 <none> Red Hat Enterprise Linux Server 7.5 (Maipo) 3.10.0-693.21.1.el7.x86_64
and
root@desktop:~$ kubectl describe node FE.cp.domain.com
Name: FE.cp.domain.com
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/kube-proxy-ds-ready=true
beta.kubernetes.io/os=linux
datacenter=cm
kubernetes.io/hostname=FE.cp.domain.com
machine_type=virtual
net=bo
pool=live
projectcalico.org/ds-ready=true
tier=fe
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Tue, 18 Sep 2018 18:19:45 -0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Thu, 04 Oct 2018 14:39:00 -0300 Wed, 03 Oct 2018 09:05:52 -0300 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 04 Oct 2018 14:39:00 -0300 Wed, 03 Oct 2018 09:05:52 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 04 Oct 2018 14:39:00 -0300 Wed, 03 Oct 2018 09:05:52 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 04 Oct 2018 14:39:00 -0300 Tue, 18 Sep 2018 18:19:45 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 04 Oct 2018 14:39:00 -0300 Wed, 03 Oct 2018 09:06:03 -0300 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.224.162.173
Hostname: FE.cp.domain.com
Capacity:
cpu: 2
ephemeral-storage: 6553320Ki
hugepages-2Mi: 0
memory: 3872032Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 6039539703
hugepages-2Mi: 0
memory: 3769632Ki
pods: 110
System Info:
Machine ID: e5b76d9872644f33920114109d76e091
System UUID: 462EA3EC-DD9F-CEC3-B5A1-FF12AA89F2C0
Boot ID: bd58cf6a-e6dd-4ed4-bd2c-d11a5a496786
Kernel Version: 3.10.0-693.21.1.el7.x86_64
OS Image: Red Hat Enterprise Linux Server 7.5 (Maipo)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://17.7.0
Kubelet Version: v1.11.3
Kube-Proxy Version: v1.11.3
PodCIDR: 10.244.4.0/24
ExternalID: FE.cp.domain.com
Non-terminated Pods: (11 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system fluentd-v0.14.11-qpspg 100m (5%) 0 (0%) 300Mi (8%) 300Mi (8%)
kube-system kube-router-lrd7q 250m (12%) 0 (0%) 250Mi (6%) 0 (0%)
kube-system snap-n7j88 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system tiller-deploy-d7f99b8b-kwz4z 0 (0%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
550m (27%) 0 (0%) 750Mi (20%) 300Mi (8%)
Events: <none>
Strange.
local 10.255.2.2 dev kube-dummy-if proto kernel scope host src 10.224.162.173
broadcast 10.255.2.2 dev kube-dummy-if proto kernel scope link src 10.255.2.2
Above rule should have ensured that source IP used for accessing 10.255.2.2 is 10.224.162.173. But it not used, linux kernel choose the source IP.
I will try if I can reproduce and find about routing rule is sufficient or not.
murali-reddy
on 5 Oct 2018
Thanks, please let me know if I can assist you in any way!
leandromoreira
on 5 Oct 2018
I'd like to chime in with our use case:
We have one public facing interface eth0 and one internal encrypted interface eth1 for node to node communication.
All kubernetes related traffic goes through eth1: Service traffic, Pod traffic, Node to Node traffic.
Our IP-IP tun interfaces are all mapped to eth1 and work as expected.
In addition we'd like to be able to reach kubernetes svc addresses from our nodes, in our case to access a private docker registry running in kubernetes.
In the current state if the node tries to access anything cluster related, it does go through the right tun interface to the destination host, though it has the source ip of _eth0_ (probably because default gw?).
That means the traffic cannot travel back.
We tried cloudnativelabs/kube-router/pull/668 and it does indeed fix our problem! Thanks you @lucasmundim
arminbuerkle
on 14 May 2019
@lucasmundim is kinda magic 馃槃
leandromoreira
on 15 May 2019
I'm hoping this is the right issue - since rebuilding one of my nodes (10.1.11.28) recently, I think i've been running in to this issue in two slightly different ways. Each node has multiple interfaces on different L2 subnets and can reach other nodes without going via a router (i.e. each node has an interface on 10.1.10.0/24, 10.1.11.0/24, 10.1.15.0/24 and 10.1.16.0/24).
- Connections from pods on other nodes to this node's real IP address (
10.1.11.28) fail. This is most obvious for example when using a Prometheus pod to scrape host metrics from the node (i.e.curl 10.1.11.28:9100/metrics). Looking at traffic dumps, the reply (TCP SYN+ACK) from the node exporter on this recently rebuilt node (10.1.11.28) to the other node is sent via the default gateway (10.1.11.1) with a destination of one of the service's endpoints (10.233.66.74) instead of going via/directly to the source node (10.1.11.27). This gets picked up by the firewall on the default gateway as an invalid TCP state, as it never saw the original TCP SYN that was sent directly from the source->destination node and is dropped.
Example Traffic Dumps
Cluster IP: 10.233.61.234
Service Endpoint IP: 10.233.66.74
Source Node IP: 10.1.11.27
Destination Node IP: 10.1.11.28
Source node:
tcpdump -i any -n host 10.233.66.74 -n -l -vvnneSs 0
12:24:16.229660 P 0a:58:0a:e9:42:4a ethertype IPv4 (0x0800), length 76: (tos 0x0, ttl 64, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x627e (incorrect -> 0x8432), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.229660 In 0a:58:0a:e9:42:4a ethertype IPv4 (0x0800), length 76: (tos 0x0, ttl 64, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x627e (incorrect -> 0x8432), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.229715 Out 00:02:c9:4d:69:02 ethertype IPv4 (0x0800), length 76: (tos 0x0, ttl 63, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x627e (incorrect -> 0x8432), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.229718 Out 00:02:c9:4d:69:02 ethertype 802.1Q (0x8100), length 80: vlan 110, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x627e (incorrect -> 0x8432), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.768299 P 0a:58:0a:e9:42:4a ethertype IPv4 (0x0800), length 1244: (tos 0x0, ttl 64, id 19425, offset 0, flags [DF], proto TCP (6), length 1228)
Destination node;
tcpdump -i any -n host 10.233.66.74 -n -l -vvnneSs 0
12:24:16.225694 In 00:02:c9:4d:69:02 ethertype 802.1Q (0x8100), length 80: vlan 110, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x8432 (correct), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.225694 In 00:02:c9:4d:69:02 ethertype 802.1Q (0x8100), length 80: vlan 110, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x8432 (correct), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.225694 In 00:02:c9:4d:69:02 ethertype IPv4 (0x0800), length 76: (tos 0x0, ttl 63, id 51507, offset 0, flags [DF], proto TCP (6), length 60)
10.233.66.74.39708 > 10.1.11.28.9100: Flags [S], cksum 0x8432 (correct), seq 4018199760, win 29200, options [mss 1460,sackOK,TS val 1340413070 ecr 0,nop,wscale 7], length 0
12:24:16.225859 Out a0:36:9f:08:4a:b8 ethertype IPv4 (0x0800), length 76: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.11.28.9100 > 10.233.66.74.39708: Flags [S.], cksum 0x627e (incorrect -> 0x0d5b), seq 264035802, ack 4018199761, win 26844, options [mss 8960,sackOK,TS val 3034111162 ecr 1340405778,nop,wscale 7], length 0
12:24:16.225869 Out a0:36:9f:08:4a:b8 ethertype 802.1Q (0x8100), length 80: vlan 110, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.11.28.9100 > 10.233.66.74.39708: Flags [S.], cksum 0x627e (incorrect -> 0x0d5b), seq 264035802, ack 4018199761, win 26844, options [mss 8960,sackOK,TS val 3034111162 ecr 1340405778,nop,wscale 7], length 0
12:24:16.225878 Out a0:36:9f:08:4a:b8 ethertype 802.1Q (0x8100), length 80: vlan 110, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.11.28.9100 > 10.233.66.74.39708: Flags [S.], cksum 0x627e (incorrect -> 0x0d5b), seq 264035802, ack 4018199761, win 26844, options [mss 8960,sackOK,TS val 3034111162 ecr 1340405778,nop,wscale 7], length 0
Firewall
Blocks the following packet;
Source IP:Port: 10.1.11.28.9100
Destination IP:Port: 10.233.66.74.39708
TCP Flags: SYN, ACK
What appears to be happening is that the original SYN reaches the node's real IP correctly, which then replies with a SYN ACK back towards the Service's Endpoint. However, the reply packet isn't sent directly to the node as expected but is instead sent via the default gateway. All nodes are on the same L2 subnet (10.1.11.0/24), although they are connected to multiple L2 subnets (10.1.10.0/24, 10.1.15.0/24 and 10.1.16.0/24) as well.
If I query the routing table for the Service's Endpoint IP, all seems correct - the next hop is given as the source node rather than the default gateway;
ip ro get 10.233.66.74
10.233.66.74 via 10.1.11.27 dev eth1.110 src 10.1.11.28 uid 0
cache
ipvsadm has an entry for the Service Endpoints;
TCP 10.1.11.28:9090 rr persistent 10800
-> 10.233.65.73:9090 Masq 1 0 0
-> 10.233.66.74:9090 Masq 1 0 0
- Connections from pods on the rebuilt node to other Service IPs fail. This is most noticeable with DNS queries. Similar to the first variant, the firewall/default gateway blocks a packet sourced from the node but with a destination of the DNS Service Endpoint IP. Checking the node's routing table it expects to send the packet directly to the other node, rather than via the default gateway.
I have taken @leandromoreira 's PR (#668) and deployed it across my cluster. It appears to fix the second issue (i.e. I no longer see UDP DNS packets showing up at the default gateway and pods on the rebuilt node can make DNS queries without issue), however I am still experiencing the first issue.
One thing that might be complicating this is that the rebuilt node was rebuilt using MAAS 2.6.0, which added support for multiple default gateways/replies from the same interface that they came in on. All other machines in the cluster were previously built using MAAS 2.5.0 which does not do this - although multiple interfaces are present, only one has a default gateway. As a result, the routing table rules do differ somewhat;
Existing Node (e.g. 10.1.11.27)
ip ru sh
0: from all lookup local
32765: from 10.233.66.0/24 lookup 77
32765: from all lookup 79
32766: from all lookup main
32767: from all lookup default
ip ro sh ta main
default via 10.1.11.1 dev eth1.110 proto static
10.1.10.0/24 dev eth0 proto kernel scope link src 10.1.10.27
10.1.11.0/24 dev eth1.110 proto kernel scope link src 10.1.11.27
10.1.15.0/24 dev eth1.150 proto kernel scope link src 10.1.15.27
10.1.16.0/24 dev eth1.160 proto kernel scope link src 10.1.16.27
10.233.64.0/24 via 10.1.11.21 dev eth1.110 proto 17
10.233.65.0/24 via 10.1.11.26 dev eth1.110 proto 17
10.233.66.0/24 dev kube-bridge proto kernel scope link src 10.233.66.1
10.233.67.0/24 via 10.1.11.28 dev eth1.110 proto 17
10.233.71.0/24 via 10.1.11.22 dev eth1.110 proto 17
10.233.73.0/24 via 10.1.11.23 dev eth1.110 proto 17
10.233.74.0/24 via 10.1.11.24 dev eth1.110 proto 17
10.233.75.0/24 via 10.1.11.25 dev eth1.110 proto 17
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
Rebuilt Node (10.1.11.28)
ip ru sh
0: from all lookup local
0: from 10.1.11.0/24 to 10.1.11.0/24 lookup main
0: from 10.1.16.0/24 to 10.1.16.0/24 lookup main
0: from 10.233.67.0/24 lookup 77
100: from 10.1.11.0/24 lookup 1
100: from 10.1.16.0/24 lookup 2
32765: from all lookup 79
32766: from all lookup main
32767: from all lookup default
ip ro sh ta main
default via 10.1.10.1 dev enp3s0 proto static
10.1.10.0/24 dev enp3s0 proto kernel scope link src 10.1.10.28
10.1.11.0/24 dev eth1.110 proto kernel scope link src 10.1.11.28
10.1.15.0/24 dev eth1.150 proto kernel scope link src 10.1.15.28
10.1.16.0/24 dev eth1.160 proto kernel scope link src 10.1.16.28
10.233.64.0/24 via 10.1.11.21 dev eth1.110 proto 17
10.233.65.0/24 via 10.1.11.26 dev eth1.110 proto 17
10.233.66.0/24 via 10.1.11.27 dev eth1.110 proto 17
10.233.67.0/24 dev kube-bridge proto kernel scope link src 10.233.67.1
10.233.71.0/24 via 10.1.11.22 dev eth1.110 proto 17
10.233.73.0/24 via 10.1.11.23 dev eth1.110 proto 17
10.233.74.0/24 via 10.1.11.24 dev eth1.110 proto 17
10.233.75.0/24 via 10.1.11.25 dev eth1.110 proto 17
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
ip ro sh ta 1
default via 10.1.11.1 dev eth1.110 proto static
ip ro sh ta 2
default via 10.1.16.1 dev eth1.160 proto static
There is a (strong!) possibility that this routing change on the rebuilt node is cause of this issue, but I can't quite work out exactly why. On the rebuilt node the reply packet should still drop through to the main table and then with a destination of the Service's Endpoint IP sent via the node running that pod. It's almost as if the iptables postrouting chain is not being applied?
iptables -L -t nat -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */
KUBE-ROUTER-HAIRPIN all -- 0.0.0.0/0 0.0.0.0/0 vdir ORIGINAL
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 match-set kube-router-pod-subnets src ! match-set kube-router-pod-subnets dst ! match-set kube-router-node-ips dst
SNAT all -- !10.233.67.0/24 !10.233.67.0/24 vdir ORIGINAL vmethod MASQ /* */ to:10.1.11.28
Chain KUBE-MARK-DROP (0 references)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-MARK-MASQ (0 references)
target prot opt source destination
MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000
Chain KUBE-ROUTER-HAIRPIN (1 references)
target prot opt source destination
SNAT all -- 10.233.65.73 10.233.65.73 vaddr 10.233.61.234 vport 9090 to:10.233.61.234
SNAT all -- 10.233.66.74 10.233.66.74 vaddr 10.233.61.234 vport 9090 to:10.233.61.234
(unrelated entries in the KUBE-ROUTER-HAIRPIN chain have been removed for brevity).
Apologies if this problem is purely down to the routing setup on my rebuilt node and not actually this issue specifically. I am a bit lost with this at the moment but this issue does seem incredibly similar to the two issues I am facing, and the second issue in particular appears to be fixed by applying @leandromoreira 's PR.
KingJ
on 2 Jun 2019
@leandromoreira, @murali-reddy I wonder if there are any news on that or some workaround?
trnl
on 6 Jun 2019
local 10.255.2.2 dev kube-dummy-if proto kernel scope host src 10.224.162.173 broadcast 10.255.2.2 dev kube-dummy-if proto kernel scope link src 10.255.2.2Above rule should have ensured that source IP used for accessing 10.255.2.2 is 10.224.162.173. But it not used, linux kernel choose the source IP.
What I found is that if under 10.255.2.2 VIP we have 2 real server IPs - one which have to be accessed via tunnel (nodes in different subnet) will have incorrect source IP. At the same time if pod reachable without tunnel (nodes in same subnet) - correct src (10.224.162.173 in the example above) will be used.
So you'll ending up to have 50% success rate on packets.
Moreover, checking the traffic on the interfaces I've noticed that in the IPIP tunnel the source IP is different:
22:25:08.401468 IP 172.23.125.244 > 172.24.81.53: IP 172.23.115.90.36606 > 10.0.69.8.8080: Flags [S], seq 4176119449, win 43690, options [mss 65495,sackOK,TS val 37357454 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
22:25:09.402681 IP 172.23.125.244 > 172.24.81.53: IP 172.23.115.90.36606 > 10.0.69.8.8080: Flags [S], seq 4176119449, win 43690, options [mss 65495,sackOK,TS val 37358456 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
In the above 172.23.125.244 is eth1 which we expect traffic to be send over, and 172.23.115.90 is the eth0 on the same VM.
As you see we have a tunnel 172.23.125.244 > 172.24.81.53 and 10.0.69.8 reachable via it.
But the traffic without tunnel is not changed:
22:40:47.911721 IP 172.23.125.244.39132 > 10.0.11.16.8080: Flags [.], ack 305, win 237, options [nop,nop,TS val 38296965 ecr 38293328], length 0
22:40:47.912054 IP 172.23.125.244.39132 > 10.0.11.16.8080: Flags [F.], seq 79, ack 305, win 237, options [nop,nop,TS val 38296965 ecr 38293328], length 0
22:40:47.912460 IP 172.23.125.244.39132 > 10.0.11.16.8080: Flags [.], ack 306, win 237, options [nop,nop,TS val 38296965 ecr 38293329], length 0
So either traffic is masqueraded and then snatted again in 2nd case (without tunnel)
trnl
on 6 Jun 2019
Related issues
mgrzybowski
路
6Comments
tkoeck
路
7Comments
cvhbsk
路
10Comments
murali-reddy
路
9Comments
SEJeff
路
6Comments
Most helpful comment
I'd like to chime in with our use case:
We have one public facing interface eth0 and one internal encrypted interface eth1 for node to node communication.
All kubernetes related traffic goes through eth1: Service traffic, Pod traffic, Node to Node traffic.
Our IP-IP tun interfaces are all mapped to eth1 and work as expected.
In addition we'd like to be able to reach kubernetes svc addresses from our nodes, in our case to access a private docker registry running in kubernetes.
In the current state if the node tries to access anything cluster related, it does go through the right tun interface to the destination host, though it has the source ip of _eth0_ (probably because default gw?).
That means the traffic cannot travel back.
We tried cloudnativelabs/kube-router/pull/668 and it does indeed fix our problem! Thanks you @lucasmundim