Kube-router: Feature: L3 ECMP + L4 director + DSR

Created on 22 Sep 2017 · 4Comments · Source: cloudnativelabs/kube-router

Consolidatng one blanket bug to track serveral sub-features to achieve highly available, scalable ingress model for bare-metal deployments.

       ECMP/L3 LB                    consistent hash
Router -------------> L4 director-------------------> L7 proxy--------------> pod

       ECMP/L3 LB                    consistent hash
Router -------------> L4 director-------------------> pod

[x] define a custom service annotation to indicate service needs DSR
[x] ability to advertise external ip's #161 if service is marked with annotation
[x] support consistent hashing for IPVS service #166 is marked with annotation
[x] Ability to create IPVS service with FWMARK's if service is marked with annotation
[x] Ability to create IPVS service with IPIP tunnel mode
[x] add additional tunnel interface inside the service endpoint pod, and configure the external ip

enhancement

Source

murali-reddy

Most helpful comment

I am working on a prototype. Will share once i have something in testable state, for feedback.

Here are the manual steps. Assuming 192.168.100.101 is the external IP and 10.1.1.56 is service endpoint

iptables -t mangle -A PREROUTING -d 192.168.100.101 -m tcp -p tcp --dport 80 -j MARK --set-mark 10

ipvsadm -A --fwmark-service 10 -s sh
ipvsadm -a --fwmark-service 10 -r 10.1.1.56 -i

get the pid of the pod by container id

docker inspect --format '{{.State.Pid}}' 292a9274a97d

add the tunnel interface inside the pod

 nsenter -t 2358 -n ip tunnel add tunnel0 mode ipip local 10.1.1.56

add the vip to tunl0 interface

 nsenter -t 2358 -n ip addr  add 192.168.100.101 dev tunl0

Last disable rp_filter on container, as packet arrives on tunnel interface and leaves on the eth0

nsenter -t 2358 -n sysctl -w net.ipv4.conf.all.rp_filter=0

murali-reddy on 22 Sep 2017

👍3