Kube-router: External IP routing issues when using externalTrafficPolicy=Local (without DSR)

[jdconti]

I'm currently using --advertise-external-ip to advertise (via BGP) an arbitrary static IP (10.129.245.5) for use with nginx-ingress-controller. Changing the nginx-ingress-controller service to use externalTrafficPolicy=Local we found that the external IP was no longer reachable from pods which did _not_ exist on the same node where nginx-ingress-controller pods were running.

Further troubleshooting revealed that certain routes aren't being cleaned up after setting externalTrafficPolicy=local. The following is the relevant contents of the routing tables on a kubelet that's not running any pods for the ingress controller service (after updating the ingress controller service's externalTrafficPolicy to Local):

# ip route show table all |grep 245.5
10.129.245.5 dev kube-bridge  table 79  scope link
local 10.129.245.5 dev kube-dummy-if  table local  proto kernel  scope host  src 10.129.0.131

After cleaning up manually, I can see that setupRoutesForExternalIPForDSR() is attempting to add external routes to a custom route table:

# ip route del table 79 10.129.245.5
# ip route del table local 10.129.245.5
# sleep $sync_period
# ip route show table all |grep 245.5
10.129.245.5 dev kube-bridge  table 79  scope link

Which of course results in No route to host from any pod that's not on the node that's advertising 10.129.245.5 and the following error in kube-router logs (-v=10):
network_services_controller.go:1753] Failed to add route for 10.129.245.5 in custom route table for external IP's due to: exit status 2

Is the current expectation that external IP advertisement is only for use with DSR (since it seems we're triggering DSR-specific methods)?

[murali-reddy]

Is the current expectation that external IP advertisement is only for use with DSR (since it seems we're triggering DSR-specific methods)?

External IP advertisement is indepedently usable functionality irrespective of DSR is in use. I will try to reproduce the issue.

[murali-reddy]

Fixed by #495