Kube-router: Route reflector and advertise-cluster-ip=true

[mgrzybowski]

Hi
Kube-router is an awesome project. Thanks ! I was playing with route reflector setup and found something that is counter-intuitive. In my network setup kube-route reflectors peer to externals routers ( and only kube-route-reflector peer outside k8s ) .

Route reflector by definition should not send "self" as next hop to the peers. For advertise-cluster-ip=true this is exactly what they do . I can see on external routers for all /32 cluster-ip services that next-hop are route reflectors ( for /24 pods networks everything is fine ) . What i expect is that each node that has running pod for service will advertise /32 to the route reflector and route reflector will relay this NLRI to other peers.

Despite counter-intuitive design this make route-reflectors bottleneck for traffic to/form cluster-ip services from/to outside the k8s cluster .

Please voice Your opinions. Will this be changed in the future versions of this software ?

Kind Regards
Marek Grzybowski

[sanjmonkey]

Hey Marek,

I found your issue after I was trying to work out what the reasoning was behind not advertising cluster-ip to iBGP peers was (which is breaking when services are externalTrafficPolicy: local and clusterIP lives on another node to the source). See:

https://github.com/cloudnativelabs/kube-router/blob/master/pkg/controllers/routing/export_policies.go

// - each node is NOT allowed to advertise service VIP's (cluster ip, load balancer ip, external IP) to iBGP peers

I've not found the answer to that, but I thought I would comment on your issue at the same time.
How do you mean route-reflector? Kube-router is not acting as a route-reflector, is it? I think what you're explaining you're seeing is just the rules of advertising everything to an eBGP neighbor?

"Route reflector by definition should not send "self" as next hop to the peers. For advertise-cluster-ip=true this is exactly what they do ."
--advertise-cluster-ip=true only affects eBGP, not iBGP, and that default behaviour is indeed to set next-hop-self.

Maybe you could clarify your topology a bit more?

Sandy

[mgrzybowski]

Hi Sandy
Thank You for reply . This export policy You found for not advertaising (cluster ip, load balancer ip, external IP ) to iBGP peers is definitly part of not optimal routing issue and counter-intuitive route-reflector behavior, Also it seems that this rule is for internal kube-router peering only and is not concerning external ibgp peers .

Kube-router can act as route-reflector or route reflector client ( https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#route-reflector-setup--without-full-mesh ).

Network design goal: All host form outside kubernetes cluster can reach to all kubernetes pods and services. And all kubernetes pods can reach to all external hosts.

Everything is in a single AS65000. External route reflectors are needed for filters and more managable peering to ExtenalBoxes. Diagram for BGP peering we use:

• kube-router flags:
  args:
- --run-router=true
- --run-firewall=false
- --run-service-proxy=false
- --enable-ibgp=true
- --nodes-full-mesh=false
- --enable-pod-egress=false
- --advertise-cluster-ip=true

• Kube-RR nodes annotations:
  kube-router.io/rr.server=42
kube-router.io/node.asn=65000
kube-router.io/peer.asns=65000,65000
kube-router.io/peer.ips=172.16.63.1,172.16.63.2

• nodes annotations:
  kube-router.io/rr.client=42
kube-router.io/node.asn=65000

In this setup kube-RR advertise nexhop for cluster-ip as "self" . This mean that for all ExtenalBoxes kube-RR are a getway for every cluster-ip. Cluster-ip services are working, but every packet have to travel via kube-RR node, with is not optimal route.

/24 pod networks are advertised as expected by kube-RR. nexhop for each /24 pod network is the IP of the node that have it ( not ip of route reflector host like for cluster-ip ).

Kind Regards
Marek Grzybowski

[sanjmonkey]

Hi Marek,

I see exactly what you're saying now, thanks for the detailed diagram and explanation.

I (personally) think advertising cluster-ip as self is the right thing for kube-RR to do. After all, kube-RR does live on a kubernetes node, which could have pods on it, but moreover that in kubernetes behaviour, the cluster-ip lives on every node in the kubernetes deployment whilst externalTrafficPolicy is not set to Local (externalTrafficPolicy defaults to cluster ip, which means you will see that IP appear locally on every node in the cluster even if a pod with that service is not running on that node).

Though whilst you have everything in the same AS, you will not see your cluster-ip's be advertised amongst those iBGP sessions - which I coincidentally opened https://github.com/cloudnativelabs/kube-router/issues/730 to seek clarification on.

Personally, I run all my kube-router nodes in a different ASN to my upstream kit and utilise ---advertise-external-ip=true along with externalTrafficPolicy set to local. Then I get the most optimal routing as you could do (these external-ip's are anycast from whichever nodes have pods with that service). I can see how that wouldn't work for all though...

Sandy

[mgrzybowski]

Hi Sandy
Thank You for pointing out externalTrafficPolicy it makes more sense now, but still
I'm failing to understand why cluster-ip lives on every node ( this seems to be wrong, logic suggest that cluster-ip should be on nodes where endpoints are ) . There is also this:

spec.externalTrafficPolicy: Invalid value: "Local": ExternalTrafficPolicy can only be set on NodePort and LoadBalancer service

After reading https://github.com/kubernetes/kubernetes/issues/67202 and https://github.com/kubernetes/kubernetes/issues/69811 i think that k8s cluster-ip is not designed to be exposed/adverised outside k8s cluster or load balanced by ECMP. So far i ended up with updated design that seems to workaround issue with advertising service/cluster/external IP outside k8s cluster and use kube route-reflectors at the same time:

Each kube-node/kube-reouter have four iBGP sessions. Two of them to kube-RR , this sessions are for internal k8s preffix exchange. Another two sessions are to external route reflectors and those sessions have load-balancer IP advartised . It would be great if kube-RR could act as route reflector for ClusterIP , LoadBalancer , ExtrenalIP but unfortunately for unknown reason it reflects only pod networks.

In this setup, service:
```apiVersion: v1
kind: Service
metadata:
name: servicelocal
spec:
type: LoadBalancer
externalTrafficPolicy: Local
selector:
app: servicelocal
ports:

• protocol: TCP
  port: 80
  targetPort: 8080

will have LoadBalancer ip assigned by MetalLB controller and advartised by kube-router to extrenat route reflectors. 

My recepie for ECMP Load Balanced service IP so far:

  * kube router flags

    args:
    - --run-router=true
    - --run-firewall=true
    - --run-service-proxy=false
    - --enable-ibgp=true
    - --nodes-full-mesh=false
    - --enable-pod-egress=false
    - --advertise-cluster-ip=true
    - --advertise-loadbalancer-ip=true

  * Kube-RR annotations:

kube-router.io/rr.server=42
kube-router.io/node.asn=65000
kube-router.io/peer.asns=65000,65000
kube-router.io/peer.ips=172.16.63.1,172.16.63.2

  * Kube-nodes annotations:

kube-router.io/rr.client=42
kube-router.io/node.asn=65000
kube-router.io/peer.asns=65000,65000
kube-router.io/peer.ips=172.16.63.1,172.16.63.2

  * MetalLB controler config:

apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
# speeker is not running, peers will be ignored
- peer-address: 172.16.63.1
peer-asn: 65000
my-asn: 65000
- peer-address: 172.16.63.2
peer-asn: 65000
my-asn: 65000
address-pools:
- name: default
protocol: bgp
addresses:
- 172.16.65.0/24
```

Kind Regards
Marek Grzybowski

[wu0407]

same need brocaster cluster ip out of cluster, and support ecmp--gobgp support add-path

[aauren]

I think that @sanjmonkey's answer is still best: https://github.com/cloudnativelabs/kube-router/issues/729#issuecomment-491092294

It is just the basic premise of kubernetes that all nodes are able to ingress from various service endpoints. Maybe it stems from starting with NodePort where all nodes have to function as an ingress and grew from there?

In general, upstream ClusterIP get's sorted out before the traffic ever leaves the source host via IPVS. It is only with kube-router where we expose the ability to advertise cluster IPs where routing has the ability to go a different way.

I'm glad that you found a work around!