Cilium: BGP routing in cilium

[mehrdadpfg]

Hi,

do you have any plan to implement BGP routing in cilium?
like the work done in kube-router project using gobgp library

[tgraf]

do you have any plan to implement BGP routing in cilium?
like the work done in kube-router project using gobgp library

If you run Cilium in direct routing mode, it will be compatible with all routing daemons that install the routes in the regular Linux routing tables. That said, we are not opposed to somebody adding gobgp to Cilium.

[mrostecki]

BGP routing in Cilium with gobgp is something that is interesting for us too. I will get back soon with a more detailed idea how it might be implemented.

/cc @nirmoy

[mehrdadpfg]

thank's for following up 😍

[tgraf]

Another option discussed with @andrewsykim is to use kube-router directly and use CNI chaining or another form of mutual awareness. This would include running Cilium in direct routing mode with kube-router using gobgp to distribute routes.

[andrewsykim]

After chatting with @tgraf and @mrostecki, I think an implementation that uses kube-router (only for BGP advertisement) and cilium for everything else is definitely a viable option! @mrostecki happy to work with you to get this working, I will also try to get a working PoC to see if there is anything we need to do in kube-router for this to work :)

cc @murali-reddy

[mrostecki]

The approach which we agreed on is to try to run both Cilium (with direct route mode) and kube-router (only with --run-router option) as DaemonSets and see whether it's working. I will try that in next 1-2 days and let you know about the results.

[tgraf]

@andrewsykim @mrostecki The only open question I have at this point is IPAM. Cilium does it's own IPAM right now but it's easy to plug in other stuff. How would this work together with kube-router?

[andrewsykim]

@tgraf I could be missing some context here as I'm still getting familiar with the project. I had the impression that cilium uses IPAM from kubernetes (http://cilium.readthedocs.io/en/latest/kubernetes/install/#enable-automatic-node-cidr-allocation-recommended) which is what kube-router uses as well.

[tgraf]

I could be missing some context here as I'm still getting familiar with the project. I had the impression that cilium uses IPAM from kubernetes (http://cilium.readthedocs.io/en/latest/kubernetes/install/#enable-automatic-node-cidr-allocation-recommended) which is what kube-router uses as well.

@andrewsykim Yes we do. Cilium will allocate out of the CIDR that is associated with the node resource. We even use the k8s node allocator go package for this. Sounds like this is a non issue. All good 👍

[andrewsykim]

Merged the necessary changes in kube-router so we can disable all it's CNI pieces and use it only for BGP routing. If someone can test the following config (replacing BGP peering information as necessary) that would be great.

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-router
    tier: node
  name: kube-router
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        k8s-app: kube-router
        tier: node
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: kube-router
      containers:
      - name: kube-router
        image: andrewsykim/kube-router:cilium-2463
        imagePullPolicy: Always
        args:
        - "--run-router=true"
        - "--run-firewall=false"
        - "--run-service-proxy=false"
        - "--enable-cni=false"
        - "--enable-ibgp=false"
        - "--enable-overlay=false"
        - "--peer-router-ips=<CHANGE ME>"
        - "--peer-router-asns=<CHANGE ME>"
        - "--cluster-asn=<CHANGE ME>"
        - "--advertise-cluster-ip=true"
        - "--advertise-external-ip=true"
        - "--advertise-loadbalancer-ip=true"
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        livenessProbe:
          httpGet:
            path: /healthz
            port: 20244
          initialDelaySeconds: 10
          periodSeconds: 3
        resources:
          requests:
            cpu: 250m
            memory: 250Mi
        securityContext:
          privileged: true
      hostNetwork: true
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-router
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kube-router
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    resources:
      - namespaces
      - pods
      - services
      - nodes
      - endpoints
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - "networking.k8s.io"
    resources:
      - networkpolicies
    verbs:
      - list
      - get
      - watch
  - apiGroups:
    - extensions
    resources:
      - networkpolicies
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kube-router
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-router
subjects:
- kind: ServiceAccount
  name: kube-router
  namespace: kube-system

[mrostecki]

@andrewsykim Cool. I will test it soon.

[Shinzu]

@andrewsykim im not that familiar with kuberouter, so i have a question:

if i dont want/need to advertise the routes to an external router for now , do i omitting just these 2 peer-router flags?

[andrewsykim]

Hi @Shinzu! The above config wouldn't work if you removed the 2 peer route flags. If you don't need to advertise routes externally (i.e. only in-cluster traffic) then there's little reason to use kube-router since you can just use cilium routing in your cluster instead of having it overlay with kube-router.

[Shinzu]

hi @andrewsykim ,i wanted to test cilium without the overlay network with the --device flag ,for that i would still need that inside the cluster the routes the the pod cidr of each node is advertised . i hoped that would that be done by kuberouter since cilium cannot do it by its own.

[Shinzu]

when i read the bgp docs i would assume that should be possible

[andrewsykim]

Ahh I see, in that case yes you can set "--enable-ibgp=true" to enable in-cluster BGP peering.

[Shinzu]

ok thy @andrewsykim

[Shinzu]

tried it and it is working with the enabled ibgp flag

[ianvernon]

Is there anything else that needs to be done here?

[tgraf]

The kube-router integration implements this properly. Closing.