Ingress-nginx: Bare-metal Ingress HA using keepalived

[mqliang]

bare-metal ingress is not HA. Our team implement a keepalived-vip side car container for Ingress. Which could be deployed with ingress-controller container to provide HA. For example, first create a Ingress Replicaset with the keep-alived-vip side car:

apiVersion: v1
kind: ReplicationController
metadata:
  namespace: "kube-system"
  name: "ingress-test1"
  labels:
    run: "ingress"
spec:
  replicas: 2
  selector:
    run: "ingress"
  template:
   metadata:
    labels:
      run: "ingress"
   spec:
    hostNetwork: true
    containers:
      - image: "ingress-keepalived-vip:v0.0.1"
        imagePullPolicy: "Always"
        name: "keepalived-vip"
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
        securityContext:
          privileged: true
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: SERVICE_NAME
            value: "ingress"
      - image: "ingress-controller:v0.0.1"
        imagePullPolicy: "Always"
        name: "ingress-controller"
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 200m
            memory: 200Mi

Then create a Service named "ingress" pointing to the Ingress ReplicaSet, and assign a vip to the service using annotation.

apiVersion: v1
kind: Service
metadata:
  name: ingress
  namespace: kube-system
  annotations:
    "ingress.alpha.k8s.io/ingress-vip": "192.168.10.1"
spec:
  type: ClusterIP
  selector:
    - run: "ingress"
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80

keepalived-vip side car container will watch the "ingress" Service and update keepalived's config. Thus make Ingress Pod be HA. user should access in-cluster service by the Ingress VIP.

The keepalived approache is first describe in https://github.com/kubernetes/kubernetes/pull/34013

see implementation here: https://github.com/mqliang/contrib/tree/ingress-admin/ingress-admin/ingress-keepalived-vip

[mqliang]

We'd like contribute the keepalived-vip side car to community. Is here the right place? @bprashanth

[aledbf]

@mqliang what's the difference with https://github.com/kubernetes/contrib/tree/master/keepalived-vip?

[mqliang]

@aledbf The idea is same: using keepalived for HA. The difference is :

• IIUC, https://github.com/kubernetes/contrib/tree/master/keepalived-vip aims at: provide HA for NodePort type service, deployed as DaemonSet

• my keepalived side car aims at: provide HA for Ingress-controller on bare-metal, deployed as a side-car with the ingress-container (in the same Pod).

[aledbf]

provide HA for NodePort type service, deployed as DaemonSet

that's not true. It will expose any service type and daemonset is not a requirement.

my keepalived side car aims at: provide HA for Ingress-controller on bare-metal, deployed as a side-car with the ingress-container (in the same Pod).

I think this should not run as a sidecar because you are coupling the keepalived to the controller

[mqliang]

@aledbf
IIUC, https://github.com/kubernetes/contrib/tree/master/keepalived-vip approache: put a HA IPVS cluster at the front of Ingress? So, request need first reach the keepalived-vip pod, then the keepalived-vip pod forward the request to the ingress-controller pod using IPVS?

[mqliang]

and my approache is: put the keepalived-vip container as a side car, together with ingress-controller container, in one pod. And the pod are runnning using hostNetwork. So the Ingress controller Pod itself is HA. This also prevent the double hop.

Another reason I want run keepalived-vip container as a sidecar is: it dramatically simplify Ingress creation and deployment, since in this way the Ingress controller itself is HA, the only requirements is:

• run Ingress controller pod using host network
• ensure Ingress Pod of the same Service run on different node, which is not hard to do with Anti-affinity feature.

I think this should be considered a complement, not a replacement for https://github.com/kubernetes/contrib/tree/master/keepalived-vip

[ddysher]

@aledbf we'll put together a repo of what we did and how it went about ingress ha, hope that will make things more clear.

[DanyC97]

@ddysher am i right the above repo you mentioned is https://github.com/caicloud/ingress-admin/pull/1 ? so i can subscribe and keep track ;)

[ddysher]

Sure, we'll sort it out ASAP. Feel free to subscribe :)