Aws-load-balancer-controller: Configuration for 0 downtime deployments

[benwilson512]

Hey folks, I'm having a fair bit of trouble getting 0 downtime deployments to work. The issue:

After a new pod is passes its readiness check, the ALB target group places the Pod's IP in an "initial" state which can last for a couple of seconds.

However since the new pod is ready as far as K8s is concerned, it begins to terminate an old pod, which immediately enters a "draining" state on the target group. At this point there are no pods available to answer requests.

To some extent this can be handled by simply increasing the number of pods. This isn't really a solution though, it just lowers the probability that the rolling deployment outpaces the ALB's ability to keep up. If the AWS API were to undergo any kind of delay or outage, the deployment could complete without any live pods actually registered in the target group.

Is there any known way to require a pod show up as "healthy" in the target group before K8s considers it alive?

[M00nF1sh]

Hi Ben,

As far as i know, there is no good solution this under "mode ip" other than increasing the number of pods. The root cause is the ingress resources is only aware of service, and it's not aware of pods

However, to achieve 0 downtime deployment, you can use "mode instance" 😸

[benwilson512]

Hey @M00nF1sh thanks for the pointer! I can confirm that on my end it does indeed look like using the instance mode helps. Would a PR be welcomed to elaborate on the significance of the mode in the docs?

[M00nF1sh]

sure, PR are welcome 😸 .

talked with @bigkraig , mode ip might also support 0 downtime in the future, but we don't know how to achieve that. maybe needs k8s core change

[benwilson512]

Right, it would seem that to do 0 downtime with the ip mode you'd need some sort of "availableProbe" that worked distinct from readinessProbe, which would check that the pod was healthy and reachable from the target group.

Anyway, I'll look into making a PR to note that instance is required for 0 downtime, thank you!

[BrianChristie]

Could _Pod Readiness Gates_ help here?

• KEP: https://github.com/kubernetes/enhancements/issues/580
• Docs: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate

It introduces ReadinessGate on Pods, which looks for a matching item in status.conditions.

At first glance, it looks like a custom PodCondition like amazonaws.com/alb-pod-ip-healthy could be introduced, set to Ready by the ingress controller once the PodIP has gone Healthy in the ALB TargetGroup.

PodDisruptionBudget could then be used to pause a Deployment rolling update until a sufficient number of new Pods are Healthy.

It seems this would solve the issue of Pods being terminated before they are Healthy in the ALB.

Edit: Actually I think that a normal Deployment rolling update should suffice, with .spec.strategy.rollingUpdate.MaxUnavailable. No PodDisruptionBudget required.

/CC: @bigkraig @M00nF1sh for discussion