Aws-load-balancer-controller: Pod readiness gate not working after v2 upgrade

Created on 23 Oct 2020 · 6Comments · Source: kubernetes-sigs/aws-load-balancer-controller

After migrate to the controller v2, looks like pod readiness gate stopped working. This is the state the readiness gate is showing after the upgrade:

Readiness Gates:
  Type                                                                     Status
  target-health.alb.ingress.k8s.aws/web-qa-ckapp-web_web-qa-ckapp-web_80   <none>

This is the ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:XXXXXXXXXXXX:certificate/XXXXXXXXXXXX,arn:aws:acm:us-east-2:XXXXXXXXXXXX:certificate/8abac528-ea4d-4529-b994-ee0e4694287a
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "10"
    alb.ingress.kubernetes.io/healthcheck-path: /users/login
    alb.ingress.kubernetes.io/healthcheck-port: "80"
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/inbound-cidrs: 0.0.0.0/0
    alb.ingress.kubernetes.io/ip-address-type: ipv4
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/load-balancer-attributes: deletion_protection.enabled=true,routing.http2.enabled=true,idle_timeout.timeout_seconds=60
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=35,load_balancing.algorithm.type=round_robin
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "3"
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: web-qa
    meta.helm.sh/release-namespace: ckapp-qa-web
  labels:
    app.kubernetes.io/component: puma
    app.kubernetes.io/instance: web-qa
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ckapp-web
    app.kubernetes.io/version: 16d965a8
    helm.sh/chart: ckapp-web-0.0.466
  name: web-qa-ckapp-web
  namespace: ckapp-qa-web
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
        pathType: ImplementationSpecific
      - backend:
          serviceName: web-qa-ckapp-web
          servicePort: 80
        path: /*
        pathType: ImplementationSpecific

After the migration existing pod readiness gates are just fine, so looks like it only affects to the new ones that tries to register:

➜ kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP              NODE                                          NOMINATED NODE   READINESS GATES
web-qa-ckapp-web-5bf6d745dd-7wdg9   3/3     Running   0          3m1s    10.110.17.214    ip-10-110-24-8.us-east-2.compute.internal      <none>           0/1
web-qa-ckapp-web-78c68ff54-f6m6v    3/3     Running   0          6h51m   10.110.144.226   ip-10-110-153-222.us-east-2.compute.internal   <none>           1/1

Any idea why is this not working with the new v2?

Source

ivanmp91

Most helpful comment

@ivanmp91
sorry, we should update the docs about readinessGate.
now we require you to label your namespace with elbv2.k8s.aws/pod-readiness-gate-inject: enabled.
After you labels your namespace with above, we have a webhook to automatically inject new format of readinessGate, so you don't need to add the old format of readinessGate into your deployments manually anymore. However, (if the old format readinessGate is there, the webhook will remove it upon pod creation)

M00nF1sh on 23 Oct 2020

👍2

All 6 comments

I've just tried to rollback, deleting the new v2 controller deployment and deploy the old v1 and automatically readiness gate gets successfully updated:

Readiness Gates:
  Type                                                                               Status
  target-health.alb.ingress.k8s.aws/web-qa-ckapp-web_web-qa-ckapp-web_80   True
Conditions:
  Type                                                                               Status
  target-health.alb.ingress.k8s.aws/web-qa-ckapp-web_web-qa-ckapp-web_80   True
  Initialized                                                                        True
  Ready                                                                              True
  ContainersReady                                                                    True
  PodScheduled                                                                       True

ivanmp91 on 23 Oct 2020

M00nF1sh on 23 Oct 2020

👍2

gotcha! Thank you so much for your quick response @M00nF1sh

ivanmp91 on 23 Oct 2020

@M00nF1sh, I've added the NS annotations like you suggested, however newly-created pods' readiness gates are still failing to be updated, and the new format doesn't appear to have been injected. CRDs are all installed correctly, and I can't find any entries concerning readiness gates in the Controller logs. Any suggestions? What exactly is the new readiness gate format?

Whoops, nevermind, I can't read and added annotations instead of labels. I'm seeing the new format injected as expected now.