Aws-load-balancer-controller: alb.ingress.kubernetes.io/tags not updating properly

[t0rr3sp3dr0]

If you update alb.ingress.kubernetes.io/tags, tags are not created, deleted, or updated on ELB. For it to update you need to change alb.ingress.kubernetes.io/tags again, and it will show the previous tags. Then you change it again and the tags from the second application will appear, but not the ones from the last application.

| | Tags on Kubernetes YAML | Tags on AWS ELB |
|-|-|-|
| Initial Apply | A=a | A=a |
| 2nd Apply | B=b | A=a |
| 3rd Apply | C=c | B=b |
| 4th Apply | | C=c |
| 5th Apply | D=d | |
| 6th Apply | E=e | D=d |
| 7th Apply | E=e,F=f | E=e |
| 8th Apply | F=f | E=e,F=f |
| 9th Apply | G=g | F=f |
| 10th Apply | G=g | G=g |

The AWS tags are always one state behind from the ones specified on Kubernetes YAML.

It is even stranger because AddTags and DeleteTags events are logged on CloudTrail and on AWS search bar I can see the tags and its values, but if I filter no ELBs are listed. The same happens on AWS CLI and Boto3 if you describe the ELBs, its tags do not appear or appear outdated.

I've waited more than 10min to verify if there was some delay on tag application but nothing happened. The tags just update if I change Ingress YAML again.

The changes on YAML doesn't necessarily need to be on alb.ingress.kubernetes.io/tags, any change the forces Kubernetes to reconfigure Ingress will make the tags from the previous configuration appear.

• Initial Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: A=a
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 2nd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: B=b
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 3rd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: C=c
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 4th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 5th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: D=d
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 6th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 7th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e,F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 8th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 9th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 10th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
    H: h
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

[fimbulvetr]

I can confirm this behavior. Tags are properly applied to TargetGroups, however.

[rifelpet]

waf-acl-id is also experiencing similar behavior. Adding the annotation to an existing ingress doesnt do anything. removing the annotation causes the ingress controller to add the waf acl to the alb. I believe that the ingress controller's sync period will eventually reconcile the proper behavior, but this defaults to 1 hour.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[salemgolemugoo]

I have the same issue and it's not being fixed

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swarmer]

/reopen

_Well, I tried_

[k8s-ci-robot]

@swarmer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[t0rr3sp3dr0]

/reopen

[k8s-ci-robot]

@t0rr3sp3dr0: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tibin-mfl]

Any update on this issue ? Even I am facing same issue while enabling logs on alb.

[k8s-ci-robot]

@tibin-mfl: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[msonnabaum]

Just ran into the exact same issue trying to update alb.ingress.kubernetes.io/load-balancer-attributes.

[MaxBango]

Also seeing the problem with alb.ingress.kubernetes.io/load-balancer-attributes.

ALB Ingress Controller v1.1.8, Kubernetes v1.17.