Aws-load-balancer-controller: Deleting a service/ingress does not delete AWS ALB resources created

[crsantini]

Hi @bigkraig

[ISSUE]
I'm using K8S v1.10.5 on Rancher 2.0.6 RKE cluster with provider AWS. When I set my service and ingress, the ALBB resources are being successfully created on AWS. However, once I delete my service and ingress value-ad-vwowal/valueadvwowal-cui-ingress (not the cluster), the resources are still showing on AWS and ALB controller shows the following on the logs:

[LOGS]
E0808 19:41:25.931565 1 albingress.go:166] value-ad-vwowal/valueadvwowal-cui-ingress: error instantiating load balancer: Unable to find the value-ad-vwowal/valueadvwowal-cui service: no object matching key "value-ad-vwowal/valueadvwowal-cui" in local store
E0808 19:41:25.931615 1 albingress.go:167] value-ad-vwowal/valueadvwowal-cui-ingress: Will retry in 1m0.178083172s
I0808 19:41:25.931933 1 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"value-ad-vwowal", Name:"valueadvwowal-cui-ingress", UID:"6c16f206-9b42-11e8-bb65-02f78957f2aa", APIVersion:"extensions/v1beta1", ResourceVersion:"5807", FieldPath:""}): type: 'Warning' reason: 'ERROR' error instantiating load balancer: Unable to find the value-ad-vwowal/valueadvwowal-cui service: no object matching key "value-ad-vwowal/valueadvwowal-cui" in local store

[EXPECTANCY]
All ALB resources associated to the Service/Ingress to be deleted

Thanks
Carlos

[dpavlov-smartling]

Hello, same on EKS with k8s version 1.10 and aws-alb-ingress-controller 1.0-beta.6. All related resources are still persists even after service/ingress deleted.

[bigkraig]

Yep, the reasoning behind this is that when it assembles the state for the ingress it will error when the dependencies are missing, we will need to ignore those errors when its a delete

[crsantini]

hi @bigkraig , hope you are doing well! do you have an estimation when the deletion issue would be fixed?

[jansony1]

meet the same problem with version 1.0-beta.7

[Morriz]

@bigkraig this is still not resolved with:

  repository: docker.io/amazon/aws-alb-ingress-controller
  tag: "v1.1.2"

ALB resources are not deleted

[sdarwin]

@Morriz , you are reporting the ALB resources are not deleted. Something to watch out for is deleting and creating many resources in the cluster all at once (which might happen during repeated tests).

• Make sure to only delete the ingress object, while leaving the ingress controller in place.
• If deleting multiple/all items, the ingress object should be deleted some seconds before the ingress controller.

[M00nF1sh]

@sdarwin is correct.
And we'll add Finalizer support to block deleting Ingress object when ingress controller is gone.

[gregsidelinger]

When was the finalizer support added? I'm running v1.1.2 and an ingress was deleted however I still see the ALB in AWS. Now sure what happened as the logs do not seem to show anything odd and normally they do get cleaned up.

Also does the controller scan for ALBs with the kubernetes.io/cluster/ owned tag and clean up things that do not have a corresponding ingress anymore? This is how external-dns and a few other controllers work.

[rakhee-balaraman]

I had the alb ingress controller 1.1.4 and a few ingress objects in my EKS 1.16 cluster. I deleted the controller before deleting the ingress objects and deployed the newer aws load balancer controller. The previous ALBs were not deleted, and now I am not able to delete them anymore. What do you suggest I should do to delete the previous ALBs from my AWS account?

[Morriz]

Is this still an issue when using eksctl to create and then destroy a cluster with running loadbalancers? I thought I read eksctl was equipped to do that nowadays. I don't remember having issues destroying a cluster that way. I might be mistaken. (I am just closing old GH issues and came across this.)

[M00nF1sh]

@rakhee-balaraman
When migrate to v2, You'll need to manual actions on worker node's securityGroup rules added by v1.
checkout this migration doc for instructions :D If security-groups annotation isn't used, the SecurityGroup rule on worker node's SecurityGroup that allow LoadBalancer traffic should be manually adjusted post migration

[M00nF1sh]

@Morriz as long as you make sure to delete ingresses(and wait for ingress's deletion), there won't be resource leaks.
i don't think EKSCTL automatically delete ingresses for u.