Application-gateway-kubernetes-ingress: App Gateway sslcert not deleted during terraform destroy (also max 100 sslcerts)

Created on 28 Aug 2020 · 2Comments · Source: Azure/application-gateway-kubernetes-ingress

Describe the bug
We use Helm and Terraform to deploy services into AKS, it seems when we 'terraform destroy' a deployed service the sslcert data that was applied to the App Gateway is not deleted.

To Reproduce
Deployed service with an ingress controller, here is the ingress.yaml that is used as part of the deployment

kubectl get ingress -n auto-terratestjdr1 -o yaml

`apiVersion: v1
items:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
appgw.ingress.kubernetes.io/backend-protocol: https
appgw.ingress.kubernetes.io/request-timeout: "60"
appgw.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: azure/application-gateway
creationTimestamp: "2020-08-28T14:34:10Z"
generation: 1
labels:
netsec.barracuda.com/environment: dev
netsec.barracuda.com/name: terratestjdr1
name: terratestjdr1
namespace: auto-terratestjdr1
resourceVersion: "27567478"
selfLink: /apis/extensions/v1beta1/namespaces/auto-terratestjdr1/ingresses/terratestjdr1
uid: fdd32ed1-a5b6-4fbe-95b5-016749007494
spec:
rules:
- host: springboot-demo-appjdr1.dev.bnngm.azure.cudaops.com
  
  http:
  
  paths:
  - backend:
    
    serviceName: terratestjdr1
    
    servicePort: 8443
    
    tls:
- hosts:
  - springboot-demo-appjdr1.dev.bnngm.azure.cudaops.com
    
    secretName: star-dev-bnngm-azure-cudaops-com
    
    status:
    
    loadBalancer:
    
    ingress:
  - ip: 51.105.147.67
    
    kind: List
    
    metadata:
    
    resourceVersion: ""
    
    selfLink: ""`

Verify that the cert has been installed

az network application-gateway ssl-cert list -g bnngm-dev-westeurope --gateway-name bnngm-dev-westeurope-aks-appgw --subscription blah-blah | jq

{ "data": null, "etag": "W/\"cf3dcf6a-0fe4-4038-a59a-428923b3362e\"", "id": "/subscriptions/4b7cd783-c55a-4319-a0d7-a3a68ef112b1/resourceGroups/bnngm-dev-westeurope/providers/Microsoft.Network/applicationGateways/bnngm-dev-westeurope-aks-appgw/sslCertificates/auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "keyVaultSecretId": null, "name": "auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "password": null, "provisioningState": "Succeeded", "publicCertData": "FAKEDATA", "resourceGroup": "bnngm-dev-westeurope", "type": "Microsoft.Network/applicationGateways/sslCertificates" }

Destroy the deployment and check for the cert again;
az network application-gateway ssl-cert list -g bnngm-dev-westeurope --gateway-name bnngm-dev-westeurope-aks-appgw --subscription blah-blah | jq

[ { "data": null, "etag": "W/\"738c1667-1f66-4560-ba3e-b28ae25b4460\"", "id": "/subscriptions/4b7cd783-c55a-4319-a0d7-a3a68ef112b1/resourceGroups/bnngm-dev-westeurope/providers/Microsoft.Network/applicationGateways/bnngm-dev-westeurope-aks-appgw/sslCertificates/auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "keyVaultSecretId": null, "name": "auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "password": null, "provisioningState": "Succeeded", "publicCertData": "FAKEIT", "resourceGroup": "bnngm-dev-westeurope", "type": "Microsoft.Network/applicationGateways/sslCertificates" } ]

The sslcert is not removed with the rest of the deployment.
Once we hit 100 deployments then we cannot deploy anymore.

Ingress Controller details
kubectl describe pod -n ingress-azure ingress-azure-69c7c9d66d-sjgkl

Name: ingress-azure-69c7c9d66d-sjgkl
Namespace: ingress-azure
Priority: 0
Node: aks-agentpool-18118092-vmss000000/10.153.40.4
Start Time: Mon, 27 Jul 2020 21:21:10 +0100
Labels: aadpodidbinding=ingress-azure
app=ingress-azure
pod-template-hash=69c7c9d66d
release=ingress-azure
Annotations: checksum/config: 92368a0b96949a0f3e010d229a45caa26fea001a744095ee3f780595cc508397
kubectl.kubernetes.io/restartedAt: 2020-07-15T11:10:45+01:00
prometheus.io/port: 8123
prometheus.io/scrape: true
Status: Running
IP: 10.153.40.9
IPs:
IP: 10.153.40.9
Controlled By: ReplicaSet/ingress-azure-69c7c9d66d
Containers:
ingress-azure:
Container ID: docker://cffae75aa13e593b9d1422ac3c1aa8ea991768bd607af7b8e29fdf8467178e91
Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0
Image ID: docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:de458f962eab0cd2de19d23dfeb9a0e4bc2565a38f8c45cc98a74f3cda8b940c
Port:
Host Port:
State: Running
Started: Mon, 27 Jul 2020 21:21:23 +0100
Ready: True
Restart Count: 0
Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
ingress-azure ConfigMap Optional: false
Environment:
AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json
AGIC_POD_NAME: ingress-azure-69c7c9d66d-sjgkl (v1:metadata.name)
AGIC_POD_NAMESPACE: ingress-azure (v1:metadata.namespace)
Mounts:
/etc/appgw/azure.json from azure (rw)
/var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-token-dhdlh (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
azure:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/azure.json
HostPathType: File
ingress-azure-token-dhdlh:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-azure-token-dhdlh
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:

terraform -version Terraform v0.12.29

SSL improvement

Source

jimusbobus

👍2

Most helpful comment

Hi,

I'm experiencing the same issue. I don't think that it classifies as an improvement but must be treated as a bug as the ingress controller simply stops working in my environment after I've added/deleted 100 ingress definitions. I think it's important that the ingress controller cleans up after itself and when it deletes a listener it also has to delete the associated SSL certificate.

Nils