Application-gateway-kubernetes-ingress: Ingress controller not configuring App Gateway

[andrewCluey]

Describe the bug
Deployed the AKS cluster and APP gateway following 'tutorial' steps here (https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-aks-applicationgateway-ingress).

Ingress controller helm seemingly deployed fine, no errors reported at deployment.

running kubectl get pods shows the following.

NAME                                       READY   STATUS    RESTARTS   AGE
ingress-azure-1590841283-989fd77f8-wdtkv   0/1     Running   4          37m

Running ```kubectl apply -f aspnetapp.yaml (as described in the tutorial and the troubleshooting guide) seems to go fine, but nothing is actually configured on the app gateway.

App gateway logs show the following:

I0530 13:01:10.358915       1 environment.go:210] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I0530 13:01:10.359109       1 main.go:124] App Gateway Details: Subscription: 7df4fea2-d719-4abe-xxxxxxx, Resource Group: aci, Name: AG01
I0530 13:01:10.359127       1 auth.go:46] Creating authorizer from Azure Managed Service Identity
I0530 13:01:10.359252       1 httpserver.go:57] Starting API Server on :8123

Rather than the expected:

I0927 22:34:51.281437       1 process.go:156] Applied Application Gateway config in 20.461335266s
I0927 22:34:51.281585       1 process.go:165] cache: Updated with latest applied config.
I0927 22:34:51.282342       1 process.go:171] END AppGateway deployment

Also, describing the ingress pod gets the following:

Name:           ingress-azure-1590843666-c89688c49-drccn
Namespace:      default
Priority:       0
Node:           aks-agentpool-35064155-vmss000001/10.0.0.20
Start Time:     Sat, 30 May 2020 13:01:08 +0000
Labels:         aadpodidbinding=ingress-azure-1590843666
                app=ingress-azure
                pod-template-hash=c89688c49
                release=ingress-azure-1590843666
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             10.0.0.22
IPs:            <none>
Controlled By:  ReplicaSet/ingress-azure-1590843666-c89688c49
Containers:
  ingress-azure:
    Container ID:   docker://bfd6e4ca423dcb4bc8b2e55582d29036c591c13ab390903ae1aed6da26e3ee6f
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0-rc1
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@5a6ea8a5d19bf22035fdcad78b18941
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 30 May 2020 13:01:10 +0000
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-azure-1590843666  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  ingress-azure-1590843666-c89688c49-drccn (v1:metadata.name)
      AGIC_POD_NAMESPACE:             default (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590843666-token-(ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  ingress-azure-1590843666-token-:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-azure-1590843666-
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                        Message
  ----     ------     ----                  ----                                        -------
  Normal   Scheduled  4m50s                 default-scheduler                           Successfully assigned default/ingress-azure-1590843666-c89688c49-drccn to aks-agentpool-35064155-vmss000001
  Normal   Pulling    4m49s                 kubelet, aks-agentpool-35064155-vmss000001  Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0-rc1"
  Normal   Pulled     4m49s                 kubelet, aks-agentpool-35064155-vmss000001  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0-rc1"
  Normal   Created    4m48s                 kubelet, aks-agentpool-35064155-vmss000001  Created container ingress-azure
  Normal   Started    4m48s                 kubelet, aks-agentpool-35064155-vmss000001  Started container ingress-azure
  Warning  Unhealthy  79s (x21 over 4m39s)  kubelet, aks-agentpool-35064155-vmss000001  Readiness probe failed: Get http://10.0.0.22:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

To Reproduce
Follow steps in https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-aks-applicationgateway-ingress.

Every time I re-deploy using these steps, same result.

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

SEE ABOVE.

• Output of `kubectl logs .
  SEE ABOVE.
• Any Azure support tickets associated with this issue.
  NA

[akshaysngupta]

@andrewCluey Can you please upgrade to 1.2.0-rc2 which addresses a breaking change in AAD Pod identity which helps AGIC Pod get an AAD token.

[andrewCluey]

Thanks akshaysngupta, that's taken me a couple of steps further, letsEncrypt and CertManager seem to be issuing cert now and validating with http validation. Unfortunately the website/appgateway don't seen to be listening on 443/SSL. So while kubectl get ingress shows both listeners, attemtping to browse to the site over https fails.

[akshaysngupta]

@andrewCluey Can you check these:
1) check if the secret exists
2) look into the AGIC logs and see if you find an error related to secret not found.

[arodriguezdlc]

I have the same issue @andrewCluey. In my case, the problem is that AGIC deploys rules before we have the secret with the certificate configured. AGIC should detect the change on the secret and reapply the configuration with the certificate, but it isn't doing anything.

I have some logs to show it. Here we have the AGIC logs:

E1021 05:41:19.787950       1 secretstore.go:85] Code="ErrorMalformedSecret" Message="secret [baikal-system/shared-azure-next-default-dev-wildcard-certificate-tls] is malformed, tls.key or tls.crt is not defined"
E1021 05:41:19.840067       1 secretstore.go:85] Code="ErrorMalformedSecret" Message="secret [baikal-system/shared-azure-next-default-dev-wildcard-certificate-tls] is malformed, tls.key or tls.crt is not defined"
I1021 05:41:19.841266       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.841505       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.841690       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.841841       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842058       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842233       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842409       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842580       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842746       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.842906       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:19.843112       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843147       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843167       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843189       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843211       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843229       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843255       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843274       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843294       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:19.843319       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
E1021 05:41:19.843577       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-5c57e8e2e3d26f385f552d2e516e200d
E1021 05:41:19.843730       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-7a49275109db8aff8fe8602ad4975ee8
E1021 05:41:19.843875       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-439ba6e5189784fab9807dcbc19cbb57
E1021 05:41:19.844051       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-1e42c1da7b05e4109bffb4fd6642e468
E1021 05:41:19.847442       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-32e21e641594fdff187d6dc69b6df767
E1021 05:41:19.847634       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-af0e9699e66d361527aa6cb09d0c6847
E1021 05:41:19.847837       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-3a2b45d69f6ababa5a0fdd96928accd4
E1021 05:41:19.847989       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-186df5f9faee1db22269e6c1ccda3a8c
E1021 05:41:19.848131       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-03e1eea1ac8ff1a9a4214cf3793d896c
E1021 05:41:19.848272       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-a38451e3f141002bd7f4f2bef8c16514
I1021 05:41:19.862335       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I1021 05:41:50.264843       1 mutate_app_gateway.go:185] Applied generated Application Gateway configuration
I1021 05:41:50.264865       1 mutate_app_gateway.go:200] cache: Updated with latest applied config.
I1021 05:41:50.266890       1 mutate_app_gateway.go:204] END AppGateway deployment
I1021 05:41:50.266914       1 controller.go:151] Completed last event loop run in: 30.478900191s
I1021 05:41:51.365825       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.366157       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.366373       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.371552       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.371913       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.372187       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.372438       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.372689       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.372933       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.373224       1 internaltypes.go:100] Prop name pb-baikal-system-traefik-http-azure-application-gateway-shared-azure-next-default-dev with length 85 is longer than 80 characters; Transformed to pb-baikal-system-traefik-http-azure-application-fa9f0606715d1882f4310772e758aa8d
I1021 05:41:51.373545       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373584       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373614       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373645       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373678       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373711       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373744       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373777       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373812       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
I1021 05:41:51.373845       1 internaltypes.go:100] Prop name bp-baikal-system-traefik-http-80-azure-application-gateway-shared-azure-next-default-dev with length 88 is longer than 80 characters; Transformed to bp-baikal-system-traefik-http-80-azure-applicat-f0894a31f2cf53a4b88f17d014a74326
E1021 05:41:51.374842       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-5c57e8e2e3d26f385f552d2e516e200d
E1021 05:41:51.376977       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-7a49275109db8aff8fe8602ad4975ee8
E1021 05:41:51.377259       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-439ba6e5189784fab9807dcbc19cbb57
E1021 05:41:51.377472       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-1e42c1da7b05e4109bffb4fd6642e468
E1021 05:41:51.377685       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-32e21e641594fdff187d6dc69b6df767
E1021 05:41:51.377901       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-af0e9699e66d361527aa6cb09d0c6847
E1021 05:41:51.378114       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-3a2b45d69f6ababa5a0fdd96928accd4
E1021 05:41:51.378267       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-186df5f9faee1db22269e6c1ccda3a8c
E1021 05:41:51.378402       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-03e1eea1ac8ff1a9a4214cf3793d896c
E1021 05:41:51.378526       1 requestroutingrules.go:258] Will not attach default redirect to rule; SSL Redirect does not exist: /subscriptions/01e91b43-8d5b-41ac-b57d-bdc669732c24/resourceGroups/baikal-shared-azure-next-rg/providers/Microsoft.Network/applicationGateways/baikal-appgw/redirectConfigurations/sslr-fl-a38451e3f141002bd7f4f2bef8c16514
I1021 05:41:51.386584       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I1021 05:42:21.780835       1 mutate_app_gateway.go:185] Applied generated Application Gateway configuration
I1021 05:42:21.780863       1 mutate_app_gateway.go:200] cache: Updated with latest applied config.
I1021 05:42:21.782558       1 mutate_app_gateway.go:204] END AppGateway deployment
I1021 05:42:21.782579       1 controller.go:151] Completed last event loop run in: 30.51553505

As you can see, there is a log that says the secret is malformed (because cert-manager have not configured it yet):
E1021 05:41:19.787950 1 secretstore.go:85] Code="ErrorMalformedSecret" Message="secret [baikal-system/shared-azure-next-default-dev-wildcard-certificate-tls] is malformed, tls.key or tls.crt is not defined"

The problem is that, when cert-manager configures the secret, AGIC doesn't detect the change and doesn't apply the correct SSL endpoint configuration. Here I show cert-manager log:

I1021 05:42:38.606224 1 dns.go:133] cert-manager/controller/challenges/Check "level"=0 "msg"="ACME DNS01 validation record propagated" "dnsName"="default-dev.shared-azure-next.baikalplatform.es" "domain"="default-dev.shared-azure-next.baikalplatform.es" "resource_kind"="Challenge" "resource_name"="shared-azure-next-default-dev-wildcard-2602329146-0" "resource_namespace"="shared-azure-next-default-dev" "type"="dns-01" "fqdn"="_acme-challenge.default-dev.shared-azure-next.baikalplatform.es."

You can see that the secret has been configured AFTER AGIC apply the configuration, and AGIC has detected nothing.ç

PD: restarting the AGIC controller solves the problem because it reloads the secret again.

[arodriguezdlc]

@akshaysngupta any news about this?

[arodriguezdlc]

I have opened a support ticket with Microsoft Azure, and the support engineer told me that this is a recognized bug that will be fixed in future releases. Could you share the issue that is tracking this fix? Thank you.