Application-gateway-kubernetes-ingress: Health probe format is not validated and stops AGIC-pod from syncing because of "ApplicationGatewayProbePathIsInvalid"

[gitflo1]

Describe the bug

• Kubernetes version: 1.19.6
• AGIC Helm chart version: ingress-azure-1.3.0

We are using the AGIC, deployed with the helm chart. It stopped syncing all Ingress definitions with the attached Azure Application Gateway due to the following error: ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayProbePathIsInvalid" Message="Path specified for Probe <...> is not valid. Path must start with a forward slash (/) and meet the format requirements." Details=[]

To Reproduce

Steps to reproduce the behavior:

1. Install the AGIC chart with Helm on your AKS cluster and connect it properly to the Azure Application Gateway
2. Create a valid Ingress configuration for a service
3. Create a deployment yaml with an invalid health probe (e.g. missing forward slash)

apiVersion: apps/v1
kind: Deployment
  ...
      containers:
        ...
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: healthz
            port: 8080
            scheme: HTTP

The deployment will be applied without any issues or validation errors. As soon as the deployment is applied, AGIC will stop the sync with the attached Azure Application Gateway with the following error message. Output of `kubectl logs : ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayProbePathIsInvalid" Message="Path specified for Probe <...> is not valid. Path must start with a forward slash (/) and meet the format requirements." Details=[]

[akshaysngupta]

@gitflo1 Thanks for sharing this.
As such, we should be able to fix this on AGIC by prefixing a "/". This also might be impacting the paths in the ingress manifest.

[sureshgoli25]

We also have similar issue. Single Ingress resource blocked entire AGIC Updates to App GW.

[macleanj]

Unfortunately we are suffering the same. Until fixed we will go ahead and add some alarming in Splunk to catch this before the user is affected. In our case the ingress path was configured "/api/*". OK in itself, but when the probe fails we will hit this issue.