Application-gateway-kubernetes-ingress: Application Gateway Ingress Controller only works with root path /

[christianskovholm]

Describe the bug

I have deployed the standard aspnet app from Microsoft, from the following YAML:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: microservicesapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /test
        backend:
          serviceName: aspnetapp
          servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: aspnetapp
spec:
  selector:
    app: aspnetapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp
  labels:
    app: aspnetapp
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP

Everything works fine when I call the public IP associated with my AGW resourced. However, I want to configure the path of the ingress, so that it is /test. Like so:

  - http:
      paths:
      - path: /test
        backend:
          serviceName: aspnetapp
          servicePort: 80

When I attempt to call the endpoint from outside the cluster, this results in 502 Bad Gateway.

To Reproduce
Steps to reproduce the behavior:

1. Follow all the steps outlined in: https://docs.microsoft.com/en-us/azure/developer/terraform/create-k8s-cluster-with-aks-applicationgateway-ingress

2. Add appgw.ingress.kubernetes.io/backend-path-prefix: "/" to ingress metadata annotations.

3. Change path in aspnetapp.yaml from / to /test.

4. run kubectl apply -f aspnetapp.yaml to deploy changes.

Ingress Controller details

NOTE: The warning at the bottom is because i tried changing the path to different values to find a solution.

Name:         ingress-azure-1600451618-8454589c75-47mrq
Namespace:    default
Priority:     0
Node:         aks-agentpool-35064155-vmss000000/15.0.0.4
Start Time:   Fri, 18 Sep 2020 19:53:41 +0200
Labels:       aadpodidbinding=ingress-azure-1600451618
              app=ingress-azure
              pod-template-hash=8454589c75
              release=ingress-azure-1600451618
Annotations:  checksum/config: cdb5cf9a90944e08180557ca44bacecc24772078fcf588e315b818b529eefecb
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           15.0.0.5
IPs:
  IP:           15.0.0.5
Controlled By:  ReplicaSet/ingress-azure-1600451618-8454589c75
Containers:
  ingress-azure:
    Container ID:   docker://a36a71e75d09c633921b7b625110983af0d7dd686c90cfa619e904b641d3e37f
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:de458f962eab0cd2de19d23dfeb9a0e4bc2565a38f8c45cc98a74f3cda8b940c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 18 Sep 2020 19:53:43 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-azure-1600451618  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  ingress-azure-1600451618-8454589c75-47mrq (v1:metadata.name)
      AGIC_POD_NAMESPACE:             default (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1600451618-token-wq7b2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  ingress-azure-1600451618-token-wq7b2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-azure-1600451618-token-wq7b2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                     Age                  From                                                                  Message
  ----     ------                     ----                 ----                                                                  -------
  Normal   Scheduled                  12m                  default-scheduler                                                     Successfully assigned default/ingress-azure-1600451618-8454589c75-47mrq to aks-agentpool-35064155-vmss000000
  Normal   Pulling                    12m                  kubelet, aks-agentpool-35064155-vmss000000                            Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0"
  Normal   Pulled                     12m                  kubelet, aks-agentpool-35064155-vmss000000                            Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0"
  Normal   Created                    12m                  kubelet, aks-agentpool-35064155-vmss000000                            Created container ingress-azure
  Normal   Started                    12m                  kubelet, aks-agentpool-35064155-vmss000000                            Started container ingress-azure
  Warning  Unhealthy                  11m (x4 over 12m)    kubelet, aks-agentpool-35064155-vmss000000                            Readiness probe failed: Get http://15.0.0.5:8123/health/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  FailedApplyingAppGwConfig  6m26s (x2 over 10m)  azure/application-gateway, ingress-azure-1600451618-8454589c75-47mrq  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /org(/|$)(.*) in the path rule /subscriptions/95480706-7ff6-4b17-82ae-a6e739302f25/resourceGroups/microservicesapp/providers/Microsoft.Network/applicationGateways/agw-prod-northeurope/urlPathMaps/url-e1903c8aa3446b7b3207aec6d6ecba8a/pathRules/pr-default-microservicesapp-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]

[3quanfeng]

Hey @christianskovholm , could you please share the healthy probe you configured for your backend deployments?

[Vishal2696]

@christianskovholm @3quanfeng This seems to be a error in configuration. I'm just highlighting what "Chris" has mentioned here. In the ingress yaml you have added the annotation appgw.ingress.kubernetes.io/backend-path-prefix: "/" and then you have mentioned the path as /test. If you want the ingress URL to be http://IP-Address/test then you have to remove the annotation mentioned here . the role of the annotation is to redirect to / no matter the what the path is mentioned in the URL. Correct me If I'm wrong.

[DevOpsGeek1]

Even I am facing same issue works with path / and not working with /any-other-path. it seems to be backend node is not changing even after giving /any-other-path its still residing in /

[DevOpsGeek1]

@vishal8k if you remove annotation having / , we will get 502 error for /test

[kumarg25]

Yes, I am also facing the same issue. It only works with path / and not working with /any-other-path. Removing the annotation having "/" results in 502 Bad Gateway error.

[3quanfeng]

Hi @vishal8k, you are correct, Thanks!
@kumarg25 , basically if appgw.ingress.kubernetes.io/backend-path-prefix: "/" is used, it basically means your AKS workload exposes only "/", for example, http://hostnameorip/
If your workload exposes like http://hostnameorip/any-other-path, then you don't need that annotation
You could also verify your healthy probe, if it's heathy and the path is "/", then it means your workload exposes at least "/"

[DevOpsGeek1]

But removing annotation also doesnt help I dont know what is the resolution for this issue closure?

[DevOpsGeek1]

after removing / annotation why are we getting 502 bad gateway? Can some one help ?

[christianskovholm]

I second @DevOpsGeek1 - I also still receive 502 bad gateway after removing the annotation.

[Vishal2696]

@DevOpsGeek1 This is clearly not an AGIC issue but just an URL path issue. Firstly you have to check whether your application has any page to serve on the path you are trying (Eg: /test). If you are sure about it then try giving /test/*.

[DevOpsGeek1]

I configured / and /test in ingress yaml file and also there are back end services are running. but strangle it works only for / and /test is always giving 502 error. so this is clearly issue with AGIC. also tried /test/* as well but the issue is same 502

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
     paths:
      - path: /vote
        backend:
          serviceName: azure-vote-front 
          servicePort: 80
      - backend:
          serviceName: aspnetapp
          servicePort: 80

[DevOpsGeek1]

@vishal8k Kindly address this and help us.

[Vishal2696]

@DevOpsGeek1 Let's first understand the issue here. The AGIC is able to send requests successfully when configured with path /.
When configured with /test we receive a 502 error. What does a 502 mean? 502 means the application gateway was not able to find any backend configured with the path in the URL request that it received. This means the request we sent did not contain an exact /test or either the backend application doesn't have any response for the URL path. This issue is already closed and doesn't seem like an issue on AGIC side. Anyway since you still doubt on it , I would suggest you to open a question on a forum such as stackoverflow where we can continue this discussion. Please post the link for your question in the next reply. We shall discuss this further there, If it is confirmed that this is an AGIC issue , then we shall open a new issue here again.

[DevOpsGeek1]

@vishal8k I understand that it is unable to reach backend pool but when I check AppGateway from azure portal everything configured properly in backend pool and there is no issue with backend configuration also, There is already an issue opened in stack overflow as well, but not having much progress.

Stackoverflow

[DevOpsGeek1]

Repro:
Followed below doc and installed application gateway ingress controller and then sample app yaml I will mention below. it doent work when we give path /test

AGIC install document

sample_app.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp
  labels:
    app: aspnetapp
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: aspnetapp
spec:
  selector:
    app: aspnetapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - path: /test
        backend:
          serviceName: aspnetapp
          servicePort: 80