Application-gateway-kubernetes-ingress: How to configure ingress backend

Created on 2 Oct 2019  路  7Comments  路  Source: Azure/application-gateway-kubernetes-ingress

Describe the bug
Following this article I successfully setup AKS cluster with Applciation Gateway. I can sucessfully browse the public endpoint http://Public-IP and it lands me to the sample asp.net webapp home page.

Next, I have deployed guestbook and it is deployed successfully. I can see two ingress in my cluster:

NAME        HOSTS   ADDRESS         PORTS   AGE
aspnetapp   *       20.36.242.135   80      105m
guestbook   *       20.36.242.135   80      23m

The challenge is that I cannot seem to setup the path in YAML (see below) in a way that guestbook application can be accessed using http://20.36.242.135/guestbook while sample asp.net application can be accessed via http://20.36.242.135. Indepedently (if only one of them exposed via ingress). does work fine.

I want to confim if this is even supported or not.

guestbook-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: guestbook
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - path: /guestbook/
        backend:
          serviceName: frontend
          servicePort: 80

aspnet-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: aspnetapp
          servicePort: 80

To Reproduce
As I decribed above.

Ingress Controller details

Name:           odd-opossum-ingress-azure-6c55b6f549-htdrw
Namespace:      default
Priority:       0
Node:           aks-agentpool-35064155-0/10.0.0.35
Start Time:     Wed, 02 Oct 2019 00:26:15 +0000
Labels:         aadpodidbinding=odd-opossum-ingress-azure
                app=ingress-azure
                pod-template-hash=6c55b6f549
                release=odd-opossum
Annotations:    <none>
Status:         Running
IP:             10.0.0.57
IPs:            <none>
Controlled By:  ReplicaSet/odd-opossum-ingress-azure-6c55b6f549
Containers:
  ingress-azure:
    Container ID:   docker://1b24dce25be97ff3c0dcd60524bf321abf7759484f510ed5b6366ab360a3c231
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:0.9.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:ba9cb3968e02cbc61f313865c3bc7cbddf0d3e8a0e62bc1f502a67168e5e311c
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 02 Oct 2019 00:26:34 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      odd-opossum-cm-ingress-azure  ConfigMap  Optional: false
    Environment:                    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from odd-opossum-sa-ingress-azure-token-gg8t2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  odd-opossum-sa-ingress-azure-token-gg8t2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  odd-opossum-sa-ingress-azure-token-gg8t2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

I1002 00:26:34.569672       1 main.go:192] Creating authorizer from Azure Managed Service Identity
I1002 00:26:35.590192       1 main.go:117] Ingress Controller will observe all namespaces.
I1002 00:26:35.622503       1 context.go:122] k8s context run started
I1002 00:26:35.622550       1 context.go:161] Waiting for initial cache sync
I1002 00:26:35.723298       1 context.go:169] Initial cache sync done
I1002 00:26:35.723342       1 context.go:170] k8s context run finished
I1002 00:26:35.723423       1 worker.go:33] Worker started
I1002 00:26:35.723453       1 main.go:140] Starting Health Probe Server on :8123
E1002 00:26:35.819426       1 process.go:87] no Ingress in the pruned Ingress list. Please check Ingress events to get more information
E1002 00:29:29.691087       1 process.go:87] no Ingress in the pruned Ingress list. Please check Ingress events to get more information
I1002 00:29:29.809784       1 process.go:133] BEGIN AppGateway deployment
I1002 00:29:50.180625       1 process.go:156] Applied App Gateway config in 20.370754302s
I1002 00:29:50.180665       1 process.go:165] cache: Updated with latest applied config.
I1002 00:29:50.199593       1 process.go:171] END AppGateway deployment
I1002 00:29:50.269462       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:29:52.177859       1 process.go:133] BEGIN AppGateway deployment
I1002 00:30:12.461631       1 process.go:156] Applied App Gateway config in 20.283733843s
I1002 00:30:12.461672       1 process.go:165] cache: Updated with latest applied config.
I1002 00:30:12.462152       1 process.go:171] END AppGateway deployment
I1002 00:30:12.525555       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:43:16.305072       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:43:16.424162       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:43:16.727013       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:46:06.860200       1 process.go:133] BEGIN AppGateway deployment
I1002 00:46:27.160928       1 process.go:156] Applied App Gateway config in 20.300681364s
I1002 00:46:27.160967       1 process.go:165] cache: Updated with latest applied config.
I1002 00:46:27.182210       1 process.go:171] END AppGateway deployment
I1002 00:46:27.241079       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:52:04.299153       1 process.go:133] BEGIN AppGateway deployment
I1002 00:52:24.722395       1 process.go:156] Applied App Gateway config in 20.423204771s
I1002 00:52:24.722429       1 process.go:165] cache: Updated with latest applied config.
I1002 00:52:24.736149       1 process.go:171] END AppGateway deployment
I1002 00:52:24.736267       1 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"guestbook", UID:"04d2c1b8-e4ae-11e9-9631-aa2a24c4815d", APIVersion:"extensions/v1beta1", ResourceVersion:"5634", FieldPath:""}): type: 'Warning' reason: 'UnableToUpdateIngressStatus' Unable to get ingress default/guestbook
I1002 00:54:01.372467       1 process.go:133] BEGIN AppGateway deployment
I1002 00:54:21.711019       1 process.go:156] Applied App Gateway config in 20.338468552s
I1002 00:54:21.711062       1 process.go:165] cache: Updated with latest applied config.
I1002 00:54:21.730521       1 process.go:171] END AppGateway deployment
I1002 00:54:21.800121       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 00:58:59.921517       1 process.go:133] BEGIN AppGateway deployment
I1002 00:59:20.240974       1 process.go:156] Applied App Gateway config in 20.319412891s
I1002 00:59:20.241006       1 process.go:165] cache: Updated with latest applied config.
I1002 00:59:20.257552       1 process.go:171] END AppGateway deployment
I1002 00:59:20.306354       1 process.go:133] BEGIN AppGateway deployment
I1002 00:59:40.593508       1 process.go:156] Applied App Gateway config in 20.287119455s
I1002 00:59:40.593547       1 process.go:165] cache: Updated with latest applied config.
I1002 00:59:40.594194       1 process.go:171] END AppGateway deployment
I1002 01:04:41.536259       1 process.go:133] BEGIN AppGateway deployment
I1002 01:05:01.816096       1 process.go:156] Applied App Gateway config in 20.279791908s
I1002 01:05:01.816211       1 process.go:165] cache: Updated with latest applied config.
I1002 01:05:01.828522       1 process.go:171] END AppGateway deployment
I1002 01:05:01.828624       1 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"guestbook", UID:"d7ebe733-e4af-11e9-9631-aa2a24c4815d", APIVersion:"extensions/v1beta1", ResourceVersion:"6823", FieldPath:""}): type: 'Warning' reason: 'UnableToUpdateIngressStatus' Unable to get ingress default/guestbook
I1002 01:06:43.116933       1 process.go:133] BEGIN AppGateway deployment
I1002 01:07:03.498172       1 process.go:156] Applied App Gateway config in 20.381203313s
I1002 01:07:03.498207       1 process.go:165] cache: Updated with latest applied config.
I1002 01:07:03.509816       1 process.go:171] END AppGateway deployment
I1002 01:07:03.570338       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 01:12:31.266156       1 process.go:133] BEGIN AppGateway deployment
I1002 01:13:41.705165       1 process.go:156] Applied App Gateway config in 1m10.438972645s
I1002 01:13:41.705202       1 process.go:165] cache: Updated with latest applied config.
I1002 01:13:41.719621       1 process.go:171] END AppGateway deployment
I1002 01:13:41.772620       1 process.go:133] BEGIN AppGateway deployment
I1002 01:14:02.112283       1 process.go:156] Applied App Gateway config in 20.339617599s
I1002 01:14:02.112323       1 process.go:165] cache: Updated with latest applied config.
I1002 01:14:02.112968       1 process.go:171] END AppGateway deployment
I1002 01:48:12.617623       1 process.go:133] BEGIN AppGateway deployment
I1002 01:48:33.256647       1 process.go:156] Applied App Gateway config in 20.638991411s
I1002 01:48:33.256679       1 process.go:165] cache: Updated with latest applied config.
I1002 01:48:33.267465       1 process.go:171] END AppGateway deployment
I1002 01:48:33.267555       1 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"guestbook", UID:"d9c46a96-e4b1-11e9-9631-aa2a24c4815d", APIVersion:"extensions/v1beta1", ResourceVersion:"10903", FieldPath:""}): type: 'Warning' reason: 'UnableToUpdateIngressStatus' Unable to get ingress default/guestbook
I1002 01:49:51.238027       1 process.go:133] BEGIN AppGateway deployment
I1002 01:50:31.592012       1 process.go:156] Applied App Gateway config in 40.353950679s
I1002 01:50:31.592043       1 process.go:165] cache: Updated with latest applied config.
I1002 01:50:31.614804       1 process.go:171] END AppGateway deployment
I1002 01:51:52.107732       1 process.go:133] BEGIN AppGateway deployment
I1002 01:52:12.453765       1 process.go:156] Applied App Gateway config in 20.345989275s
I1002 01:52:12.453810       1 process.go:165] cache: Updated with latest applied config.
I1002 01:52:12.471867       1 process.go:171] END AppGateway deployment
I1002 01:52:12.544344       1 process.go:129] cache: Config has NOT changed! No need to connect to ARM.
I1002 01:56:19.539855       1 process.go:133] BEGIN AppGateway deployment
I1002 01:57:30.074654       1 process.go:156] Applied App Gateway config in 1m10.534754999s
I1002 01:57:30.074692       1 process.go:165] cache: Updated with latest applied config.
I1002 01:57:30.082682       1 process.go:171] END AppGateway deployment
Bug
Source
picture rbinrais

Most helpful comment

@rbinrais @ahojman with release [0.10.0-rc2](https://github.com/Azure/application-gateway-kubernetes-ingress/releases/tag/0.10.0-rc2] we addressed the issue here.

Would love to get your feedback and find out whether this version fixes the bug for you.

Will keep the issue open until we hear from you.

picture draychev on 15 Oct 2019
🎉1 😄1

All 7 comments

Hi @rbinrais,

I want to confirm if this is even supported or not.

Yes it is supported! You can create 2 separate ingress resources.

I suspect that the Backend health for the pods under /guestbook/ might not be good:

This is what I see on an installation of the same Ingress and apps on my cluster:

image

To fix that - I used the backend prefix annotation -- appgw.ingress.kubernetes.io/backend-path-prefix: "/" -- which keeps the guetsbook app on /guestbook/ but rewrites the backend path to /. With that - your pod's will stop reporting 404.

image

Would you be so kind to add the appgw.ingress.kubernetes.io/backend-path-prefix: "/" annotation to both Ingress resources and report back?

draychev picture draychev on 3 Oct 2019

@rbinrais I do see that we may have an issue with the exact scenario you described - two Ingresses, one with / and the other with /guestbook/. That results in the / being forwarded to the incorrect backend pool:

image

The wrong backend pool is being allocated as the default backend pool for the given rule. This is most likely due to the order in which we process the Ingress resources.

We have a workaround - if we combine the two Ingress resources like this: 

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: guestbook
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /guestbook/
        backend:
          serviceName: frontend
          servicePort: 80
      - path: /
        backend:
          serviceName: aspnetapp
          servicePort: 80

AGIC creates the correct App Gateway config:

image

In the screenshot above we have the right backend pool assigned as the default one; / traffic is correctly forwarded to aspnetapp service.

I am going to tag this PR with bug and we'll take a stab at fixing this as soon as possible.

Another workaround would be to use /aspnetapp/ in the first Ingress and /guetsbook/ in the second.

Would these workarounds work for you, @rbinrais ?

draychev picture draychev on 3 Oct 2019

Let me try to apply these updates @draychev and see how it goes. Thanks!

rbinrais picture rbinrais on 4 Oct 2019

Hello all,

I'm having the same behaviour with an AKS holding 50+ Ingresses on the same namespace (which is not default).
AGIC automation is catching OK different paths and adjusting Backend Pools and Rules as it should on Azure Portal; but failing to do so, when declaring simple / for the main Ingress path, which is serving my home page.

ahojman picture ahojman on 7 Oct 2019

@rbinrais @ahojman with release [0.10.0-rc2](https://github.com/Azure/application-gateway-kubernetes-ingress/releases/tag/0.10.0-rc2] we addressed the issue here.

Would love to get your feedback and find out whether this version fixes the bug for you.

Will keep the issue open until we hear from you.

draychev picture draychev on 15 Oct 2019
🎉1 😄1

So far, so good. No more problems observed.

ahojman picture ahojman on 30 Oct 2019

Hi, hoping someone gets to see this issue
I think I encountered the situation now and I get a rule defined on a default pool which is not the required one

My yaml looks like this, where I tried creating a stub rule thinking about the order suggestion raised here
What else can you suggest I should check?

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: appgw-ingress
  namespace: application
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt-app-gateway
    appgw.ingress.kubernetes.io/health-probe-path: "/some/health"
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - my-appgw.northeurope.cloudapp.azure.com
    secretName: my-appgw-tls-secret
  rules:
  - host: my-appgw.northeurope.cloudapp.azure.com
    http:
      paths:
      - path: /defaultstub/
        backend:
          serviceName: api-gateway-service
          servicePort: 80
      - path: /(.*)
        backend:
          serviceName: api-gateway-service
          servicePort: 80
assaf-grth picture assaf-grth on 12 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jasonchester picture jasonchester  路  7Comments
gitflo1 picture gitflo1  路  3Comments
Joon picture Joon  路  3Comments
Jawshua picture Jawshua  路  4Comments
jweyrich picture jweyrich  路  5Comments