Application-gateway-kubernetes-ingress: AGIC keeps saying backend pool is unhealthy even if pods are up and running

Created on 25 Aug 2020  路  16Comments  路  Source: Azure/application-gateway-kubernetes-ingress

Describe the bug
AKS went down and after performing some node replace operations, the nodes\\pods came up fine by AGS is still reporting backend pool errors and not forwarding any requests to backend. Appgateway and AKS are within same vNET.

To Reproduce
AGIC backend health reporting errors
image

Pod is showing 200 for the probe url
image

probe setup:
image

Nothing changed from AKS or AppGateway side.. AKS nodes went down and came back up after some time but backend pool keep showing as unhealthy..

Ingress logs:
image

@akshaysngupta
this is a prod issue.. any quick response is appreciated.

Ingress Controller details

  • Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
  • Output of `kubectl logs .
  • Any Azure support tickets associated with this issue.
    120082524000376
    120082424006138
    120082424004967

.

picture nreddipalle

Most helpful comment

@nreddipalle

  1. The log line is just for information purpose. I think we can improve the logging in that area.
  2. The second one is known issue with portal. We have a new feature rolling out for HTTP Listeners property in AppGW which we have started using in AGIC. Portal changes are going to arrive soon.
picture akshaysngupta on 31 Aug 2020
👍2

All 16 comments

Hi @nreddipalle , would you be able to share with us a sanitized version of your Ingress(yaml content)?

3quanfeng picture 3quanfeng on 25 Aug 2020

@3quanfeng how can I retrieve it?

nreddipalle picture nreddipalle on 25 Aug 2020

kubectl get ingress -n your_namespace -o yaml
Is the issue consistent?

3quanfeng picture 3quanfeng on 25 Aug 2020

Yes. Currently it still exists and no communication is going on between AG and AKS.. backend health keeps showing as unhealthy even when I am able to browse fine from one of the pod internally.

bash-5.0$ curl newsite.senseilabs.com/manifest.json
<html>
<head><title>522 Origin Connection Time-out</title></head>
<body bgcolor="white">
<center><h1>522 Origin Connection Time-out</h1></center>
<hr><center>cloudflare-nginx</center>
</body>
</html>
bash-5.0$ curl 10.200.49.101:85/manifest.json
{"build-date":"2020-08-18T03:49:52.4376324Z","vcs-sha":"88d101138e6e9cc859cb8018c0e8993a6d3f801a","vcs-tag":"(no branch)","previous-vcs-sha":null}bash-5.0$
nreddipalle picture nreddipalle on 25 Aug 2020

here is the ingress yaml:

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      appgw.ingress.kubernetes.io/appgw-ssl-certificate: senseilabsappservcert
      appgw.ingress.kubernetes.io/connection-draining: "true"
      appgw.ingress.kubernetes.io/connection-draining-timeout: "300"
      appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
      appgw.ingress.kubernetes.io/request-timeout: "300"
      appgw.ingress.kubernetes.io/ssl-redirect: "true"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{"appgw.ingress.kubernetes.io/appgw-ssl-certificate":"appservcert","appgw.ingress.kubernetes.io/connection-draining":"true","appgw.ingress.kubernetes.io/connection-draining-timeout":"300","appgw.ingress.kubernetes.io/cookie-based-affinity":"true","appgw.ingress.kubernetes.io/request-timeout":"300","appgw.ingress.kubernetes.io/ssl-redirect":"true","kubernetes.io/ingress.class":"azure/application-gateway"},"labels":{"app.kubernetes.io/instance":"newsite-prod","app.kubernetes.io/managed-by":"Tiller","app.kubernetes.io/name":"sensei-newsite","helm.sh/chart":"20200702-13137"},"name":"sensei-newsite","namespace":"newsite"},"spec":{"rules":[{"host":"newsite.senseilabs.com","http":{"paths":[{"backend":{"serviceName":"sensei-newsite","servicePort":85},"path":"/"},{"backend":{"serviceName":"sensei-newsite-conductor","servicePort":85},"path":"/conductor/*"},{"backend":{"serviceName":"sensei-newsite-conductor","servicePort":85},"path":"/api/conductor/*"},{"backend":{"serviceName":"sensei-newsite-academy","servicePort":85},"path":"/academy/*"},{"backend":{"serviceName":"sensei-newsite-academy","servicePort":85},"path":"/api/academy/*"}]}}]}}
      kubernetes.io/ingress.class: azure/application-gateway
    creationTimestamp: "2020-06-04T21:18:09Z"
    generation: 11
    labels:
      app.kubernetes.io/instance: newsite-prod
      app.kubernetes.io/managed-by: Tiller
      app.kubernetes.io/name: sensei-newsite
      helm.sh/chart: senseios-20200817-22843
    name: sensei-newsite
    namespace: newsite
    resourceVersion: "15224933"
    selfLink: /apis/extensions/v1beta1/namespaces/newsite/ingresses/sensei-newsite
    uid: 3a627ed3-543a-42e0-b913-136f2a605b7e
  spec:
    rules:
    - host: newsite.senseilabs.com
      http:
        paths:
        - backend:
            serviceName: sensei-newsite
            servicePort: 85
          path: /
  status:
    loadBalancer:
      ingress:
      - ip: 
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
nreddipalle picture nreddipalle on 25 Aug 2020

FYI.. this was working perfectly fine for the past couple of months... today due to a node issue AKS went down and once it came back up fine, AG still showing backend as unhealthy and not serving traffic..

nreddipalle picture nreddipalle on 25 Aug 2020

one more thing is since you replaced your AKS nodes, do you have any node affinity or node selector defined in your pod yaml or deployment yaml? are all of your service pods running fine(check by kubectl get pods -n your_namespace) after you replace aks nodes?

3quanfeng picture 3quanfeng on 25 Aug 2020

Our node selector is mainly for OS which is windows and spot instances. Yes all the service pods are running fine.. we have a failover to switch to ngnix and some sites that switched to ngnix are loading fine.. but some sites we left on AGIC for further troubleshooting and still not up yet..

nreddipalle picture nreddipalle on 25 Aug 2020

Hi @nreddipalle , since new nodes added, could you try to update the annotation into "false"?
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
You can enable the affinity cookie later once the probe works

3quanfeng picture 3quanfeng on 25 Aug 2020

No difference. still same errors.

nreddipalle picture nreddipalle on 25 Aug 2020

any update on this??

nreddipalle picture nreddipalle on 26 Aug 2020

@nreddipalle looks like something broken in network connectivity b/w AppGW and Pod.
As a test, can you see if you are able to reach your Pods from the outside cluster, maybe using a VM ? A more appropriate test would be test from a different subnet.

akshaysngupta picture akshaysngupta on 26 Aug 2020

Thanks @akshaysngupta. YES. It looks like some network connectivity issue between App GW and Pod. We are able to fallback to use ngnix controller meanwhile... We tried tel-netting from a VM which is in same subnet and unable to telnet to pod port... so something weird going on between node-pod connectivity.. by doing a network trace, AG is sending requests to node but node is not responding back.... working with microsoft support for further troubleshooting.

nreddipalle picture nreddipalle on 26 Aug 2020
👍1

This is confirmed to be network subnet-subnet communication issue. Before closing of this issue, I am noticing few errors which I would like to understand whether to ignore or not.
Seeing the below warning in ingress pod logs
1 redirects.go:44] Created redirection configuration sslr-fl-deca3d57d408a18bc8a79cd269957987 for ([newsite.senseilabs.com ],443); not yet linked to a routing rule

App Gateway Listeners Tab:
image

image

The site(s) are loading fine.. Does the listener(s) suppose to configure as multisite instead of basic site??

nreddipalle picture nreddipalle on 31 Aug 2020

@nreddipalle

  1. The log line is just for information purpose. I think we can improve the logging in that area.
  2. The second one is known issue with portal. We have a new feature rolling out for HTTP Listeners property in AppGW which we have started using in AGIC. Portal changes are going to arrive soon.
akshaysngupta picture akshaysngupta on 31 Aug 2020
👍2

Thanks @akshaysngupta .

nreddipalle picture nreddipalle on 31 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

samrobillard picture samrobillard  路  3Comments
galvesribeiro picture galvesribeiro  路  6Comments
pbalexlear picture pbalexlear  路  3Comments
MarkDNL picture MarkDNL  路  5Comments
sowsan picture sowsan  路  6Comments