Application-gateway-kubernetes-ingress: Duplicate rule error for 2 distinct listeners: 80 and 443

Created on 11 May 2019  ·  5Comments  ·  Source: Azure/application-gateway-kubernetes-ingress

Describe the bug
When I create 2 ingresses, HTTP and HTTPS, using the domains and rules shown below in ingress.yaml, the application-gateway-kubernetes-ingress container reveals many errors related to duplicate rules for a domain. Even if the rules are equal, they are intended to be attached to different listeners (80 and 443). AFAIK, via the Azure portal it's not possible to attach a single rule to 2 distinct listeners - you need two rules with distinct names - but it looks like application-gateway-kubernetes-ingress doesn't handle that.

a) Or am I missing something? Is there another way of doing this?
b) Actually, my end goal is to have the site being server over HTTPS, and have the HTTP listener returning a permanent redirect to the HTTPS address. Is that possible?
c) AFAIK, I still need to declare 2 separate ingresses, each one with its own rules. However I don't think the project supports basic redirect rules right now. Is that right?

To Reproduce
helm install -f appgw.yaml application-gateway-kubernetes-ingress/ingress-azure
kubectl create -f ingress.yaml

appgw.yaml

appgw:
    subscriptionId: <redacted>
    resourceGroup: MC_Kubernetes_KBTCluster_eastus
    name: app-gw-site
kubernetes:
  watchNamespace: default
armAuth:
    type: aadPodIdentity
    identityResourceID: /subscriptions/<redacted>/resourcegroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-appgw-identity
    identityClientID: <redacted>
rbac:
    enabled: true # true/false
aksClusterConfiguration:
    apiServerAddress: <redacted>.hcp.eastus.azmk8s.io

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wordpress-http-only
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - host: dominio.com.br
    http:
      paths:
      - path: /wp-content/*
        backend:
          serviceName: wordpress-default
          servicePort: 80
      - path:
        backend:
          serviceName: wordpress-default
          servicePort: 80
  - host: www.dominio.com.br
    http:
      paths:
      - path: /wp-content/*
        backend:
          serviceName: wordpress-default
          servicePort: 80
      - path:
        backend:
          serviceName: wordpress-default
          servicePort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: wordpress-https-only
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  tls:
    - hosts:
      - dominio.com.br
      - www.dominio.com.br
      secretName: aks-ingress-tls-secret
  rules:
  - host: dominio.com.br
    http:
      paths:
      - path: /wp-content/*
        backend:
          serviceName: wordpress-default
          servicePort: 80
      - path:
        backend:
          serviceName: wordpress-default
          servicePort: 80
  - host: www.dominio.com.br
    http:
      paths:
      - path: /wp-content/*
        backend:
          serviceName: wordpress-default
          servicePort: 80
      - path:
        backend:
          serviceName: wordpress-default
          servicePort: 80

Ingress Controller details

  • Output of kubectl describe pod <ingress controller> .
Name:               wobbly-antelope-ingress-azure-57c6cf9455-t2cr7
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-nodepool1-23041372-vmss000008/10.240.0.5
Start Time:         Fri, 10 May 2019 19:49:56 -0300
Labels:             aadpodidbinding=wobbly-antelope-ingress-azure
                    app=ingress-azure
                    pod-template-hash=57c6cf9455
                    release=wobbly-antelope
Annotations:        <none>
Status:             Running
IP:                 10.244.1.11
Controlled By:      ReplicaSet/wobbly-antelope-ingress-azure-57c6cf9455
Containers:
  ingress-azure:
    Container ID:   docker://12a75da28cc9c7dde5803504738f7b60b3a148d9576ae8c868ca482b1cb2e435
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:0.3.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c110f441f81fc53060d4a88367ca9430c681e9a513cc978d55bed8c9025e9121
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 10 May 2019 19:49:59 -0300
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      wobbly-antelope-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      KUBERNETES_PORT_443_TCP_ADDR:  <redacted>.hcp.eastus.azmk8s.io
      KUBERNETES_PORT:               tcp://<redacted>.hcp.eastus.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:       tcp://<redacted>.hcp.eastus.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:       <redacted>.hcp.eastus.azmk8s.io
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from wobbly-antelope-sa-ingress-azure-token-qbtfn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  wobbly-antelope-sa-ingress-azure-token-qbtfn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  wobbly-antelope-sa-ingress-azure-token-qbtfn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                        Message
  ----    ------     ----  ----                                        -------
  Normal  Scheduled  11m   default-scheduler                           Successfully assigned default/wobbly-antelope-ingress-azure-57c6cf9455-t2cr7 to aks-nodepool1-23041372-vmss000008
  Normal  Pulling    11m   kubelet, aks-nodepool1-23041372-vmss000008  pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:0.3.0"
  Normal  Pulled     11m   kubelet, aks-nodepool1-23041372-vmss000008  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:0.3.0"
  Normal  Created    11m   kubelet, aks-nodepool1-23041372-vmss000008  Created container
  Normal  Started    11m   kubelet, aks-nodepool1-23041372-vmss000008  Started container
  • Output of `kubectl logs . 
ERROR: logging before flag.Parse: I0510 22:49:59.753425       1 main.go:60] Creating authorizer from MSI
ERROR: logging before flag.Parse: I0510 22:50:31.088963       1 context.go:287] k8s context run started
ERROR: logging before flag.Parse: I0510 22:50:31.089006       1 context.go:374] start waiting for initial cache sync
ERROR: logging before flag.Parse: I0510 22:50:31.199300       1 secretstore.go:119] converted secret [default/aks-ingress-tls-secret]
ERROR: logging before flag.Parse: I0510 22:50:31.289391       1 context.go:389] ingress initial sync done
ERROR: logging before flag.Parse: I0510 22:50:31.289432       1 context.go:289] k8s context run finished
ERROR: logging before flag.Parse: I0510 22:50:31.289449       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289469       1 eventqueue.go:119] Processing event begin, time since event generation: 18.5µs
ERROR: logging before flag.Parse: I0510 22:50:31.289477       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:31.289566       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289643       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289655       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289661       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289668       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289675       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.289681       1 eventqueue.go:60] Enqueuing skip(false) item
ERROR: logging before flag.Parse: I0510 22:50:31.356204       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:31.535055       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:31.535086       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:31.535098       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:31.535108       1 eventqueue.go:119] Processing event begin, time since event generation: 245.545894ms
ERROR: logging before flag.Parse: I0510 22:50:31.535114       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:31.585223       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:31.737993       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:31.738024       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:31.738032       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:31.738043       1 eventqueue.go:119] Processing event begin, time since event generation: 448.407282ms
ERROR: logging before flag.Parse: I0510 22:50:31.738050       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:31.792233       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:31.935016       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:31.935061       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:31.935069       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:31.935090       1 eventqueue.go:119] Processing event begin, time since event generation: 645.434339ms
ERROR: logging before flag.Parse: I0510 22:50:31.935096       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:31.983780       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:32.133959       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:32.133990       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:32.133998       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:32.134010       1 eventqueue.go:119] Processing event begin, time since event generation: 844.347307ms
ERROR: logging before flag.Parse: I0510 22:50:32.134016       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:32.187008       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:32.339013       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:32.339044       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:32.339053       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:32.339064       1 eventqueue.go:119] Processing event begin, time since event generation: 1.049395304s
ERROR: logging before flag.Parse: I0510 22:50:32.339072       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:32.389663       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:32.543939       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:32.543986       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:32.543994       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:32.544005       1 eventqueue.go:119] Processing event begin, time since event generation: 1.254329501s
ERROR: logging before flag.Parse: I0510 22:50:32.544012       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:32.595995       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:32.742997       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-443-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:32.743033       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:32.743042       1 eventqueue.go:126] Processing event failed
ERROR: logging before flag.Parse: I0510 22:50:32.743053       1 eventqueue.go:119] Processing event begin, time since event generation: 1.453370269s
ERROR: logging before flag.Parse: I0510 22:50:32.743059       1 controller.go:50] controller.processEvent called with type k8scontext.Event
ERROR: logging before flag.Parse: I0510 22:50:32.796174       1 controller.go:98] ~~~~~~~~ ↓ ApplicationGateway deployment ↓ ~~~~~~~~
ERROR: logging before flag.Parse: W0510 22:50:32.952231       1 controller.go:105] unable to send CreateOrUpdate request, error [network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /teste/* is already specified in another PathRule in UrlPathMap /subscriptions/<redacted>/resourceGroups/MC_Kubernetes_KBTCluster_eastus/providers/Microsoft.Network/applicationGateways/app-gw-site/urlPathMaps/k8s-ag-ingress-dominio.com.br-80-url." Details=[]]
ERROR: logging before flag.Parse: I0510 22:50:32.952260       1 controller.go:106] ~~~~~~~~ ↑ ApplicationGateway deployment ↑ ~~~~~~~~
ERROR: logging before flag.Parse: I0510 22:50:32.952267       1 eventqueue.go:126] Processing event failed
  • Any Azure support tickets associated with this issue.
    None.
Bug
Source
picture jweyrich

All 5 comments

@jweyrich we are working on an ssl-redirect annotation that will allow you to specify and ingress resource with TLS spec and ssl-redirect turned on. The annotation would than tell ingress controller to setup the HTTP redirect automatically for the HTTP end point.
Here is the WIP PR: #132

asridharan picture asridharan on 11 May 2019
👍1

Could we make it leave manually created rules and listeners alone? I mean, then I could manually configure the redirect for now. A redeploy currently deletes and recreates all listeners and rules.

jweyrich picture jweyrich on 11 May 2019

@jweyrich unfortuantely its non-trivial to do this for listeners and rules. So the ingress controllers ends up overwriting any listeners and rules that was manually added. We will look into a more surgical approach at a later date.

Could we make it leave manually created rules and listeners alone? I mean, then I could manually configure the redirect for now. A new redeploy currently deletes and recreates all listeners and rules.

asridharan picture asridharan on 11 May 2019

@jweyrich We have SSL Redirect annotation - this is available in the latest version 0.7.1. Here is documentation on how to enable it: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/76064d31bc4beb29e574e43378591f472a347a4a/docs/annotations.md#ssl-redirect

We are also working on a feature that allows the ingress controller to leave manually created rules and listeners alone as you proposed - will keep you updated on this. Here is the proposal we are implementing: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/proposals/config_ownership_crd.md (This is also tracked with the following issue: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/223)

draychev picture draychev on 12 Jul 2019
❤1

Awesome @draychev. Will give it a try. Thank you for the update!

jweyrich picture jweyrich on 15 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

abenck picture abenck  ·  5Comments
pbloigu picture pbloigu  ·  6Comments
rhollins picture rhollins  ·  3Comments
marc-mueller picture marc-mueller  ·  4Comments
Jawshua picture Jawshua  ·  4Comments