Application-gateway-kubernetes-ingress: Support services of type ExternalName

[jasonchester]

Add support for App Gateway proxying of ExternalName services.

https://kubernetes.io/docs/concepts/services-networking/service/#externalname
https://github.com/kubernetes/ingress-nginx/pull/629#issue-116679227

Describe the solution you'd like
External name services defined in k8s should be exposed through app gateway when configured with app gateway ingress annotations.

[asridharan]

@jasonchester could you elaborate a bit more on the ask here? The links linked to ingress-nginx are extremely cryptic and look very hacky. Would like to see a proposal in the AG context here rather than copying whatever nginx does.

I am trying to see a use-case for this. The whole idea of ExternalName was to expose the external service (running outside k8s) to internal pods running in the k8s cluster. So wondering how AG will even come in play in a such a scenario. A description of the use-case you had in mind would help here.

[jasonchester]

We have an multitenant (100's of tenant specific hostnames) application that we are in the process of modernizing. The application consists of a an ASP.NET forms monolith and new services we are developing in dotnet core that will run in Kubernetes.

• The aspnet forms application running on IaaS Windows Virtual Machines and will listen to the be the default route.
  -- https://<tenanturl>.company.co/*
• New services build in dotnet core will be hosted in cluster and published on the same tenanturls.
  -- https://<tenanturl>.company.co/api/service/v1/*

The use case we are looking to enable is:

1. Create an external service for the aspnet forms app backend. This is likely a load balancer fronting a handful of IaaS Web Servers.
2. Configure ingress for tenants exposing both the legacy and kubernetes services on the same app gateway.

here's an example of what resources in kubernetes might look like for this scenario... not tested or complete but for example.

apiVersion: v1
kind: Service
metadata:
  name: svc-legacy-forms-ext
spec:
  type: ExternalName
  externalName: lb.formsapp.company.com

---

apiVersion: v1
kind: Service
metadata:
  name: svc-checkout-api-v1
spec:
  selector:
    app: checkout-api-v1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 9376

---

apiVersion: v1
kind: Service
metadata:
  name: svc-notifications-api-v1
spec:
  selector:
    app: notifications-api-v1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 9376

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hybrid-app-ingress
spec:
  tls:
  - hosts:
    - tenantA.company.com
    - tenantB.company.com
    secretName: wildcard-company-com-tls
  rules:
  - host: tenantA.company.com
    http:
      paths:
      - path: /
        backend:
          serviceName: svc-legacy-forms-ext  #external service
          servicePort: 80
      - path: /api/notifications/v1/
        backend:
          serviceName: svc-notifications-api-v1 #kubernetes service
          servicePort: 80
      - path: /api/checkout/v1/
        backend:
          serviceName: svc-checkout-api-v1 #kubernetes service
          servicePort: 80
  - host: tenantB.company.com
    http:
      paths:
      - path: /
        backend:
          serviceName: svc-legacy-forms-ext  #external service
          servicePort: 80
      - path: /api/notifications/v1/
        backend:
          serviceName: svc-notifications-api-v1 #kubernetes service
          servicePort: 80
      - path: /api/checkout/v1/
        backend:
          serviceName: svc-checkout-api-v1 #kubernetes service
          servicePort: 80

[jasonchester]

@asridharan is additional clarification needed or can the needs-clarity tag be removed.

thanks

[shinji]

@jasonchester
@draychev

We are temporarily implementing the following workaround using a TCP Proxy, it is a bad practice, but it can help with the tests and understand what we want with the use case:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-forms-ext
  labels:
    app: legacy-forms-ext
spec:
  replicas: 4
  selector:
    matchLabels:
      app: legacy-forms-ext
  template:
    metadata:
      labels:
        app: legacy-forms-ext
    spec:
      containers:
        - name: legacy-forms-ext
          imagePullPolicy: Always
          image: tecnativa/tcp-proxy:latest
          env:
            - name: LISTEN
              value: ":443"
            - name: TALK
              value: lb.formsapp.company.com
          ports:
            - containerPort: 443
          livenessProbe:
            httpGet:
              path: /
              port: 443
              host: lb.formsapp.company.com
              scheme: HTTPS
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /
              port: 443
              host: lb.formsapp.company.com
              scheme: HTTPS
            periodSeconds: 10



---
apiVersion: v1
kind: Service
metadata:
  name: svc-legacy-forms-ext
spec:
  selector:
    app: legacy-forms-ext
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443



---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: legacy-app-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-hostname: lb.formsapp.company.com
    appgw.ingress.kubernetes.io/backend-protocol: https
spec:
  tls:
  - hosts:
    - tenantA.company.com
    secretName: wildcard-company-com-tls
  rules:
  - host: tenantA.company.com
    http:
      paths:
      - path: /
        backend:
          serviceName: svc-legacy-forms-ext
          servicePort: 443

[jasonchester]

@shinji glad to see some activity on this request...

The lack of any progress on this issue from the @azure team which hasn't even removed the needs-clarity tag has caused me to lose confidence in app-gateway as a serious solution for our ingress needs. We're using the nginx ingress behind a internal azure LoadBalancer service and other than the same namespace service limitations (which to be fair are by design) it has been doing great for us.

[tplive]

We are in the same situation as @jasonchester where we are migrating a system partly from a legacy system, but need to route certain paths to the legacy system for now. We are solving this by letting AKS and AGIC handle configuring of AG for the moved resources, and then setting up another AG "in front" of that to handle the external legacy routing. It would be of great benefit if AGIC would configure ExternalName and ExternalIP resources for us, as we are looking at having to repeat this extra configuration for multiple clusters for both test and prod environments. So please look into providing this functionality.

[andrejbrummelhuis]

also desperately missing this feature as we have the same requirements as mentioned by @jasonchester