External-dns: [Openshift] external-dns is adding CNAME entries instead of A records for routes

[ahmedwaleedmalik]

What happened:

We deployed external-dns on an Openshift cluster by following the guide.
The issue with the current implementation is that it's only aware of the ROUTER_CANONICAL_HOSTNAME lets say it's value is example.com. And it adds CNAME entry that point the host name of the route(app.example.com) to the router canonical hostname i.e. app.example.com to example.com.

This setup requires the LB IP to be added manually which in my opinion serves as a major block in terms of automation. If the wild card entry is already there i don't see the purpose of adding entries for subdomains.

What you expected to happen:

An A record for the route that points to the LB IP

How to reproduce it (as minimally and precisely as possible):

1. Deploy external-dns on openshift using this guide.
2. Create a new router that serves a specific domain with endpointPublishingStrategy set to loadbalancer.
3. Create a route for that router.

Anything else we need to know?:

In OCP 4.x the routers are managed by openshift-ingress-operator. Routers are created by configuring Ingress Controller resources and the hostname is the value that is used as ROUTER_CANONICAL_HOSTNAME. In our case we define a new ingress controller and set it's endpointPublishingStrategy to loadbalancer. Which creates a loadbalancer svc for each router. You cannot manually override ROUTER_CANONICAL_HOSTNAME anymore since it's managed by ingress operator now.

When we create a route that this router is supposed to serve, external-dns takes the value of host from the route and the routerCanonicalHostname and adds an entry i.e. app.example.com to example.com. Which does it's job but still doesn't provide full automation.

Annotating with external-dns target annotation may be possible since it still involves manually retrieving the IP and adding annotations on routes. We'd definitely like to automate and remove the manual process of adding the first entry.

What would you like to be added:

External-dns to be aware of the loadbalancer that would serve a router. For the route shard that a particular router serves, it should add A record based on the LB IP instead of CNAME records that just point to the ROUTER_CANONICAL_HOSTNAME

Why is this needed:

For end-to-end automation this is required, since manually adding the DNS entry for the loadbalancer IP somehow restricts the usage of external-dns. Also, if the wildcard entry is already there, adding entries for subdomain might be redundant.

Environment:

• External-DNS version (use external-dns --version): Built an image from the master branch since these changes haven't been released yet. @jgrumboe had an image pushed at porscheinformatik/external-dns:latest
• DNS provider: AWS
• Others: Openshift OCP v4.4

[ahmedwaleedmalik]

@jgrumboe I opened this issue based on our discussion. Let me know if it's coherent enough and if any additional details are required regarding this.

[jgrumboe]

@ahmedwaleedmalik so far I do understand the use case. Since I'm working with an onpremise OCP cluster I'm not quite familiar with AWS and OCP integration.
Could you share an "oc describe route" output of your route with endpointtype loadbalancer?

[ahmedwaleedmalik]

DNS provider wouldn't be an issue in this case. But as far as i remember our conversation, you were on OCP v3.x right ? Things have changed majorly in OCP v4.x since everything is managed by Openshift Ingress Operator now.

Anyways, route doesn't have endpointtype set to loadbalancer. It's the router that has endpointPublishingStrategy set to loadbalancer:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: custom-domain-router
  namespace: openshift-ingress-operator
spec:
  domain: app.example.com
  endpointPublishingStrategy:
    loadBalancer:
      scope: External
    type: LoadBalancerService
  replicas: 1
  routeSelector:
    matchLabels:
      router: custom-domain-router

This would create a new router and provision a LoadBalancerService and a load balancer. That load balancer would then serve as an entry point from all the ingress traffic directed towards that router.
Just create a new openshift route and add the label router: custom-domain-router. The router will then serve that particular route.

[ahmedwaleedmalik]

I'd like to start working on this and adding the required support in external-dns. Let me know if any additional details are required from my side, or if there is anything regarding this particular issue that I should know beforehand.

[jgrumboe]

Thanks. Right now the openshift-route source is only working with the route resource itself. Mainly with its annotations and status fields. Could you share one route example (oc describe route) after the router picked it up and it's accessable?

[ahmedwaleedmalik]

Yes sure thing. I deployed a new router and used a sample app for testing the routing.

Output of oc describe route frontend:

Name:           frontend
Namespace:      route-demo
Created:        2 minutes ago
Labels:         app=ruby-hello-world
            app.kubernetes.io/component=ruby-hello-world
            app.kubernetes.io/instance=ruby-hello-world
            router=dev-router
Annotations:        <none>
Requested Host:     app.dev.stakater.com
              exposed on router dev-router (host dev.stakater.com) about a minute ago
Path:           <none>
TLS Termination:    edge
Insecure Policy:    Redirect
Endpoint Port:      8080-tcp

Service:    ruby-hello-world
Weight:     100 (100%)
Endpoints:  172.30.217.167:8080

Output of oc get route frontend -oyaml:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  creationTimestamp: "2020-05-28T07:59:32Z"
  labels:
    app: ruby-hello-world
    app.kubernetes.io/component: ruby-hello-world
    app.kubernetes.io/instance: ruby-hello-world
    router: dev-router
  name: frontend
  namespace: route-demo
  resourceVersion: "25484476"
  selfLink: /apis/route.openshift.io/v1/namespaces/route-demo/routes/frontend
  uid: 8a6e2408-10e6-4362-97eb-9e269abb727c
spec:
  host: app.dev.stakater.com
  port:
    targetPort: 8080-tcp
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: edge
  to:
    kind: Service
    name: ruby-hello-world
    weight: 100
  wildcardPolicy: None
status:
  ingress:
  - conditions:
    - lastTransitionTime: "2020-05-28T08:00:07Z"
      status: "True"
      type: Admitted
    host: app.dev.stakater.com
    routerCanonicalHostname: dev.stakater.com
    routerName: dev-router
    wildcardPolicy: None

Currently, external-dns would just add a CNAME entry of app.dev.stakater.com pointing to dev.stakater.com.

[jgrumboe]

And the DNS entry for dev.stakater.com is pointing where and who manages it?
I would suspect the Ingresscontroller/operator to do this.

[ahmedwaleedmalik]

When we create IngressController it tries to add the wild card DNS entry *.dev.stakater.com by itself, currently it supports AWS, Azure, GKE. But the issue here is that the automated DNS records that would be added for routers are restricted to the cluster wide hosted zones that are defined in dns.config.openshift.io/cluster, that corresponds to the cluster's base domain.

So, when we provision a cluster with the base domain openshift.customer.com it would create a corresponding hosted zone. On adding a secondary router it tries to create a DNS entry for lets say backend.customer.com in a hosted zone that serves openshift.customer.com , which fails as expected.

Also, this is not the right approach since it makes DNS cloud dependent. That is the part where external-dns fits it's purpose. We can have DNS independent of the infrastructure provider.

[jgrumboe]

Openshift 4.x is completly new for me and I never used it, so I'm just thinking loud here:
https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L263 states that you end up with an ingresscontroller deployed by an operator which has a service-type Loadbalancer (like this one https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer).

So according to your ingresscontroller yaml (https://github.com/kubernetes-sigs/external-dns/issues/1605#issuecomment-635128298):
Is it true, you have a service called something like "custom-domain-router" in the namespace openshift-ingress-operator?
This object should held your external Loadbalancer IP (provisioned by AWS) in status.loadBalancer.ingress.ip, right?

So we would need somehow lookup this value of each router corresponding to the route object.
I hope I got it right.

[jgrumboe]

But regarding the topic of wildcard DNS and multiple/sharded routers: have you tried asking how to solve the DNS "overlap" problem at the #openshift-users channel at k8s Slack (https://kubernetes.slack.com/archives/C6AD6JM17)?

[ahmedwaleedmalik]

Hi @jgrumboe,
Sorry, I got caught up with a lot of work at my end and didn't get time to look at this. Will start working on this probably around this weekend.
Yes, that's exactly what i am looking to implement. Regarding DNS "overlap", that is something that will be handled in openshift API. An additional field at https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L265 for ManagedDNS and corresponding handling in IngressController CR should do the trick. Something along those lines, I have to explore that code base so i'll probably come up with a concise solution once i have looked at it a bit

So, openshift will provision a loadbalancer on the specific cloud that hosts the infrastructure. And then external-dns can take care of DNS entries and hence make DNS cloud independent.

[seanmalloy]

Hi @ahmedwaleedmalik and @jgrumboe

I'm very interested in having external-dns properly support OpenShift 4.x. Unfortunately I have not been able to find the time to help move it forward. I was going to try and submit an OKD enhancement proposal to have external-dns officially shipped as part of OpenShift, but have not found the time to write up a proposal yet. See https://github.com/openshift/external-dns/pull/1#issuecomment-618524096. Please feel free to chat me in k8s slack if you want to discuss further.

@ahmedwaleedmalik I'd be willing to at least help review your PR once you have it ready. I might even have some time to test out the code in our OpenShift 4.x environment.

CC: @sanbornick @vinny-sabatini

[ahmedwaleedmalik]

Hi @seanmalloy @jgrumboe
I was able to take some time out for this today and implemented this. Can you guys kindly review the PR #1725

[jgrumboe]

Hi @ahmedwaleedmalik
I will review it next week but I can only verify if it doesn't break OCP 3.11 support. I don't have an OCP 4.x cluster.
Nice weekend.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[seanmalloy]

/remove-lifecycle stale