external-dns not adding records

[CaptTofu]

Hi there!

I'm preparing for kubeconf, and working on slides, testing out demos, and wanted to get external-dns working on my kops-built cluster.

I followed this page verbatim, https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/aws.md, and I cannot get this to work.

This is what I have:

logs of the pod:

kubectl logs -f external-dns-2784180466-m22w2 --namespace=kube-system
time="2017-11-19T20:22:57Z" level=info msg="config: &{Master: KubeConfig: Sources:[service ingress] Namespace: FQDNTemplate: Compatibility: Provider:aws GoogleProject: DomainFilter:["kubeconf-17.kubernetes-cluster.com"] AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: Policy:sync Registry:txt TXTOwnerID:/hostedzone/ZxxxxxxxxxxxxxxxU TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 Debug:false}" 
time="2017-11-19T20:22:57Z" level=info msg="Connected to cluster at https://100.64.0.1:443" 
time="2017-11-19T20:22:57Z" level=info msg="All records are already up to date" 
time="2017-11-19T20:23:57Z" level=info msg="All records are already up to date" 

"all records are already up to date" no matter what.

My manifest (yes, I do own kubernetes-cluster.com :)

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.4.2
        args:
        - --source=service
        - --source=ingress
        - --domain-filter="kubeconf-17.kubernetes-cluster.com"
        - --provider=aws
#        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --registry=txt
        - --txt-owner-id=/hostedzone/ZxxxxxxxxxxxxxU

Never creates the record, and when I create the test service (nginx), I get the same "All records are already up to date"

What else am I missing? Thank you!

[mikkeloscar]

Could you run it with --log-level=debug to get as much information from the logs as possible? (you will need the latest version v0.4.7 for that to work though, otherwise the flag is called --debug.)

[oesah]

I have the same issue. When I do the tutorial with a subdomain, it works. But when I try with top-level domain "example.com"*, it will tell me also "All records are already up to date". I do have created the zone before, because I need it to create the cluster as far as I know. So now I would like external-dns to update the records accordingly. Is there a special case with top-level domains?

*of course I use a domain that I own in Route53

[linki]

@CaptTofu Our logging should be improved to immediately see what's wrong :unamused:

Until then, try out the following things:

• I never tried having / in the --txt-owner-id flag, try with something simple like foo. (Also note with a public zone you would expose your hosted zone ID to the world with this value)
• Services must have the annotation set, Ingresses need at least one rule
• Make sure that your Service/Ingress actually got an ELB assigned, via kubectl get svc,ing and check that the ADDRESS field is populated.

Get back to me and we can debug further to make you happy until KubeCon.

[linki]

@oesah Not that I recall. I'll try to double-check the correctness of the tutorial tomorrow.

[timspencer]

I have been having similar problems.

 : tspencer.local []$; kubectl get ing
NAME             HOSTS     ADDRESS            PORTS     AGE
signup-ingress   *         a21164361d3e9...   80        31m
 : tspencer.local []$; ~/go/bin/external-dns --registry txt --txt-owner-id kubernetes.foo.hungry.com --provider aws --source ingress --once --dry-run --metrics-address :7980 --log-level debug
INFO[0000] config: &{Master: KubeConfig: Sources:[ingress] Namespace: AnnotationFilter: FQDNTemplate: Compatibility: PublishInternal:false Provider:aws GoogleProject: DomainFilter:[] AWSZoneType: AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InMemoryZones:[] Policy:sync Registry:txt TXTOwnerID:kubernetes.foo.hungry.com TXTPrefix: Interval:1m0s Once:true DryRun:true LogFormat:text MetricsAddress::7980 LogLevel:debug} 
INFO[0000] running in dry-run mode. No changes to DNS records will be made. 
INFO[0000] Connected to cluster at https://api.kubernetes.foo.hungry.com 
DEBU[0001] No endpoints could be generated from ingress default/signup-ingress 
DEBU[0001] Skipping endpoint api.kubernetes.foo.hungry.com 60 IN A 54.186.139.126 because owner id does not match, found: "", required: "kubernetes.foo.hungry.com" 
DEBU[0001] Skipping endpoint api.internal.kubernetes.foo.hungry.com 60 IN A 172.20.48.7 because owner id does not match, found: "", required: "kubernetes.foo.hungry.com" 
DEBU[0001] Skipping endpoint etcd-a.internal.kubernetes.foo.hungry.com 60 IN A 172.20.48.7 because owner id does not match, found: "", required: "kubernetes.foo.hungry.com" 
DEBU[0001] Skipping endpoint etcd-events-a.internal.kubernetes.foo.hungry.com 60 IN A 172.20.48.7 because owner id does not match, found: "", required: "kubernetes.foo.hungry.com" 
INFO[0001] All records are already up to date           
 : tspencer.local []$; cat signup-ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: signup-ingress
  annotations:
    kubernetes.io/ingress.class: istio
    external-dns.alpha.kubernetes.io/hostname: signup.apps.hungry.com
spec:
  rules:
  - http:
      paths:
      - path: /.*
        backend:
          serviceName: signup
          servicePort: 3000

 : tspencer.local []$; 

I thought for a while that it was something to do with the fact that I was using istio, but I tried it without it too. I am suspicious about "DEBU[0001] No endpoints could be generated from ingress default/signup-ingress". Why wouldn't it find the endpoint? I can connect to the endpoint load balancer without any problems, so it seems like it ought to be suitable for external-dns to want to generate a record for it.

I am seeing this with the latest master code as well as with whatever comes when kops creates a 1.7.1 cluster.

[linki]

@timspencer That's currently not a bug: the /hostname annotation only works on Services in order for the user to declare the desired DNS name. For Ingresses we use the hostnames that are defined in the rules section of the ingress. So if you can just add it there it should work. Othwerwise please create an issue.

[linki]

@oesah I created a little PR that prints the zones that are matched and considered in the zone detection code. Give it a shot if you have time: https://github.com/kubernetes-incubator/external-dns/pull/401

Also please run ExternalDNS with log-level=debug grep for your desired record. It should show up with some more information as to why it doesn't create the record.

[linki]

@CaptTofu Got it 😄

The reason are the surrounding quotes around your domain filter in the manifest yaml. Kubernetes interprets them as part of the value. So your domain filter's value is something like that: "\"kubeconf-17.kubernetes-cluster.com\"" which then doesn't match any of your zones. Just leave them out.

[Raffo]

This seemed to be solved, closing.