external-dns not updating Route53 Record on target change

[juanmatias]

I have external-dns working on an EKS cluster. I'm using ALB ingress controller.

When I deploy my ingresses (I'm using Helm), external-dns creates the records with no problem. If I delete them, external-dns deletes the records ok.

---

Problem

• if for some reason (e.g. re create the ingress), the ALB address changes
• then, the record on Route 53 for the hostname exists, but with a diferent alias

In this scenario, external-dns won't update the Record, resulting on an error when I'm trying to access the url.

---

Workaround

If I delete manually the Record and wait, eventually external-dns creates the record with the right alias.

---

Analysis

As far as I understand external-dns actually is checking for changes on the Target (here), but since I'm not a Go developer I'm not sure.

But, reading the symptoms, I think for some reason external-dns is not realizing that the target (ALB's address) has changed. It can go like this for a long period of time with no changes.
Only when I delete the record, and wait a couple of minutes, external-dns finds that the record for the host is not there and then creates it.

I had no policy option set on first place. Then, and just to test it, I added --policy=sync with no success.

---

Environment

external-dns

~ $ external-dns --version
v0.5.17-23-g80555587

    Image:         registry.opensource.zalan.do/teapot/external-dns:latest
...
    Args:
      --source=service
      --source=ingress
      --provider=aws
      --aws-zone-type=
      --registry=txt
      --txt-owner-id=mysolution-prod-eks-cluster-external-dns
      --policy=sync

aws-zone-type is set this way so external-dns can handle private and public zones.

K8s

Kubernetes: v1.13.10-eks-d6460e

Sample ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: release-name-mysolution-private
  labels:
    app: release-name-mysolution-private
  annotations:
    app.kubernetes.io/name: mysolution-private
    app.kubernetes.io/instance: release-name 
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internal
    kubernetes.io/ingress.class: albprivate
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:999999999999:certificate/fakehash
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    external-dns.alpha.kubernetes.io/hostname: mysolution.stage.mysolution.private
spec:
  rules:
    - host: mysolution.stage.mysolution.private
      http:
        paths:
          - backend:
             serviceName: ssl-redirect
             servicePort: use-annotation
          - backend:
             serviceName: mysolution
             servicePort: 80

---

Please, let me know if you need more data. Thanks.

[njuettner]

Could you paste your debug logs?

[tiagocostasilva]

Hello, i am having the same issue, did anyone resolve this?

[ricardoptcosta]

This problem is due to incompatibility between Nginx ingress controller and Kubernetes versions. You have two alternatives:

• either update Kubernetes version from 1.13 to 1.14, or

• update the following lines on the mandatory.yaml file of the nginx setup:
  line 217: from "kubernetes.io/os: linux" to "beta.kubernetes.io/os: linux"
  line 220: from "from "image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0" to "image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.0"
  line 236: changed from "runAsUser: 101" to "runAsUser:33"

[juanmatias]

Hi, just in case someone is facing the same issue.

Despite the version issue, enabling debugging allowed us to discover that we had another problem.

On some point, and due to project changes, we changed the txt-owner-id. Looking into the logs, we found that the problem was that we had old records, with the previous owner id, that couldn't be changed because the owner id must be the same.

We're working on collect more info and delete all the old records. When you delete a record, eventually external-dns will detect it's missing and will create it again.

Thanks.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten