External-dns: Constant Updates on CF DNS with v0.5.18 (related to duplicate IPs maybe)

[Xnyle]

After updating to 0.5.18 from 0.5.17 the CF provider constantly updates records resulting in them not being available 50% of the time on the CF DNS servers.

Downgrading to 0.5.17 solved the problem

Environment is a Bare Metal Cluster, actually a few of them, all writing to the same zone but with different --txt-owner-id=X --registry=txt

Each Cluster also runs a second ExternalDNS Deployment which updates a Local BIND9 instance via --provider=rfc2136. This instance is working correctly and only updating when changes occur.

Debug log (every minute):
Removing duplicate endpoint a.b.org 0 IN A X.X.X.1 []
Removing duplicate endpoint b.b.org 0 IN A X.X.X.1 []
[...]
Changing record." action=UPDATE record=a.b.org targets=1 ttl=1 type=A zone=xyzabc
Changing record." action=UPDATE record=b.b.org targets=1 ttl=1 type=A zone=xyzabc

So it's maybe related to duplicate IPs?
OFC I need duplicate IPs to still work, as the Loadbalancer IP for the Ingress serving all of them is obviously the same.

Maybe related to this issue #883

[Xnyle]

Actually the Removing duplicate endpoint messages are also present on 0.5.17 but not the UPDATE afterwards

[AaronFriel]

Seeing this issue as well, possibly related to #1421? No duplicate IPs.

[DurinMusicspear]

I also see this issue a lot in my cluster. Currently on 0.6.0.

[Bluesboy]

Same issue: GKE, Cloudflare, External DNS 0.6.0

[alexanderbuhler]

Same issue: AWS, Cloudflare, External DNS 0.6.0

[gioamato]

Same issue: Cloudflare, External DNS 0.7.0
Fixed reverting to v0.5.17

[alexanderbuhler]

Updated now to External DNS 0.7.0 with AWS and CF

time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint report.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com [{external-dns.alpha.kubernetes.io/cloudflare-proxied false}]"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint kiali.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com []"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint metrics.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com []"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint notification.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com [{external-dns.alpha.kubernetes.io/cloudflare-proxied false}]"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint dashboard.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com []"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint cart.prod2.example.com 0 IN CNAME  123456-2025305524.eu-central-1.elb.amazonaws.com [{external-dns.alpha.kubernetes.io/cloudflare-proxied false}]"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint api.prod.example.com 0 IN CNAME  456789-1571992763.eu-central-1.elb.amazonaws.com []"
time="2020-04-09T07:46:00Z" level=debug msg="Removing duplicate endpoint api.prod2.example.com 0 IN CNAME  456789-1571992763.eu-central-1.elb.amazonaws.com []"

time="2020-04-09T07:46:02Z" level=info msg="Changing record." action=UPDATE record=dashboard.prod2.example.com targets=1 ttl=1 type=CNAME zone=01010101010101010101010101010101
time="2020-04-09T07:46:02Z" level=info msg="Changing record." action=UPDATE record=jaeger.prod2.example.com targets=1 ttl=1 type=CNAME zone=01010101010101010101010101010101
time="2020-04-09T07:46:03Z" level=info msg="Changing record." action=UPDATE record=metrics.prod2.example.com targets=1 ttl=1 type=CNAME zone=01010101010101010101010101010101
time="2020-04-09T07:46:04Z" level=info msg="Changing record." action=UPDATE record=api.prod2.example.com targets=1 ttl=1 type=CNAME zone=01010101010101010101010101010101
time="2020-04-09T07:46:04Z" level=info msg="Changing record." action=UPDATE record=kiali.prod2.example.com targets=1 ttl=1 type=CNAME zone=01010101010101010101010101010101
time="2020-04-09T07:46:05Z" level=info msg="Changing record." action=UPDATE record=dashboard.prod2.example.com targets=1 ttl=1 type=TXT zone=01010101010101010101010101010101
time="2020-04-09T07:46:06Z" level=info msg="Changing record." action=UPDATE record=jaeger.prod2.example.com targets=1 ttl=1 type=TXT zone=01010101010101010101010101010101
time="2020-04-09T07:46:07Z" level=info msg="Changing record." action=UPDATE record=metrics.prod2.example.com targets=1 ttl=1 type=TXT zone=01010101010101010101010101010101
time="2020-04-09T07:46:09Z" level=info msg="Changing record." action=UPDATE record=api.prod2.example.com targets=1 ttl=1 type=TXT zone=01010101010101010101010101010101
time="2020-04-09T07:46:09Z" level=info msg="Changing record." action=UPDATE record=kiali.prod2.example.com targets=1 ttl=1 type=TXT zone=01010101010101010101010101010101

Behaviour:
<= 0.5.17: removing duplicate endpoint logs
Where it comes from is explained here: https://github.com/kubernetes-sigs/external-dns/issues/1070#issuecomment-523645614

> 0.5.17: removing duplicate endpoint + changing record logs (actually updates records and partially confusing CF if using loadbalancing there)

[brpaz]

I think I am having the same issue, but I am not seeing any message regarding Duplicate records. Only updates.

time="2020-04-14T20:37:29Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:37:30Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:37:30Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:37:31Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:38:29Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:38:30Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:38:33Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:38:33Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:39:29Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:39:30Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:39:30Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:39:31Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:40:29Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:40:30Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:40:30Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:40:31Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:41:29Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:41:30Z" level=info msg="Changing record." action=UPDATE record=labs.brunopaz.dev targets=1 ttl=1 type=A zone=1bb86879fb7ea053bd1352c7a9b60ca2
time="2020-04-14T20:41:30Z" level=info msg="Changing record." action=UPDATE record=directus.brunopaz.dev targets=1 ttl=1 type=TXT zone=1bb86879fb7ea053bd1352c7a9b60ca

I am also seeing my websites going down for many time randomly with DNS_NOT_RESLVED error.
I tought it was something with Cloudflare, but now I am thinking this constants updates of external-dns may have something to do with it.

[ainmosni]

I had this problem on 0.7.1 on a baremetal k3s server while using CF DNS as well.

[keslerm]

Just wanted to put a note this seems to have a fix in pr #1536

There is a temp fix here while that gets tested/merged https://github.com/kubernetes-sigs/external-dns/issues/1540#issuecomment-625240804

[sheerun]

This should be now fixed. If not, please create another issue with steps to reproduce, or ideally a test in cloudflare_test.go or other affected providers. Tests for Cloudflare provider are really easy to write :)

/close

[k8s-ci-robot]

@sheerun: Closing this issue.

In response to this:

This should be now fixed. If not, please create another issue with steps to reproduce, or ideally a test in cloudflare_test.go or other affected providers. Tests for Cloudflare provider are really easy to write :)

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.