External-dns: Cloudflare provider causes repeated UPDATE

[al45tair]

What happened:
We're using the Cloudflare provider, and we're seeing it update all of the DNS records every minute. This is disruptive because when updating Cloudflare, the record is momentarily deleted; if something caches the NXDOMAIN that comes back from resolving the record during that period, it can be very annoying (the TTL on Cloudflare's NXDOMAIN responses is not adjustable, nor is it particularly short).

What you expected to happen:
We'd like it to update only when we change things :-)

How to reproduce it (as minimally and precisely as possible):
Try to use the Cloudflare provider with External DNS 0.5.18 or higher (master works), and with it set to not proxy by default.

Anything else we need to know?:
Yes. The problem is that the Cloudflare provider always returns its annotation, external-dns.alpha.kubernetes.io/cloudflare-proxied when reading records from Cloudflare. That's great, but if you don't specify that annotation on your ingresses (so, you're using the default set in the Cloudflare provider), the desired state won't have it but the records read from the provider do, so the planner thinks it needs to push an update.

This might be a problem for any other provider if they add an annotation with a "default" value.

Here's what the planner saw:

(*plan.Plan)(0xc0002c4780)({
 Current: ([]*endpoint.Endpoint) (len=1 cap=4) {
  (*endpoint.Endpoint)(0xc0004b4cb0)(foo.example.com 1 IN A  10.1.2.3 [{external-dns.alpha.kubernetes.io/cloudflare-proxied false}]),
 },
 Desired: ([]*endpoint.Endpoint) (len=1 cap=4) {
  (*endpoint.Endpoint)(0xc000384800)(foo.example.com 1 IN A  10.1.2.3 []),
 },
 Policies: ([]plan.Policy) <nil>,
 Changes: (*plan.Changes)(0xc0006143c0)({
  Create: ([]*endpoint.Endpoint) <nil>,
  UpdateOld: ([]*endpoint.Endpoint) (len=1 cap=4) {
   (*endpoint.Endpoint)(0xc0004b4cb0)(foo.example.com 1 IN A  10.1.2.3 [{external-dns.alpha.kubernetes.io/cloudflare-proxied false}]),
  },
  UpdateNew: ([]*endpoint.Endpoint) (len=1 cap=4) {
   (*endpoint.Endpoint)(0xc000384800)(foo.example.com 1 IN A  10.1.2.3 []),
  },
  Delete: ([]*endpoint.Endpoint) <nil>,
 }
})

It then pushed an update, very reasonably, but of course that didn't actually change anything, so the next time External DNS checked, it was in the exact same state.

Environment:

• External-DNS version (use external-dns --version): 0.5.18 or above
• DNS provider: Cloudflare
• Others: n/a

[AlexLast]

Also seeing this issue

[travisghansen]

As a ramification of this (I believe) I've been seeing situations where the record is actually in cloudflare but in a bad state that also results as an NXDOMAIN. Apparently they've seen this issue before from the API and I suspect it has something to do with a serial number for the record but not entirely sure.

If you login to the UI I see this:

Invalid DNS record identifier (Code: 1032)

I believe the problem is outside the scope of this issue, but I'm seeing it proportionally more due to the constant updates (only a handful of records in a test cluster and I'm regularly seeing NXDOMAIN when the record does actually exist in cloudflare).

[ninja-]

This is a very big problem. The repeated updating of records ultimately leads to bugging out the cloudflare api and the records are no longer visible over DNS even though they are visible in the cliudflare control panel! To fix it you need to switch proxied status on/off in the control panel for example

Super annoying. Had to remove my external-dns instance and only run it for a small period of time when I am changing something in my setup.

[travisghansen]

@ninja- good to know I'm not the only one! I've had some really really strange behaviors going on and am currently working with the CF support on this very issue from their side as well.

[dschaaff]

The workaround for now is to use 0.5.17 which doesn't have the bug.

[travisghansen]

Interesting this is my first foray into external-dns and didn't realize it was a regression.

I've installed from the bitnami chart which if it's to be believed I'm running 0.7.0

bitnami/external-dns    2.20.8          0.7.0       ExternalDNS is a Kubernetes addon that configures public ...

[nicolasdonoso]

I'm also seeing this issue and also when adding multiple different records it changed the action to CREATE and the ones that where in UPDATE status were removed from Cloudflare.

[mwmahlberg]

It would be nice to know when the PR will be merged, because there are even more grave ramifications: one of the major reasons to use cloudflare is its DDoS protection. With the way it is now, you can either set up the policy to upsert-only (with which everything works as expected) OR you need to disable proxying the respective IP address altogether. Frankly that is a situation I would call less than ideal, either way.

[demon]

Started hitting this earlier today. Also annoying is that commenting out the external-dns.alpha.kubernetes.io/* annotations entirely _doesn't make the issue go away_, the controller just plugs away at trying to add a record that it's confused about.

[nickbp]

Had a site go completely offline due to this. The incessant updates from external-dns eventually resulted in Cloudflare dropping all A records for the domain, and I needed to manually go in and kick the entries in the Cloudflare zone configuration to get them back. Good thing it was only a personal site!

I have stopped using external-dns for now because it’s no longer trustworthy. It had one job.

[pmcjury]

Upsert doesn’t always work eliminate the issue FYI. It’ll be less periodic I found anyways. 

Sent from Yahoo Mail for iPhone

On Thursday, April 9, 2020, 4:30 AM, Markus Mahlberg [e-mail] wrote:

It would be nice to know when the PR will be merged, because there are even more grave ramifications: one of the major reasons to use cloudflare is its DDoS protection. With the way it is now, you can either set up the policy to upsert-only (with which everything works as expected) OR you need to disable proxying the respective IP address altogether. Frankly that is a situation I would call less than ideal, either way.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[travisghansen]

@nickbp that sounds horrific. Seems unlikely this tool deleted the whole zone no?

Does anyone know if reverting back to 0.5.17 as has been suggested actually resolves the issue?

[nickbp]

To be clear the rest of the zone was fine - only the A records managed by external-dns had vanished from live DNS while still being listed in the Cloudflare control panel for the domain. For example some MX records against the same domain and ignored by external-dns were still working fine and showed up in dig throughout the outage.

I was only able to get the A records to return to live DNS by disabling and enabling proxying for them in the control panel. I think what happened matched the behavior reported by ninja- a few days ago.

I had also tried deleting all of the A and TXT records managed by external-dns, and then waiting for external-dns to recreate them automatically. After they were recreated by external-dns the issue still remained, the A records were still missing entirely from dig mydomain.com despite being again listed in the control panel after external-dns had recreated them. As found in an earlier comment by ninja-, the way I finally got the A records “unwedged” was to disable and re-enable proxying. After doing this the A records reappeared in dig queries.

This seems to be a combination of two bugs:

• external-dns spamming Cloudflare with spurious update commands
• Cloudflare eventually going haywire and dropping the DNS records entirely from live DNS despite still being listed in the control panel

To be fair to external-dns, the bigger problem seems to be on Cloudflare’s end of things. If the bug on Cloudflare’s end didn’t exist, the update bug in external-dns would just be obnoxious, rather than catastrophic. It’s even possible that external-dns has unmasked a problem (race condition?) that Cloudflare would have eventually exhibited anyway, given enough attempts to update the A records. In this situation external-dns would have just accelerated things.

But the point remains that anyone seeing spurious UPDATE requests in their external-dns logs is potentially exposed to this issue and may see their A records disappear from live DNS given enough time for external-dns to eventually expose the bug on Cloudflare’s end.

[travisghansen]

@nickbp interesting, so the record was still in CF correct? Did you happen to see the following in the admin UI?

Invalid DNS record identifier (Code: 1032)

This sounds like exactly the issue I'm working through with them ATM. The record was there (in the UI and API) but when using DIG a NXDOMAIN was the response.

[nickbp]

I do not remember seeing NXDOMAIN nor that error on the control panel. The domain was still found (NOERROR) but it didn’t have any A records - instead an SOA record was returned to the A query. Meanwhile everything looked completely normal in the Cloudflare control panel from what I remember.

I had A records for three different names: mydomain.com, www.mydomain.com, and mail.mydomain.com. All three of these A records were managed by external-dns, and all three had gotten into the “wedged” state of being visible in the Cloudflare control panel but missing from live DNS (SOA record returned instead).

Meanwhile other record types against the same domain like MX and SRV, which weren’t managed by external-dns, continued to function normally.

[al45tair]

@nickbp How did you tell whether they were in the DNS? If you did it by querying using your local DNS server, it's probably the same issue that I mention above, namely that when updating Cloudflare, there is a moment when the record doesn't exist; if your name server queries Cloudflare's at that precise moment, it gets an NXDOMAIN with a TTL of 3600s, so you won't be able to see the record for the next hour.

You can directly query the authoritative server at Cloudflare instead, which will tell you whether it's really in the DNS or not without the NXDOMAIN problem. If you get an NXDOMAIN from that, it doesn't matter because it won't cache it.

[sra]

@travisghansen Reverting to 0.5.17, chart version external-dns-2.13.0, prevents the behavior for us for now, we did see the drop-out above several times where published hosts with A records were just not resolving anymore.

[travisghansen]

@al45tair I was diging the authoritative server(s). I confirmed the records using the CF api directly. At that point someone had to login directly to the interface and tried (this part wasn't me) to update something in the UI which was completely disallowed (I got the sense the UI was failing to allowing any updates due to the bad state) with the error mentioned above.

Ultimately CF support had to do something...not sure exactly what to get the error to disappear in the UI and ultimately the domain started resolving again.

I do understand the ramifications of the brief moment with the TTL of 3600 etc etc...it was not that scenario but something else altogether.

[nickbp]

@al45tair This was not a blip or sporadic outage. I first knew there was a problem because I had “freshping” checking for outages. They reported the site down at 4:07AM UTC and it remained continuously unreachable to them (with checks every minute) for over 7 hours*, until reporting that the site was back up a few minutes after I had gotten around to fixing the issue via the “toggle proxy” workaround. So whatever broke the A records on Cloudflare’s end was definitely permanent until I finally went in and got it unwedged manually.

• Again, personal site, so wasn’t in any particular rush - good thing I’m not using this for anything serious!

[logan2211]

I can confirm, I've seen the same issue multiple times over the past few weeks where the records are still active in the cloudflare control panel, however querying the records from the cloudflare authoritative NS's for the domain returns NXDOMAIN. Deleting the A records and allowing external-dns to re-create them sometimes works. One time I had to delete the A records several times before they started showing up in the authoritative NS responses again.

One of the instances I waited several days to see if the issue would resolve itself. As @nickbp confirms above, it does not self resolve with time. Manual intervention (sometimes multiple times) is needed.

I'm using upsert mode with external-dns, so I can also confirm upsert is not immune to this issue.

[travisghansen]

If any of you hit that crazy state and can share DNS names with me (can email if you prefer) that would be great. CF is asking for examples..

[cameronbraid]

@travisghansen

I have an open ticket with cloudflare to try and resolve a CNAME issue where dns lookups aren't resulting in the expected values. I believe this issue may be the cause of my issue and have mentioned it to them, and provided logs and DNS name examples.

[travisghansen]

Ok thanks. I've received some emails as well. What would be particularly helpful would be if anyone encounters the issue again and can leave it in a bad state for a little bit so we can send it off to CF to have a look at.

[ramwolken9]

Hi, I recently started using external-dns and CF as provider, Today I observed this situation - half of the sub-domains in CF are not functioning which were added using external-dns. When I visit the site it shows me

Hmmm… can't reach this page
xxx.xxx.com’s server IP address could not be found.

Could you please help me fix this.

[al45tair]

@ramwolken9 If it's the problem this ticket is about, you can either downgrade your external-dns (reportedly), or make sure you explicitly specify the external-dns.alpha.kubernetes.io/cloudflare-proxied attribute on your ingresses. If it's explicitly specified, this bug won't trigger and you won't have external-dns sending updates all the time.

It sounds like there might be another problem as well, at the Cloudflare end, that is causing some people grief, but if you're having this problem, then the reason you're getting the failure to resolve is probably that your local caching DNS server has cached an NXDOMAIN that happened when external-dns was doing a DNS update. Normally you'll struggle to see that (you'd be very unlucky to try to resolve at a moment when a DNS update was in progress), but if you haven't got the attribute specified then external-dns makes updates all the time without my patch.

[travisghansen]

Yeah I'm of the opinion this issue simply amplifies the CF issue.

@ramwolken9 if you can let it sit in a bad state for a day or two so we can send to CF to analyze that would be great.

@al45tair separate question but what's the reasoning for doing delete/create sequence for updates anyway? Why is external-dns not simply updating the existing record at CF?

[ramwolken9]

@travisghansen Yes, I can do it as I'm testing k8 and external-dns for my project in pre-production. Please let me know what next.
Thank you for your suggestion @al45tair

[travisghansen]

@ramwolken9 if you could send the record(s) here or as a direct email I'll forward to CF on the support ticket I've got with them currently..

[ramwolken9]

@travisghansen I have emailed you.

Update : Record seems to be working now!

[al45tair]

@travisghansen I think it's a bug, actually, having looked at it. I had assumed that perhaps the Cloudflare Go API lacked an Update method, but that's not the case, and it's quite apparent that sending a delete and then a create isn't equivalent to doing an update for precisely the reason that there will be a small window during which the record has been removed.

I suppose in many cases this wouldn't be such an issue because the update is happening e.g. because of an IP address change, so a small disruption to clients might not be unexpected, but even then it still makes it an hour long (because of the TTL on NXDOMAIN responses from Cloudflare).

I've just filed Issue #1514 to cover this. I don't have the time to fix it right now — I don't think it's a difficult fix — but I might have some time to look at it next week maybe if things calm down.

[travisghansen]

Interesting...I really need to become more proficient in go :(

I'll dig through the CF API to see if there's an idempotent version (ie: upsert) or similar which seems like the ideal option if available.

[cameronbraid]

Looks like there is already a PR to fix this issue https://github.com/kubernetes-sigs/external-dns/pull/1464

[afirth]

There's a lot of noise in this thread. Repeat that this is a breaking bug for users using cloudflare. It will result in you getting ratelimited and then new records can't be created. Fix in #1464 seems reasonable. Workaround is to use 0.5.17.
/cc @Raffo maybe you have time to look at this?

[sheerun]

Another solution implemented in https://github.com/kubernetes-sigs/external-dns/pull/1534 (it adds "kinds" to different properties (in this case the kind is "boolean")

[hayderimran7]

literally seeing this issue on our end. i have bugged so many times cloudflare support (those poor guys) only to find out its issue at external DNS.

[hayderimran7]

~i can confirm using bitnami/external-dns:0.5.17 solved this issue at my end~
Nope still exists, only way to fix is by increasing the interval as --interval=30m to 30minutes for example

[sheerun]

If someone would like to test / verify fixes to this issue (pull requests #1536, #1542 and #1494 as an extra), then I've merged them into dev branch on https://github.com/sheerun/external-dns/tree/dev and released on docker hub as: sheerun/external-dns:v0.7.1-52-g4d301cf2

  repository: sheerun/external-dns
  tag: v0.7.1-52-g4d301cf2

I think currently the best way to use cloudflare is to use upsert-only strategy and noop registry.

[travisghansen]

Why noop registry? I'm pretty new to external-dns so my understanding may be off but that could be pretty harmful by deleting records not 'owned' by external-dns at all or records 'owned' by a different instance of external-dns/different cluster? Same for creates/modifies right?

[sheerun]

upsert-only strategy updates only existing records with hosts registered as services or ingresses (it doesn't delete non-registered addresses in kubernetes). For any issues with this approach you'd need to have two clusters with the same dns names registered as ingresses / services. On the other hand txt registry can cause issues with updating records (i.e. them not being updated)

[travisghansen]

Yeah ok. I'll have multiple clusters. What's up with the TXT record issue? Is that a separate issue/bug altogether?

[sheerun]

It's not an issue in software per se, but it can cause issues because you need to update TXT records manually (or delete them) if you want to change which cluster owns which record. If you use noop registry then it's as simple as configuring one dns name in one cluster and not in other.

[travisghansen]

Yeah I got ya, but nothing will ever get deleted in that scenario as well (in an automated fashion) as I understand.

Given the fixes the frequency of 'updates' should be far less making any issues with creates/updates of records also greatly mitigated/unlikely correct? Meaning, unless an IP changes the record should effectively never be touched in CF via external-dns after initial creation.

[al45tair]

I think, actually, currently the best way to use Cloudflare is to run your own authoritative DNS server somewhere using bind, and set Cloudflare up as a slave server. That way you can use external-dns's RFC2136 support and everything should just work.

This has the advantage that you aren't dependent on this being fixed, and what's more you can configure whatever records you like in your DNS — even things Cloudflare's UI won't let you have.

[travisghansen]

That's an interesting thought for sure. I'll have to think about that one. I'm not a DNS expert so I'll have to think about what that would look like and go from there.

Would that need to be a fully isolated/dedicated zone? How does bind sync to CF? Does CF remain authoritative? Sorry for the ignorance..

[al45tair]

You'd have bind running the entire zone, yes, and yes, CF would remain authoritative (you probably wouldn't list your primary server as a DNS server for your domain; you'd just know about it, and so would Cloudflare).

Sync happens using the usual DNS mechanism; bind will send NOTIFY messages to its slaves, and the slaves will do XFERs to grab the zone. You need to tell bind that CF's servers are allowed to do XFERs, and you might need to tell CF somehow that it should accept NOTIFY messages.

I do this for my own VPS (I use Linode, and I use Linode's DNS, but set them up as slaves deliberately). I haven't tried it with CF, but it should work, and has the advantages I mentioned.

[travisghansen]

OK, if I understand I don't think that would work so well for me as I have non-k8s stuff that need to create entries in the same domain/sub-domain. I guess in that case you just have to 'manage' the whole zone in bind (even non-k8s stuff) and then let it sync..interesting thought for sure.

[sheerun]

nothing will ever get deleted in that scenario as well (in an automated fashion) as I understand

With the fixes I've provided deletions happen for dns updates if number of targets for some dns is reduced. e.g. if originally there are two A records for bar.com, and one node goes down, then external-dns is going to remove one of the records.

As for total removal of some records this won't happen automatically, but I think it's good thing as much safer option and it won't hurt anyone if records stay untouched instead of being deleted. I guess external-dns could introduce feature where it would be possible to explicitly mark some domain as inactive and then external-dns would remove all records for it (e.g. a CRD that is created by post-uninstall hook of helm, or exteral-dns listening removal of ingresses and services and reacting to them, as opposed to just mapping them to dns records, as it currently does).

Given the fixes the frequency of 'updates' should be far less making any issues with creates/updates of records also greatly mitigated/unlikely correct? Meaning, unless an IP changes the record should effectively never be touched in CF via external-dns after initial creation.

Also with the fixes I've provided there should be no updates if there are no changes. On top of that updates are minimal, e.g. if "proxied" annotation is added to ingress, then existing records are updated instead of being removed and then re-created.

[travisghansen]

@sheerun yeah, understood. All good news for sure.

Regarding deletes I have some ci clusters with a high churn rate for hosts, if I let those sit around forever they'd build up fast :( That's my (perhaps unique) concern with that.

[sheerun]

I've also submitted #1543 (available as sheerun/external-dns:v0.7.1-54-g98ac8302) that introduces additional protection preventing concurrent updates and ensuring updates happen no more than once per configured interval. This should prevent this issue happening all together.

[hayderimran7]

@sheerun thanks for the fix. to test it, do we need to use any additional annotations or modify existing ones ?

[travisghansen]

@sheerun thanks for all the work! Very positive stuff getting done here.

[hayderimran7]

@sheerun with your fix i still see the UPDATE happening every 1minute. :/

[sheerun]

@hayderimran7 Maybe you don't see updates but just "checks" that happen every interval. If you indeed see updates, could you please post steps to reproduce on local kubernetes cluster? (with sheerun/external-dns:v0.7.1-54-g98ac8302 image)

[hayderimran7]

@sheerun ah i see, so the UPDATE in logs after each minute, is a check if record exists, dont send a UPDATE Api call to cloudflare? and if doesnt then create or remove based on ingresses ?

[bsundsrud]

I also ran into this. Using the bitnami helm chart (2.20.14) but overrode the image to registry.opensource.zalan.do/teapot/external-dns:v0.5.17 and its working as expected again.

[bsundsrud]

Rolled update to new cluster, seeing repeated UPDATEs with 0.5.17... Any updates or movement on this issue?

[al45tair]

We're using a private build of external-dns with my patch, to avoid this problem. I'm just doing another build and I'm going to push the result to Docker Hub so that others can use it. My PR #1464, which does fix this problem, has still not been reviewed by @linki or @njuettner, two months after I originally submitted it, and I've just had to rebase it again.

The fixed image is at al45tair/external-dns:fix-cloudflare-attr-update, if anyone else wants to try it.

[hayderimran7]

@al45tair with your image, do we need to change anything? is it okay to keep same 1min of UPDATE interval?

[al45tair]

@hayderimran7 it’s a drop-in replacement for the normal image; no need to change anything.

[hayderimran7]

@al45tair i tested and i still see it tries to run UPDATE and DELETE every minute (default interval) 😭

time="2020-05-15T01:24:56Z" level=info msg="Changing record." action=CREATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:24:56Z" level=info msg="Changing record." action=CREATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:25:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:25:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:27:02Z" level=error msg="error from makeRequest: HTTP status 429: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":10000,\\\"message\\\":\\\"Rate limited. Please wait and consider throttling your request speed\\\"}]}\\n\""
time="2020-05-15T01:27:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:27:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:28:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:29:03Z" level=error msg="failed to create record: error from makeRequest: HTTP status 429: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":10000,\\\"message\\\":\\\"Rate limited. Please wait and consider throttling your request speed\\\"}]}\\n\"" action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:29:03Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:29:11Z" level=error msg="failed to delete record: error from makeRequest: HTTP status 429: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":10000,\\\"message\\\":\\\"Rate limited. Please wait and consider throttling your request speed\\\"}]}\\n\"" action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:29:18Z" level=error msg="failed to create record: error from makeRequest: HTTP status 429: content \"{\\\"success\\\":false,\\\"errors\\\":[{\\\"code\\\":10000,\\\"message\\\":\\\"Rate limited. Please wait and consider throttling your request speed\\\"}]}\\n\"" action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:30:04Z" level=info msg="Changing record." action=CREATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:30:04Z" level=info msg="Changing record." action=CREATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:30:05Z" level=error msg="failed to create record: error from makeRequest: HTTP status 400: content \"{\\n  \\\"result\\\": null,\\n  \\\"success\\\": false,\\n  \\\"errors\\\": [\\n    {\\n      \\\"code\\\": 81057,\\n      \\\"message\\\": \\\"The record already exists.\\\"\\n    }\\n  ],\\n  \\\"messages\\\": []\\n}\\n\"" action=CREATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:30:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:30:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:31:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=120 type=CNAME zone=4f70008fc86e94af26a8be34e217cfe1
time="2020-05-15T01:31:56Z" level=info msg="Changing record." action=UPDATE record=SOME_CF_RECORD targets=1 ttl=1 type=TXT 

[al45tair]

@hayderimran7 In which case I suspect you have a different problem that is causing that; it looks like whatever issue you're having is related to CNAME and TXT records rather than the A records that were causing trouble for us. In order to diagnose it properly, you'd need to build a custom version of external-dns so that we can see why the planner thinks it needs to update these records all the time (because external-dns doesn't output the necessary detail normally).

[al45tair]

@hayderimran7 It occurred to me just now what your problem is. Or what it might be. If you have a CNAME record for a given name, you can't have any other record for it. There's an option for external-dns to alter the names used for the TXT records if you need external-dns to make CNAMEs. Without it, it's possible that you will indeed get repeated updates.