External-dns: AWS provider doesn't support multiple IPs for ingress

[claytono]

I'm using the nginx ingress controller on bare metal, but the domain I want to update is hosted on Route53. When attempting an update in --once mode, I get the following output:

time="2017-07-11T09:39:31-04:00" level=info msg="config: &{Master: KubeConfig: Sources:[ingress] Namespace: FQDNTemplate: Compatibility: Provider:aws GoogleProject: DomainFilter:[k.oneill.net.] AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: Interval:1m0s Once:true DryRun:false LogFormat:text MetricsAddress::7979 Debug:true}" 
time="2017-07-11T09:39:31-04:00" level=info msg="Connected to cluster at https://k1:6443" 
time="2017-07-11T09:39:31-04:00" level=debug msg="Endpoints generated from ingress: default/plexpy: [plexpy.k.oneill.net -> 172.19.74.134 (type "") plexpy.k.oneill.net -> 172.19.74.135 (type "") plexpy.k.oneill.net -> 172.19.74.147 (type "")]" 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "172.19.74.134"
      }],
    TTL: 300,
    Type: "A"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "172.19.74.135"
      }],
    TTL: 300,
    Type: "A"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "172.19.74.147"
      }],
    TTL: 300,
    Type: "A"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "\"heritage=external-dns,external-dns/owner=default\""
      }],
    TTL: 300,
    Type: "TXT"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "\"heritage=external-dns,external-dns/owner=default\""
      }],
    TTL: 300,
    Type: "TXT"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=info msg="Changing records: CREATE {
  Action: "CREATE",
  ResourceRecordSet: {
    Name: "plexpy.k.oneill.net",
    ResourceRecords: [{
        Value: "\"heritage=external-dns,external-dns/owner=default\""
      }],
    TTL: 300,
    Type: "TXT"
  }
} ..." 
time="2017-07-11T09:39:31-04:00" level=error msg="InvalidChangeBatch: The request contains an invalid set of changes for a resource record set 'A plexpy.k.oneill.net.'
    status code: 400, request id: 5e5c4473-663e-11e7-ab1e-73be49546003" 

I believe this is occurring because the Route53 API doesn't allow creating multiple A records for a single name, but instead expects you to create a single A record with multiple ResourceRecordSet entries.

I think that the AWS provider needs to be changed to combine these A and probably TXT record entries before submitting the change set. I'd be glad to do this work if that fix seems acceptable. I tend to think that the easiest way to do this is to add a filter func that would coalesce records under the same name and RR type.

I've tested this against 0.3 and against master and had the same problem.

[ideahitme]

yes confirm the problem exists. However the question is when u want to combine these A records?
There are two scenarios:

1. Single ingress with a hostname and multiple targets (IP addresses)
2. Different ingress resources with the same hostname pointing to different targets.

For 1 it is probably fine to do ur suggestion. However for 2, I would argue it is not desirable to have ur dns record "hijacked" by someone else who just happened to use same hostname.

As for 1 there is already work in progress in: https://github.com/kubernetes-incubator/external-dns/pull/243/files (it changes the internal structs to have multiple targets)

[claytono]

I think 1 and 2 are effectively the same case for Route53. It doesn't support submitting multiple changes for the same name and A record in the same changeset. It processes the changes in the order they occur in the change set, gets to the second CREATE change and sees that there is already record with that name and rrType and throws an error. It doesn't have any concept of appending to an A record, or multiple A records with different values for the same name.

[claytono]

Also, I think, but am not sure yet that this issue exists for all record types, not just A records. I suspect this has to be fixed for TXT records, etc also.

[ideahitme]

@claytononeill on Route53 side - yes, both scenarios currently generate the same error, however whether we should handle them differently is an issue I was referring to.

Also, I think, but am not sure yet that this issue exists for all record types, not just A records. I suspect this has to be fixed for TXT records, etc also.

TXT is a supplementary record for us to store metadata to ensure record ownership by ExternalDNS. ALIAS records can co-exist with other type of records such as TXT. By default External DNS on Route53 prefers ALIAS to CNAMEs.

[claytono]

That makes sense. I think the issue is specifically that the name + rrType must be unique. Does that sound right?

[ideahitme]

Not always, u can have a delegated hosted zone where u can create a DNS record with identical name.

[claytono]

In that case the FQDN will be different though. I believe Route53 is always using the FQDN as the name internally. Do you agree that FQDN + rrType must be unique?

[claytono]

This appears to still be an issue in 0.4. It no longer dies when multiple nodes are hosting a single ingress, but it only updates the A records with a single IP address. I'd be willing to take a shot at fixing this if that would be ok.

[linki]

@claytononeill Sorry about that. We should make this limitation more clear in the docs.

You can definitely take a swing at it. The proposed solution would be to group a list of endpoints by its hostname so that that the providers can create the correct request. One way to do that is to extent the Endpoint struct's Target property to store a list of []Targets instead. This way we could later add things like Weights etc. more easily. But we are open to other suggestions.

I dumped a work in progress of mine a couple of weeks ago and @sethpollack also started working on a similar topic. It would be great to make progress regarding this topic and any help is appreciated.

[james-masson]

This also seems to be true for GCE DNS.

time="2017-09-22T13:15:40Z" level=debug msg="Endpoints generated from ingress: default/nginx: [testrecord.testzone.io -> <public_ip_1> (type "") testrecord.testzone.io -> <public_ip_2> (type "") testrecord.testzone.io -> <public_ip_3> (type "")]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. A [<public_ip_1>]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. A [<public_ip_2>]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. A [<public_ip_3>]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. TXT ["heritage=external-dns,external-dns/owner=helm-dev"]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. TXT ["heritage=external-dns,external-dns/owner=helm-dev"]"
time="2017-09-22T13:15:40Z" level=info msg="Add records: testrecord.testzone.io. TXT ["heritage=external-dns,external-dns/owner=helm-dev"]"
time="2017-09-22T13:15:41Z" level=error msg="googleapi: Error 409: The resource 'entity.change.additions[1]' named 'testrecord.testzone.io. (A)' already exists
More details:
Reason: alreadyExists, Message: The resource 'entity.change.additions[1]' named 'testrecord.testzone.io. (A)' already exists
Reason: alreadyExists, Message: The resource 'entity.change.additions[2]' named 'testrecord.testzone.io. (A)' already exists
Reason: alreadyExists, Message: The resource 'entity.change.additions[4]' named 'testrecord.testzone.io. (TXT)' already exists
Reason: alreadyExists, Message: The resource 'entity.change.additions[5]' named 'testrecord.testzone.io. (TXT)' already exists

[dgregoire]

True also with Cloudflare, and by the same token if multiple A records for multiple controllers are created, and then the ingress deleted, the TXT heritage record is deleted and only one of the A record, I think that makes sense under the circonstances. Is there currently a work around this condition?

time="2017-10-13T16:18:23Z" level=info msg="Changing record." action=UPDATE record=XXX1 type=A zone=ZONE1 
time="2017-10-13T16:18:23Z" level=error msg="failed to update record: error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":81058,\"message\":\"A record with those settings already exists.\"}],\"messages\":[],\"result\":null}"" action=UPDATE record=XXX1 type=A zone=ZONE1

time="2017-10-13T16:18:23Z" level=info msg="Changing record." action=UPDATE record=XXX2 type=A zone=ZONE2 
time="2017-10-13T16:18:24Z" level=error msg="failed to update record: error from makeRequest: HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":81058,\"message\":\"A record with those settings already exists.\"}],\"messages\":[],\"result\":null}"" action=UPDATE record=XXX2 type=A zone=ZONE2

[Promaethius]

Route53 allows several DNS routing policies for several IP's. I'd like to see that implemented in this module.

[bitvector2]

I would be happy to help on this. What is the current planned approach?

[Promaethius]

A good example would probably be the kops dns controller: https://github.com/kubernetes/kops/tree/master/dns-controller

However, I'm not familiar enough with this project to advise. I'll try to read the code this week to be some help.

The Route53 API requires that all changes for a record are submitted at once.

Provider interface allows for a list of endpoints to be inserted one at a time. We're looking for a group of endpoints associated with a record. Based on the nature of this issue with an error about multiple A records, it looks like appending an A record list would work.
https://github.com/kubernetes-incubator/external-dns/blob/master/docs/contributing/sources-and-providers.md

Here is the provider interface for the list of changes.
https://github.com/kubernetes-incubator/external-dns/blob/master/provider/aws.go#L208

It all seems to point to https://github.com/kubernetes-incubator/external-dns/blob/master/provider/aws.go#L302

[Promaethius]

This issue would be fixed by https://github.com/kubernetes-incubator/external-dns/pull/347

[keefbaker]

I seem to be running into something similar when I annotate a service with too many hostnames:

level=error msg="InvalidChangeBatch: The request contains an invalid set of changes for a resource record set

It seems to at this point stop external-dns updating records full stop. I don't know if it tries to batch too many together. Including txt records it's trying to batch 16.

I'm doing this as a work-round for traefik not automatically updating DNS records for ingresses, so instead I'm annotating the traefik service with the appropriate records.

I suppose mine is the opposite in some ways, multiple dns names for the same record. It's fine when it's about 4-5 but over that and the batch starts to fail.

[kallaikoi]

Hello, any news on this issue? (Support multiple IPs for ingress)
We're still unable to assing multiple IPs to the same A record triggered by a single ingress resource (AWS cloud provider).
I've been looking for some info about this issue on FAQ/Documentation, but I was unable to find out if this feature is not supported yet.

Thanks in advance.

time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A" 
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A" 
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A" 
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT" 
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT" 
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT" 
time="2018-04-17T12:39:50Z" level=error msg="InvalidChangeBatch: The request contains an invalid set of changes for a resource record set 'A hello.aws-stuff.es.'
    status code: 400, request id: 6b9d45ae-423c-11e8-b23a-5186740c5011" 

I just stumbled upon this PR https://github.com/kubernetes-incubator/external-dns/pull/493
I guess that PR is aimed to fix this issue.

[bitvector2]

Good question. This was being worked on many months ago. You might try reaching out in the Slack external-dns channel as well.

---

From: Jorge González [e-mail]
Sent: Tuesday, April 17, 2018 6:35 AM
To: kubernetes-incubator/external-dns
Cc: Steven Logue; Comment
Subject: Re: [kubernetes-incubator/external-dns] AWS provider doesn't support multiple IPs for ingress (#270)

Hello, any news on this issue? (Support multiple IPs for ingress)
We're still unable to assing multiple IPs to the same A record triggered by a single ingress resource.
I've been looking for some info about this issue on FAQ/Documentation, but I was unable to find out if this feature is no supported yet.
Thanks in advance.

time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A"
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A"
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es A"
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT"
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT"
time="2018-04-17T12:39:50Z" level=info msg="Desired change: CREATE hello.aws-stuff.es TXT"
time="2018-04-17T12:39:50Z" level=error msg="InvalidChangeBatch: The request contains an invalid set of changes for a resource record set 'A hello.aws-stuff.es.'
status code: 400, request id: 6b9d45ae-423c-11e8-b23a-5186740c5011"

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkubernetes-incubator%2Fexternal-dns%2Fissues%2F270%23issuecomment-381994577&data=02%7C01%7C%7Ce5c9a3aa26c14eeb9c6408d5a4682116%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636595689497563496&sdata=RnBbT5MtmtofIT7f%2Fv5USEFHBQPKXA399Sgn%2BBPxomI%3D&reserved=0, or mute the threadhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJAKlhG4nsUiE6YhWe66vG0e4JaRF4h3ks5tpe-ygaJpZM4OUQTV&data=02%7C01%7C%7Ce5c9a3aa26c14eeb9c6408d5a4682116%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636595689497563496&sdata=MSh50A4sVahvbDkTj%2BRmX4%2B%2BBdog2804Yjl7EWlyuHI%3D&reserved=0.

[kallaikoi]

This is added on 0.5.0 and working wonders for now.
I think you can close this.

[Raffo]

Closing this as for last comment. #493 fixes this.