External-dns: AWS Route53 - InvalidChangeBatch: RDATA character limit of 32000 exceeded.

[estahn]

We have been running external-dns in upsert-only for quite a while and wanted to change to sync. Due to a large number of subdomains we have accumulated (>6000) we receive the following error message:

time="2018-04-03T14:16:44Z" level=error msg="InvalidChangeBatch: RDATA character limit of 32000 exceeded.
    status code: 400, request id: 982287bd-3749-11e8-9390-8922955173be"

[ideahitme]

For reference: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html#limits-api-entities

Furthermore, seems like we have wrongly hardcoded maximum number of records in the change set per domain. As per link above it is 1000, while we are using 4000, as can be seen here:

https://github.com/kubernetes-incubator/external-dns/blob/70cdbb93620bad7c0e9b266e4d9f104dd25aecb5/provider/aws.go#L36

@estahn thanks for the issue. In your case, you can either manually drop the subdomains or wait until the fix for this issue is released.

IMO I see this as a corner-case and most likely only happens in cases described by op. To fix the problem we can decrease the number of records in the change batch to some lesser value. I think making AWS provider even more complicated by calculating byte size of payloads is not something we would like to do.

[estahn]

@ideahitme Thanks for your response and I resolved the issue by manually deleting the subdomains.

If we could make the batch size configurable that should work. Otherwise, 1000 should be a sufficient default.

[peterbale]

We saw the same issue with ~500 updates (~1000 inc TXT records). Allowing this value to be configurable would help massively as payload size can vary with each use case.

[ideahitme]

Agree with @estahn on this one. Let's make batch size configurable.

@peterbale @estahn do you want to implement this ?