External-dns: External DNS does not delete AWS aliases

[kostrahb]

I have installed external DNS v0.5.14 on our kubernetes cluster in AWS (values bellow). Right now we are using AWS aliases (not CNAME) for correct resolution to internet-facing load balancer which is pointed to nginx-ingress. So we annotate each ingress which should be accessible over internet with following statements:

kubernetes.io/ingress.class: nginx-external
external-dns.alpha.kubernetes.io/target: external-nlb.domain
external-dns.alpha.kubernetes.io/alias: "true"

The creation of route53 record works great, however if I delete the ingress, external-dns is unable to delete the records and yields following errors:

time="2019-07-18T08:06:21Z" level=error msg="Failed to submit all changes for the following zones: [/hostedzone/Z3UTxxx]"
time="2019-07-18T08:07:20Z" level=info msg="Desired change: DELETE hello.domain CNAME"
time="2019-07-18T08:07:20Z" level=info msg="Desired change: DELETE hello.domain TXT"
time="2019-07-18T08:07:21Z" level=error msg="InvalidChangeBatch: [Tried to delete resource record set [name='hello.domain.', type='CNAME'] but it was not found]\n\tstatus code: 400, request id: d26532cc-10c3-45b1-9901-1a003717bb46"

I believe this is a bug because there is no way for external-dns to tell whether the record was alias or CNAME after ingress deletion. Would it be possible to either add some marker to TXT record which which would assure that correct type is deleted or try to delete both CNAME and alias after ingress deletion?

Values:

annotationFilter: kubernetes.io/ingress.class=nginx-external
aws:
  accessKey: AKIAxxx
  credentialsPath: /.aws
  region: eu-central-1
  roleArn: ""
  secretKey: xxx
domainFilters: []
image:
  name: registry.opensource.zalan.do/teapot/external-dns
  tag: v0.5.14
interval: 1m
logLevel: info
podAnnotations:
  external-dns.alpha.kubernetes.io/target: nlb.domain
policy: sync
provider: aws
rbac:
  create: true
registry: txt
resources:
  limits:
    memory: 50Mi
  requests:
    cpu: 10m
    memory: 50Mi
sources:
- service
- ingress
txtOwnerId: Z3Uxxx
zoneIdFilters:
- Z3Uxxx

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[k]

I ran into this problem today, it would be a nice addition.

[frittentheke]

/remove-lifecycle rotten

[spohner]

It looks like deletion of ALIAS records works if referencing a amazonaws.com address directly. Then external-dns looks up the hostedZoneId from a set of hardcoded values and deletes the ALIAS. If the address is not found among the hardcoded values the record is not considered an ALIAS and it fails to delete the record because it does not match with a record in Route53.

I think we can store the hostedZoneId with the endpoints and make some changes to how external-dns determines to use an alias or not and apply the hostedZoneId without hardcoded values. This apply to the aws provider only. I may be able to make a PR on this. Any comments?

[devkid]

@kostrahb Could you try if https://github.com/kubernetes-sigs/external-dns/pull/1356 fixes this problem for you?

I believe this is a bug because there is no way for external-dns to tell whether the record was alias or CNAME after ingress deletion.

In fact this is very easy. In Route 53, the alias is either a CNAME or an alias record, so external-dns should have this information at hand already. The PR I linked implements this by adding the provider-specific property "alias" to endpoints extracted from alias records in the zone.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[zaro]

We are affected by the same problem.
We use external DNS for preview environments, and this forces us to manually go and delete records every now and then when AWS starts complaining about the number of records.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wmcnamee-tunein]

/remove-lifecycle stale

[seanmalloy]

/kind bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[seanmalloy]

/remove-lifecycle stale