External-dns: v0.7.3 regression from v0.7.2 when using istio-gateway source (all records DELETEd)

[jeffhubLR]

What happened:

Upgrading to v0.7.3 deleted all created DNS entries sourced from Istio gateway (--source=istio-gateway)
Downgrading back to v0.7.2 with no other changes recreated the entries deleted by v0.7.3

What you expected to happen:

Upgrade works with existing configuration.

How to reproduce it (as minimally and precisely as possible):

1. Have a working config using istio-gateway as source, external-dns v0.7.2
2. Upgrade to external-dns v0.7.3
3. Witness all entries created from istio-gateway source deleted by external-dns

External-dns config:

      --txt-owner-id=REDACTED
      --domain-filter=REDACTED
      --source=service
      --source=ingress
      --source=istio-gateway
      --provider=aws
      --aws-zone-type=public
      --registry=txt

External-dns logs from v0.7.2 before upgrade:

external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:13Z" level=info msg="config: {Master: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[] ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress istio-gateway] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[REDACTED] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:REDACTED TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile:}"
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:13Z" level=info msg="Instantiating new Kubernetes client"
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:13Z" level=info msg="Using inCluster-config based on serviceaccount-token"
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:13Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:22Z" level=info msg="Desired change: CREATE kafdrop.REDACTED A [Id: /hostedzone/REDACTED]"
...REDACTED...
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:32:23Z" level=info msg="24 record(s) in zone REDACTED. [Id: /hostedzone/REDACTED] were successfully updated"
external-dns-565d8dd4c7-zq8vn external-dns time="2020-08-10T19:33:23Z" level=info msg="All records are already up to date"

External-dns logs from v0.7.3 after upgrade:

external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:19Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s IstioIngressGatewayServices:[] ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress istio-gateway] Namespace: AnnotationFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[REDACTED] ExcludeDomains:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:REDACTED TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50}"
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:19Z" level=info msg="Instantiating new Kubernetes client"
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:19Z" level=info msg="Using inCluster-config based on serviceaccount-token"
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:19Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:28Z" level=info msg="Desired change: DELETE kafdrop.REDACTED A [Id: /hostedzone/REDACTED]"
...REDACTED...
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:57:29Z" level=info msg="24 record(s) in zone REDACTED. [Id: /hostedzone/REDACTED] were successfully updated"
external-dns-568764f9cb-48djw external-dns time="2020-08-10T19:58:28Z" level=info msg="All records are already up to date"

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): v0.7.3
• DNS provider: AWS Route53

[tariq1890]

can you share the Gateway that you're using?

[tariq1890]

can you also make sure you have --source=istio-gateway set when you're running the v0.7.3 instance?

[simone201]

I had been facing a pretty similar issue, but external-dns keeps looping with CREATE statements, as example:

time="2020-08-21T14:00:30Z" level=info msg="Instantiating new Kubernetes client"
time="2020-08-21T14:00:30Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2020-08-21T14:00:30Z" level=info msg="Created Kubernetes client https://10.100.0.1:443"
time="2020-08-21T14:00:39Z" level=info msg="Desired change: CREATE test.redacted.com A [Id: /hostedzone/REDACTED]"
time="2020-08-21T14:00:39Z" level=info msg="Desired change: CREATE test.redacted.com TXT [Id: /hostedzone/REDACTED]"
time="2020-08-21T14:00:40Z" level=info msg="2 record(s) in zone redacted.com. [Id: /hostedzone/REDACTED] were successfully updated"

The last 3 statements keep looping every minute, the first time it runs it correctly creates the entries, from the 2nd time on it deletes them and never recreates, or randomly creates them to delete them again.

Configuration is:

- --source=ingress
- --source=istio-gateway
- --domain-filter=redacted.com
- --provider=aws
- --policy=upsert-only
- --aws-zone-type=public
- --registry=txt
- --txt-owner-id=owner_id

Versions tested: 0.5.18, 0.7.2, 0.7.3
Provider: AWS Route 53
Source: Istio Ingress Gateway

The deployed Gateway is:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: test-gateway
  namespace: test-config
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - test.redacted.com
      port:
        name: grpc
        number: 80
        protocol: GRPC

[jeffhubLR]

@tariq1890
Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  annotations:
  labels:
    app: keycloak
    applications.argoproj.io/app-name: keycloak
    version: "1.0"
  name: keycloak-gateway
  namespace: keycloak
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - idp.REDACTED
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - idp.REDACTED
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: keycloak-cert
      mode: SIMPLE

external-dns args:

  - args:
    - --txt-owner-id=REDACTED
    - --domain-filter=REDACTED
    - --source=service
    - --source=ingress
    - --source=istio-gateway
    - --provider=aws
    - --aws-zone-type=public
    - --registry=txt

[tariq1890]

@jeffhubLR @simone201 Thank you!

Can you provide the version of your Istio control planes and the labels of your respective of istio-ingressgateway services?

[jeffhubLR]

Istio 1.4.10

Labels on keycloak gateway:

    labels:
      app: keycloak
      applications.argoproj.io/app-name: keycloak
      version: "1.0"

[tariq1890]

I meant your istio-ingressgateway Service

[jeffhubLR]

istio-ingressgateway svc:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: istio-ingressgateway
    applications.argoproj.io/app-name: istio-system
    istio: ingressgateway
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
spec:
  clusterIP: 100.71.141.24
  externalTrafficPolicy: Cluster
  ports:
  - name: status-port
    nodePort: 32378
    port: 15020
    protocol: TCP
    targetPort: 15020
  - name: http2
    nodePort: 32129
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 30109
    port: 443
    protocol: TCP
    targetPort: 443
  - name: kiali
    nodePort: 30303
    port: 15029
    protocol: TCP
    targetPort: 15029
  - name: prometheus
    nodePort: 32029
    port: 15030
    protocol: TCP
    targetPort: 15030
  - name: grafana
    nodePort: 30454
    port: 15031
    protocol: TCP
    targetPort: 15031
  - name: tracing
    nodePort: 30262
    port: 15032
    protocol: TCP
    targetPort: 15032
  - name: tls
    nodePort: 31089
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
  sessionAffinity: None
  type: LoadBalancer

[simone201]

@tariq1890

Using Istio 1.6.7 deployed with the Istio Operator on an AWS EKS Cluster.

Labels of istio-ingressgateway:

app=istio-ingressgateway
install.operator.istio.io/owning-resource=cluster-name-istiocontrolplane
install.operator.istio.io/owning-resource-namespace=istio-system
istio=ingressgateway
operator.istio.io/component=IngressGateways
operator.istio.io/managed=Reconcile
operator.istio.io/version=1.6.7
release=istio

[tariq1890]

@simone201 Thanks for sharing the labels, can you also share the label selector of the istio-ingressgateway service?

[simone201]

@tariq1890
Don't know how this could help but here it is:
Selector: app=istio-ingressgateway,istio=ingressgateway

If the external-dns keeps recreating stuff, it means that its state isn't fulfilled correctly from Route 53, i saw some similar issues on Cloudflare provider too but seems that they fixed it somehow...

[tariq1890]

@simone201 Yes, your issue is different from what @jeffhubLR Is facing

[tariq1890]

@jeffhubLR In the latest release of external-dns, we have a made a change where we allow Gateway-Service Mapping only if their selectors match. While I understand that this is breaking, this change is required as the intent of Gateway Selectors was to map to the pods and not services. In v0.7.2, we were mapping gateway selectors to services, which was why it is working for you previously.

I am hesitant to revert this behaviour as this is the recommended way forward. Is it possible for you to change your gateway label selector to app: istio-ingressgateway so that it matches with your istio-ingressgateway service's label selector?

[tariq1890]

@simone201 Could you open a different GH issue for your case?

[tariq1890]

@jeffhubLR Actually, if you can make sure your istio-ingressgateway service has the following label selector

Selector: app=istio-ingressgateway,istio=ingressgateway

then you wouldn't need to change your Gateway resource. Are you able to do that? You might need to redeploy your Istio control plane with these changes I think

[jeffhubLR]

Sorry for the delayed response. I haven't been able to test out with updated label selector. Once I do, I will post back here and close if everything works as expected. Thanks @tariq1890

[jeffhubLR]

Closing. Thanks for your patience. The updated selector on the istio-ingressgateway service works as expected.