External-dns: "No endpoints could be generated from ingress" in AWS with nginx-ingress and route53

[kerin]

I'm trying to set up external-dns with nginx-ingress, both installed via Helm, and no matter what I do external-dns just gives the following log output (loglevel debug)

time="2018-02-05T17:12:32Z" level=info msg="config: &{Master: KubeConfig: Sources:[ingress] Namespace: AnnotationFilter: FQDNTemplate: Compatibility: PublishInternal:true Provider:aws GoogleProject: DomainFilter:[apps.cluster-1.kops.integration.dsd.io.] AWSZoneType:public AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: CloudflareProxied:false InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InMemoryZones:[] Policy:upsert-only Registry:txt TXTOwnerID:cluster-1.kops.integration.dsd.io TXTPrefix: Interval:1m0s Once:false DryRun:false LogFormat:text MetricsAddress::7979 LogLevel:debug}" 
time="2018-02-05T17:12:32Z" level=info msg="Connected to cluster at https://100.64.0.1:443" 
time="2018-02-05T17:12:32Z" level=debug msg="No endpoints could be generated from ingress default/ingress-test" 

my external-dns helm config:

## This controls which types of resource external-dns should 'watch' for new
## DNS entries.
sources:
  - ingress

## The DNS provider where the DNS records will be created (options: aws, google, inmemory, azure )
provider: aws

# AWS Access keys to inject as environment variables
aws:
  region: "eu-west-1"
  # Filter for zones of this type (optional, options: public, private)
  zoneType: "public"

## Limit possible target zones by domain suffixes (optional)
domainFilters:
  - apps.cluster-1.kops.integration.dsd.io

# When using the TXT registry, a name that identifies this instance of ExternalDNS
txtOwnerId: "cluster-1.kops.integration.dsd.io"

rbac:
  ## If true, create & use RBAC resources
  ##
  create: true
  # Beginning with Kubernetes 1.8, the api is stable and v1 can be used.
  apiVersion: v1

  ## Ignored if rbac.create is true
  ##
  serviceAccountName: default

logLevel: debug
publishInternalServices: true

which results in these args to the pod:

  - args:
    - --log-level=debug
    - --publish-internal-services
    - --domain-filter=apps.cluster-1.kops.integration.dsd.io.
    - --policy=upsert-only
    - --provider=aws
    - --txt-owner-id=cluster-1.kops.integration.dsd.io
    - --source=ingress
    - --aws-zone-type=public

my nginx-ingress Helm config - the nginx service has type=LoadBalancer, and the ELB has been created:

controller:
  replicaCount: 3

rbac:
  create: true
  serviceAccountName: default

and this is the test Ingress and Deployment I'm trying (taken from the external-dns docs):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-test
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: nginx.apps.cluster-1.kops.integration.dsd.io
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http

My nodes have an IAM policy with route53 permissions, the apps.cluster-1.kops.integration.dsd.io zone has been created in route53, and I've searched issues here and tried adding various annotations to Ingresses and the nginx-ingress Service, but nothing has any effect. What am I doing wrong?

[ideahitme]

Please share the manifest of nginx-ingress controller and also please do kubectl get ingress -o yaml ingress-test. The log message suggests that ingress-test does not define any targets to use, and this value should be populated by the ingress controller.

[njuettner]

Might be related #460

[olemarkus]

I am experiencing the same issue. The only difference between a setup that I have that is working is the domain filter and that RBAC is enabled.

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx-internal"},"name":"nginx","namespace":"nginx-ingress-internal"},"spec":{"rules":[{"host":"testingress.testing.domain.io","http":{"paths":[{"backend":{"serviceName":"nginx","servicePort":80}}]}}]}}
      kubernetes.io/ingress.class: nginx-internal
    creationTimestamp: 2018-02-19T16:18:44Z
    generation: 1
    name: nginx
    namespace: nginx-ingress-internal
    resourceVersion: "2639780"
    selfLink: /apis/extensions/v1beta1/namespaces/nginx-ingress-internal/ingresses/nginx
    uid: 8eb78b1b-1590-11e8-81bd-062d096fbfa0
  spec:
    rules:
    - host: testingress.testing.domain.io
      http:
        paths:
        - backend:
            serviceName: nginx
            servicePort: 80
  status:
    loadBalancer:
      ingress:
      - hostname: internal-aa68dc8d5158811e881bdg42d096fbfa-2062718187.eu-central-1.elb.amazonaws.com
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[linki]

@kerin As a first step make sure that the status section of your Ingress is populated by your IngressController. It should look similar to what @olemarkus pasted.

[linki]

@olemarkus could you please also paste your ExternalDNS version and config (first log line without any credentials).

Your Inrgess looks good. Check your logs and if it also says No endpoints could be generated from ingress <ingressname> then it's not RBAC. It could be the domain filter but let's discuss further once you pasted your config.

[olemarkus]

I killed the external-dns pod and the DNS entry was created on the first run. I did have another ingress that wasn't created then, but this one had an incorrect ingress class so it never got its loadbalancer.

Thanks for your time. Will let you know if I experience this again in the future.

[hjacobs]

Closing this as the issue seems to be resolved, feel free to comment if there are still problems.

[derekperkins]

@hjacobs Same story, I'm using the nginx ingress and external-dns installed via helm and am getting this same error. I checked the status section of my ingress and it is empty. Deleting the external-dns pod / redeploying did not solve my problem.

  "status": {
    "loadBalancer": {
      "ingress": [
        {}
      ]
    }
  }

helm chart: external-dns-0.6.1
external-dns version: 0.5.0
provider: cloudflare
k8s version: 1.10.3
hosted on AKS
nginx ingress version: 0.15.0
helm chart: nginx-ingress-0.22.1
RBAC: disabled

[sc250024]

EDIT: Found this issue (https://github.com/kubernetes/ingress-nginx/issues/2085), adding controller.publishService.enabled=true to the --set flag did the trick 👍

Having the same issues as @derekperkins , Ingress resource looks fine, except that status.loadBalancer.ingress is empty.

@derekperkins How are you creating your NGINX Ingress controller? I'm using the following helm command:

helm install \
--name nginx-ingress \
--namespace kube-system \
--set \
controller.extraArgs.default-server-port=8282,\
controller.hostNetwork=true,\
controller.image.pullPolicy=Always,\
controller.kind=DaemonSet,\
controller.metrics.enabled=true,\
controller.service.externalTrafficPolicy=Local,\
controller.stats.enabled=true,\
serviceAccount.create=true \
stable/nginx-ingress

Some explanation:

• controller.extraArgs.default-server-port=8282 is set because port 8181 interferes with kube2iam, also running as a DaemonSet
• controller.hostNetwork=true allows preservation of the source IP address.
• controller.service.externalTrafficPolicy=Local also needed for source IP preservation

[derekperkins]

@sc250024 Thanks for the link, that solved my issue too.

[yurrriq]

I still have this issue periodically, even with controller.publishService.enabled=true. Any further troubleshooting tips? The only solution we've found is to restart the nginx controller, which is not ideal, because that, in turn, deletes and recreates all associated DNS records, meaning our deployments are inaccessible for a few minutes.

[derekperkins]

@yurrriq Where are you running? On Azure / AKS, it doesn't always publish the load balancer ip correctly.

[yurrriq]

@derekperkins, AWS (via kops). It's smelling like an nginx-ingress problem, rather than external-dns.

[derekperkins]

I'm just pointing out that it may also not be an nginx-ingress problem, but an issue with the cloud provider not pushing the data back into the ingress in a timely fashion. On Google, it has worked perfectly every time. On Azure, sometimes it can take hours to propagate. I don't have any experience on AWS.

[okgolove]

Have same problem with internal ELB on AWS.
My nginx ingress controller service:
```metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

[okgolove]

Try to add --publish-service option. It helped me.
https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/static-ip#acquiring-an-ip