Aws-load-balancer-controller: dial tcp: lookup tagging.us-east-1.amazonaws.com on 10.100.0.10:53: no such host

[moris1amar]

Hi,
when I deploy on AWS Eks Cluster (eks_consul) the aws-alb-ingress-controller, I am facing the issue:

I0118 10:46:07.170763       1 api.go:159] Request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  ResourcesPerPage: 50,  TagFilters: [{      Key: "kubernetes.io/role/internal-elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/eks_consul",      Values: ["owned","shared"]    }]}
E0118 10:46:07.175401       1 request_pagination.go:105] Failed request: tagging/GetResources, Payload: {  ResourceTypeFilters: ["ec2"],  ResourcesPerPage: 50,  TagFilters: [{      Key: "kubernetes.io/role/internal-elb",      Values: ["","1"]    },{      Key: "kubernetes.io/cluster/eks_consul",      Values: ["owned","shared"]    }]}, Error: RequestError: send request failed
caused by: Post https://tagging.us-east-1.amazonaws.com/: dial tcp: lookup tagging.us-east-1.amazonaws.com on 10.100.0.10:53: no such host
E0118 10:46:07.175523       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: RequestError: send request failed\ncaused by: Post https://tagging.us-east-1.amazonaws.com/: dial tcp: lookup tagging.us-east-1.amazonaws.com on 10.100.0.10:53: no such host"  "Controller"="alb-ingress-controller" "Request"={"Namespace":"2048-game","Name":"2048-ingress"}

kubectl edit deployment.apps/eks-aws-alb1-aws-alb-ingress-controller -n kube-system

...
        - --cluster-name=eks_consul
        - --ingress-class=alb
        - --aws-api-debug
...

where "eks_consul" is my EKS cluster name.

So, AWS does not create the ALB service ...
I don't understand the problem with the tag

Thanks
Maurice

[moris1amar]

Also, I have

kubectl edit ing 2048-ingress -n 2048-game

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/tags: app=2048,Environment=testing,AWSAlbIngressController=true
    alb.ingress.kubernetes.io/target-type: ip
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"alb.ingress.kubernetes.io/scheme":"internet-facing","kubernetes.io/ingress.class":"alb"},"labels":{"app":"2048-ingress"},"name":"2048-ingress","namespace":"2048-game"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"service-2048","servicePort":80},"path":"/*"}]}}]}}
    kubernetes.io/ingress.class: alb
    kubernetes.io/ingress.subnets: subnet-5666df21,subnet-4907c262
  creationTimestamp: 2019-01-17T13:32:02Z
  generation: 1
  labels:
    app: 2048-ingress
  name: 2048-ingress
  namespace: 2048-game
  resourceVersion: "2115642"
  selfLink: /apis/extensions/v1beta1/namespaces/2048-game/ingresses/2048-ingress
  uid: 45b8047a-1a5c-11e9-bde7-0a2ea8adb6ba
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: service-2048
          servicePort: 80
        path: /*
status:
  loadBalancer: {}

thanks for any help
Maurice

[M00nF1sh]

Hi,
Your ingress spec is fine.

From the error message, there is some problem with kube-system/kube-dns in your cluster(it's not the controller's problem).
Did you setup some networkACL or securityGroup in your VPC that blocked out-going DNS requests?

[moris1amar]

thanks, I need to check the issue with my kubeDNS and update

[M00nF1sh]

@moris1amar
Just encountered same issue, and digged it out.
Are you creating using eksctl with privateNetworking==true?
If so, there is bug that node-to-node community is blocked, so these pods on worker node cannot talk to kubedns/coredns (also on worker nodes).
Adding a inbound UDP/53->nodeSecurityGroup rule on your nodeSecurityGroup will resolve it 😄
Though it should be fixed in eksctl

[hobti01]

I also encountered this situation and it was not due to security groups. Finally, I discovered that environment variable AWS_VPC_K8S_CNI_EXTERNALSNAT must be set true on the aws-node DaemonSet if workers are run in a private subnet and use a NAT gateway in a private subnet to access the internet.

```
$ kubectl edit -n kube-system ds aws-node
...
containers:
- env:
# IMPORTANT When workers run in a private subnet and use a NAT gateway
# This must be true to use the external SNAT, not SNAT within CNI
# https://docs.aws.amazon.com/eks/latest/userguide/external-snat.html
- name: AWS_VPC_K8S_CNI_EXTERNALSNAT
value: "true"
...

[iantype]

@hobti01 do you recall if you had to do anything special after that update? Or did it work in real time after you made the change?

[hobti01]

After editing the DaemonSet, nodes were registered in real time as the Pod on each node was replaced.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[KSVarun]

anybody tried with kops, am facing the same issue, I am using kops to create the cluster