Kubespray: dns not working within pods

[dimthe]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG

these are my settings

# DNS configuration.
# Kubernetes cluster name, also will be used as DNS domain
cluster_name: cluster.local
# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
ndots: 2
# Can be dnsmasq_kubedns, kubedns or none
dns_mode: kubedns
# Can be docker_dns, host_resolvconf or none
resolvconf_mode: docker_dns
# Deploy netchecker app to verify DNS resolve as an HTTP service
deploy_netchecker: false

this fails

 kubectl exec -it glusterfs-cbhfp  /bin/bash
[root@centos03 /]# nslookup centos02
Server:     213.133.98.98
Address:    213.133.98.98#53



** server can't find centos02: NXDOMAIN

this seems to be working

this seems to be working

kubectl exec -ti busybox nslookup kubernetes
Server:    10.233.0.3
Address 1: 10.233.0.3 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local

Environment:

• Cloud provider or hardware configuration:
  Centos 7.4 on bare metal
• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  centos 7

• Version of Ansible (ansible --version):
  2.4

Kubespray version (commit) (git rev-parse --short HEAD):

2.2.0
Network plugin used:
calico

Copy of your inventory file:

[root@centos01 kubernetes]# cat /root/kubespray-2.2.0/inventory/inventory
# ## Configure 'ip' variable to bind kubernetes services on a
# ## different ip than the default iface
centos01 ansible_ssh_host=94.130.134.158
centos02 ansible_ssh_host=94.130.134.159
centos03 ansible_ssh_host=94.130.134.160
centos04 ansible_ssh_host=94.130.134.161


[kube-master]
centos01
centos02

[etcd]
centos01
centos02
centos03

[kube-node]
centos02
centos03
centos04
centos05

[k8s-cluster:children]
kube-node
kube-master

Command used to invoke ansible:

ansible-playbook -i inventory/inventory cluster.yml -vvv

some more failed and succeeded nslookups

kubectl exec -ti busybox nslookup kubernetes
Server:    10.233.0.3
Address 1: 10.233.0.3 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local



this will not


kubectl exec -ti glusterfs-ns1nz nslookup kubernetes
Server:     213.133.98.98
Address:    213.133.98.98#53

** server can't find kubernetes: NXDOMAIN

[kschroed]

Having similar issue but using flannel plugin. I'm sure some config option I've not set properly but not having much luck getting it working.

[woopstar]

Please post the content of /etc/resolv.conf of a node where it does not work
the content of the kube-apiserver manifest on the master
the content of /etc/systemd/system/docker.service.d/docker-dns.conf

[kschroed]

Resolution from the node itself works fine.

"[centos@k8s-cluster1-k8s-node-nf-3 ~]$ cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
; generated by /usr/sbin/dhclient-script
search cisco.com
nameserver 64.102.6.247
nameserver 171.70.168.183

[centos@k8s-cluster1-k8s-node-nf-3 ~]$ cat /etc/systemd/system/docker.service.d/docker-dns.conf
[Service]
Environment="DOCKER_DNS_OPTIONS=
--dns 10.233.0.3 --dns 64.102.6.247 --dns 171.70.168.183
--dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-search cisco.com
--dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2

It would appear to me that I cannot connect to the kube dns service
[centos@k8s-cluster1-bastion-1 ~]$ kubectl exec -ti nginx-ingress-controller-86c8447687-dtbt9 -- ping 10.233.0.3
PING 10.233.0.3 (10.233.0.3): 56 data bytes

[centos@k8s-cluster1-bastion-1 ~]$ kubectl exec -ti nginx-ingress-controller-86c8447687-dtbt9 -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=45 time=2.976 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=45 time=2.244 ms

[HitDaCa]

Having similar issues with default calico and default config setup on debian 9 (kubespray 2.5). DNS on nodes works fine, however external DNS on pods does not work.

[atomicleads]

same issue on ubuntu. Internal hostnames resolves fine, but external, like github.com all resolves to the same google ip.
@HitDaCa have you managed to solve this?

[HitDaCa]

Yes, in the end I got it working with fresh install using Debian 9, flannel
as the network provider, kube_dns and debian defined as the bootstrap os
(kubespeay tag 2.5, not master branch). No chance with calico...

On Fri, 11 May 2018, 6:08 PM atomicleads, [e-mail] wrote:

same issue on ubuntu. Internal hostnames resolves fine, but external, like
github.com all resolves to the same google ip.
HitDaCa have you managed to solve this?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/kubespray/issues/1918#issuecomment-388269894,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACLzEjMMg4c3fd49WH0YvdbltyQAoc6cks5txSrhgaJpZM4QOp1O
.

[tjtelan]

I was having trouble with this too, but I got it working without a fresh OS install.

I noticed that in the docs for the dns stack, the default dns_mode specified is dnsmasq_kubedns [1]. But this differs from the default dns_mode in the sample inventory, which is kubedns [2]:

[1] https://github.com/kubernetes-incubator/kubespray/blob/master/docs/dns-stack.md
[2] https://github.com/kubernetes-incubator/kubespray/blame/master/inventory/sample/group_vars/k8s-cluster.yml

When I changed my dns_mode to dnsmasq_kubedns, dns resolution behavior in pods began resolving external dns as expected.

(kubespray 2.5, flannel, ubuntu 16.04)

[jhuebner79]

It started working for me when I removed search and domain lines from my node's resolv.conf entirely.

[sunilkcs]

flannel plugin

Which file i need to modify to change to flannel from calico ? Please help