Kubespray: nodelocaldns pods listen tcp 169.254.25.10:53: bind: address already in use

[oleg-filiutsich]

Environment:

• Cloud provider or hardware configuration:
  Vsphere

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 4.4.0-131-generic x86_64
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Version of Ansible (ansible --version):
  ansible-playbook 2.7.8
  config file = /opt/ansible/kubespray-2.10.4/ansible.cfg
  configured module search path = ['/opt/ansible/kubespray-2.10.4/library']
  ansible python module location = /usr/local/lib/python3.5/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]

Kubespray version (commit) (git rev-parse --short HEAD):
2.10.4

Network plugin used:
default

Copy of your inventory file:
[all]

gl-dev-k8s-m1-msqo1 ansible_host=192.168.177.139

gl-dev-k8s-n1-msqo1 ansible_host=192.168.177.140
gl-dev-k8s-n2-msqo1 ansible_host=192.168.177.141
gl-dev-k8s-n3-msqo1 ansible_host=192.168.177.142

[kube-master]
gl-dev-k8s-m1-msqo1

[etcd]
gl-dev-k8s-m1-msqo1

[kube-node]
gl-dev-k8s-n1-msqo1
gl-dev-k8s-n2-msqo1
gl-dev-k8s-n3-msqo1

[k8s-cluster:children]
kube-master
kube-node

[all:vars]
ansible_user='kubernetes-configure-user'
ansible_ssh_private_key_file='/home/kubernetes-configure-user/.ssh/id_rsa'

docker_dns_servers_strict=false

kubelet_authorization_mode_webhook=true

Command used to invoke ansible:
ansible-playbook -b -i inventory/mycluster/hosts.ini cluster.yml

Output of ansible run:

              PLAY RECAP *********************************************************************
              gl-dev-k8s-m1-msqo1        : ok=412  changed=114  unreachable=0    failed=0   
              gl-dev-k8s-n1-msqo1        : ok=268  changed=73   unreachable=0    failed=0   
              gl-dev-k8s-n2-msqo1        : ok=268  changed=73   unreachable=0    failed=0   
              gl-dev-k8s-n3-msqo1        : ok=268  changed=73   unreachable=0    failed=0   
              localhost                  : ok=1    changed=0    unreachable=0    failed=0   

              Monday 15 July 2019  16:35:38 +0300 (0:00:00.551)       0:24:57.357 *********** 
              =============================================================================== 
              download : file_download | Download item ------------------------------ 160.20s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 83.57s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 66.91s
              container-engine/docker : ensure docker packages are installed --------- 61.94s
              download : container_download | download images for kubeadm config images -- 44.52s
              download : file_download | Download item ------------------------------- 41.77s
              gather facts from all instances ---------------------------------------- 33.48s
              download : file_download | Download item ------------------------------- 29.70s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 29.02s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 27.34s
              kubernetes/master : kubeadm | Initialize first master ------------------ 26.59s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 19.64s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 18.47s
              download : file_download | Download item ------------------------------- 18.33s
              kubernetes/kubeadm : Join to cluster ----------------------------------- 17.20s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 13.39s
              container-engine/docker : ensure docker-ce repository is enabled ------- 12.94s
              kubernetes/preinstall : Install packages requirements ------------------ 12.00s
              download : container_download | Download containers if pull is required or told to always pull (all nodes) -- 11.75s
              etcd : Configure | Check if etcd cluster is healthy -------------------- 10.65s

Anything else do we need to know:

# cat /etc/resolv.conf 
nameserver 10.1.1.40
nameserver 10.1.1.20
nameserver 127.0.0.1

✗ kubectl -n kube-system logs nodelocaldns-bslck 
2019/07/15 14:20:57 2019-07-15T14:20:57.671Z [INFO] Tearing down
2019/07/15 14:20:57 2019-07-15T14:20:57.795Z [INFO] Setting up networking for node cache
listen tcp 169.254.25.10:53: bind: address already in use

[woopstar]

You have already something else listening on *:53 on the host.

Try do a netstat -lnp to check if it's true.

[oleg-filiutsich]

Can't check cause delete cluster with nodelocaldns and setup new with enable_nodelocaldns: false
With enable_nodelocaldns: false kube dns works.

[ftab]

This happened to me too, but isn't a kubespray issue in particular. I had to disable systemd-resolved because I wanted dnsmasq instead, and then needed to set dnsmasq to bind only on the ethernet port so that nodelocaldns could bind to the virtual interface nodelocaldns

e.g. if you're using dnsmasq and your ethernet port is eno1 (according to ip a s):

cat <<EOF | sudo tee /etc/dnsmasq.d/bind-ethernet.conf
interface=eno1
bind-interfaces
EOF
sudo systemctl restart dnsmasq

Hope that helps if you decide to go back to nodelocaldns in the future

[woopstar]

It is also fixed in master branch

[oleg-filiutsich]

Thank you guys.
I'll try it if catch the problem again.