Kubespray: "localhost:8080 was refused" after rebooting master node

[grebois]

BUG REPORT

After rebooting the master node I get:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Steps to reproduce

Restart master node.

Environment:

• Cloud provider or hardware configuration:

Bare-metal

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 4.12.7-coreos x86_64
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1465.6.0
VERSION_ID=1465.6.0
BUILD_ID=2017-08-16-0012
PRETTY_NAME="Container Linux by CoreOS 1465.6.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Version of Ansible (ansible --version):

ansible 2.3.1.0
config file =
configured module search path = Default w/o overrides
python version = 2.7.10 (default, Oct 23 2015, 19:19:21) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]

Kubespray version (commit) (git rev-parse --short HEAD):

fc79056

Network plugin used:

Calico

[Starefossen]

From where exactly do you get the error message? From kubelet running on the nodes, one of the Pods in Kubernetes, or Ansible when running the playbook?

[grebois]

From within the nodes or the master, running;

$ kubectl get nodes
2017-09-05 10:52:46.980000 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-05 10:52:46.980190 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-05 10:52:46.980287 I | proto: duplicate proto type registered: google.protobuf.Timestamp
The connection to the server localhost:8080 was refused - did you specify the right host or port?

[mattymo]

@grebois I deployed on coreos-stable-1465-6-0-v20170817 and rebooted one of the masters. It takes approximately 2 minutes after reboot for kube-apiserver to get launched and start accepting queries. There is a significant delay because the following actions take place automatically:
1 - systemd starts docker and kubelet
2 - kubelet tries to connect to apiserver if it can first
3 - the connection times out and kubelet decides to start static pods on its own
4 - kube-apiserver tries to connect to etcd
5 - etcd was restarted by docker automatically and takes some time to reassemble quorum
6 - finally kube-apiserver has a connection to etcd and will start serving requests

I observed the same behavior and timing on Ubuntu. This is not a bug, in my opinion.

If you still have issues running kubectl get nodes after the host is up for 5 minutes, please provide complete output from the following command:

journalctl -u kubelet.service

[grebois]

@mattymo will try this again on a new cluster and get back to you, thanks!

[grebois]

@mattymo

core@int00web11 ~ $ uptime
 12:44:15 up 15 min,  1 user,  load average: 0.09, 0.12, 0.16
core@int00web11 ~ $ kubectl get nodes
2017-09-06 12:44:18.777759 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-06 12:44:18.777846 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-06 12:44:18.777868 I | proto: duplicate proto type registered: google.protobuf.Timestamp
The connection to the server localhost:8080 was refused - did you specify the right host or port?
core@int00web11 ~ $ journalctl --no-pager -u kubelet.service > kubelet.log

kubelet.log.zip

[grebois]

@mattymo would you mind re-opening this issue, I'm still experiencing the same behaviour with different versions and different configurations, couldn't find the root cause yet. Thanks in advance.

[Fireworks643]

DO this:
(on Centos7+ and, i believe on xenial+ should worked)

$sysctl net.bridge-bridge=nf-call-iptables=1
OR $_echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables_
$sysctl --system

[jvleminc]

I had the same issue after a clean install of kubespray, using user _ubuntu_ on my nodes. The following fixed the issue and allowed executing _kubectl_ commands:

Execute in master node:

sudo cp /etc/kubernetes/admin.conf $HOME/ && sudo chown $(id -u):$(id -g) $HOME/admin.conf && export KUBECONFIG=$HOME/admin.conf

I guess this should be added to the install scripts.

[AngelOnFira]

I'm not sure if this was fixed in the commit above, but I had to still run @jvleminc's fix for kubectl to work on Ubuntu. I'd be interested in making the fix myself, but I won't have time for the next few weeks or so.