Installer: Masters Cannot Complete Setup - Connection Refused when accessing 6443

[glennodickson]

I'm really really stuck trying to understand what issue is. I've attached Bundle Logs to assist hopefully the info below is adequate please let me know if you need more/other.

It would very appreciated if someone could assist please. Thanks.

Version

4.2.12

Platform:

baremetal on Proxmox 6.1 hypervisor

What happened?

• Boostrap starts correctly, but masters start and load except for a containers like SDN and the API container (port 6443).

• Seems like the master (and I think bootstap) has a problems getting to the container (port 6443) on bootstrap server as seen in both the Bootkube.log and kubelet.log files.

Bootkube.log File reports the following:

Jan 23 15:50:11 bootstrap bootkube.sh[1757]: E0123 15:50:11.132626       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://localhost:6443/api/v1/pods: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:11 bootstrap bootkube.sh[1757]: [#405] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:11 bootstrap bootkube.sh[1757]: [#406] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:11 bootstrap bootkube.sh[1757]: [#407] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:11 bootstrap bootkube.sh[1757]: [#408] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:11 bootstrap bootkube.sh[1757]: [#409] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Jan 23 15:50:12 bootstrap bootkube.sh[1757]: [#410] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused

Kubelet.log for Master1/2/3 report errors like:

controller.go:125] failed to ensure node lease exists, will retry in 200ms, error: Get https://api-int.test.fritz.box:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/master0?timeout=10s: context deadline exceeded

kubelet.go:2180] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network

Unable to write event: 'Patch https://api-int.test.fritz.box:6443/api/v1/namespaces/openshift-sdn/events/sdn-l95wf.15ec8fa571d26c46: unexpected EOF' (may retry after sleeping)

Jan 23 16:08:52 master0 hyperkube[2653]: E0123 16:08:52.262764    2653 kubelet_node_status.go:385] Error updating node status, will retry: error getting node "master0": Get https://api-int.test.fritz.box:6443/api/v1/nodes/master0?timeout=10s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Jan 23 16:08:52 master0 hyperkube[2653]: E0123 16:08:52.376945    2653 controller.go:125] failed to ensure node lease exists, will retry in 800ms, error: Get https://api-int.test.fritz.box:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/master0?timeout=10s: read tcp 192.168.179.80:49040->192.168.179.73:6443: use of closed network connection
Jan 23 16:08:52 master0 hyperkube[2653]: E0123 16:08:52.384846    2653 event.go:200] Unable to write event: 'Patch https://api-int.test.fritz.box:6443/api/v1/namespaces/openshift-sdn/events/sdn-l95wf.15ec8fa75bf6b57f: read tcp 192.168.179.80:49040->192.168.179.73:6443: use of closed network connection' (may retry after sleeping)

bootstrap.go:239] Unable to read existing bootstrap client config: invalid configuration: [unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory]

Jan 23 15:51:52 master0 hyperkube[2653]: E0123 15:51:52.346328    2653 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope
Jan 23 15:51:52 master0 hyperkube[2653]: E0123 15:51:52.474775    2653 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: nodes "master0" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope

remote_runtime.go:380] ExecSync 8a9dbd5eaf13083f90c87ce6e0b6960e4f396e468b79694953e3a59dbdd0647f 'test -f /etc/cni/net.d/80-openshift-network.conf' from runtime service failed: rpc error: code = Unknown desc = container is not created or running

remote_runtime.go:380] ExecSync 8a9dbd5eaf13083f90c87ce6e0b6960e4f396e468b79694953e3a59dbdd0647f 'test -f /etc/cni/net.d/80-openshift-network.conf' from runtime service failed: rpc error: code = Unknown desc = container is not created or running

What you expected to happen?

• Master start successfully

How to reproduce it (as minimally and precisely as possible)?

• Bootstrap successfully loads.
• Masters are started and subsequently reboot as expected to create the etcd service (port 2397)
• Bootstrap recognises the master etcd and successfully registers them.
• Master starts its various containers but its SDN restarts constantly.
• Logs Bootkube.log and kubelet.log files contain "connection refused" errors and others perhaps related.
• Masters constantly attempt to access container at port 6443 but never succeeds
• Bootstrap eventually cycles to reinstall cluster by pulling images again

Anything else we need to know?

• No firewall on LB

• Bundle Logs
  Log files:log-bundle-20200124001753.tar.gz

• LB
  config: haproxy.txt

• DNSMASQ
  config file: dnsmasq.txt
  host file: hosts.txt

/label platform/baremetal

[glennodickson]

Issue:
Containers taking too long to start and probes deemded SDN as unhealthly on master and therefore seems the kubelet-apisever could not start hence process

Solved:

• Needed to turn nested virtualisation on.
• Restarted bootstrap, masters and workers.

Works on Proxmox 6.1 now.

[Nurlan199206]

i have same issue on VirtualBox. Trying to up OKD with 1 master.

Pod starting takes longer than expected

Aug 23 08:29:11 localhost bootkube.sh[18132]: E0823 08:29:11.808554       1 reflector.go:251] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to watch *v1.Pod: Get https://localhost:6443/api/v1/pods?watch=true: dial tcp [::1]:6443: connect: connection refused
Aug 23 08:29:12 localhost bootkube.sh[18132]: E0823 08:29:12.810157       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://localhost:6443/api/v1/pods: dial tcp [::1]:6443: connect: connection refused
Aug 23 08:29:13 localhost bootkube.sh[18132]: E0823 08:29:13.811237       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://localhost:6443/api/v1/pods: dial tcp [::1]:6443: connect: connection refused

@vrutkovs it's possible to increase connection timeout to port 6443 when bootstraping?

log-bundle-20200823043328.tar.gz

[Nurlan199206]

finally installed. : )

[nmukkelli]

Hi @Nurlan199206 , I'm hitting the same issues, I'm using OpenShift 4.5.7 in vSphere using IPI installation. How did you end up going past issues. Could you please let me know?

[Sritechops]

I managed to fix this after giving bootstrap entries in loadbalancer.