Kubeadm: Kubernets kube-apiserver service not started after reboot system

[choudharykomal]

I have setup cluster with kubeadm its working fine and 6443 port is up. but after reboot my system cluster is not getting up.

please suggest, what should i do.

Steps:

node@node1:~$ sudo kubeadm init
[init] using Kubernetes version: v1.11.1
......

node@node1:~$ 
node@node1:~$  mkdir -p $HOME/.kube
node@node1:~$  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
node@node1:~$  sudo chown $(id -u):$(id -g) $HOME/.kube/config
node@node1:~$ 
node@node1:~$ 
node@node1:~$ kubectl get nodes
NAME      STATUS     ROLES     AGE       VERSION
node1     NotReady   master    4m        v1.11.1
node@node1:~$ ps -ef | grep 6443
root      5542  5503  8 13:17 ?        00:00:17 kube-apiserver --authorization-mode=Node,RBAC --advertise-address=172.16.2.171 --allow-privileged=true --client-ca-file=/etc/kubernetes/pki/ca.crt --disable-admission-plugins=PersistentVolumeLabel --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
node      6792  4426  0 13:20 pts/1    00:00:00 grep --color=auto 6443
node@node1:~$ 
node@node1:~$ 
node@node1:~$ 
node@node1:~$ sudo reboot
Connection to node1 closed by remote host.
Connection to node1 closed.
abc@xyz:~$ ssh node@node1
node@node1's password: 
node@node1:~$ kubectl get nodes
No resources found.
The connection to the server 172.16.2.171:6443 was refused - did you specify the right host or port?`enter code here`
node@node1:~$ 
node@node1:~$ ps -ef | grep 6443
node      7083  1920  0 13:36 pts/0    00:00:00 grep --color=auto 6443

I have tried multiple time but getting same issue.

please help

[neolit123]

did you install a pod network plugin?
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network

[choudharykomal]

i had installed this
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

[neolit123]

you should wait for the cluster to come back up after a few minutes. after setting up and rebooting type:
watch kubectl get nodes.
if the node doesn't ready up in 5 minutes, something is wrong.

[choudharykomal]

but cluster is not getting back after reboot.

node@node1:$ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
node1     Ready     master    10m       v1.11.1
node2     Ready     <none>    6m        v1.11.0
node3     Ready     <none>    6m        v1.11.0

node@node1:~$ sudo reboot
Connection to node1 closed by remote host.
Connection to node1 closed.
node@master:~$ ssh node@node1
node@node1:~$ kubectl get nodes
No resources found.
The connection to the server 172.16.2.171:6443 was refused - did you specify the right host or port?
node@node1:~$ 
node@node1:~$ ps -ef | grep 6443
node      7083  1920  0 13:36 pts/0    00:00:00 grep --color=auto 6443

[neolit123]

i cannot reproduce this problem. did you try waiting a few minutes?
watch kubectl get nodes

[choudharykomal]

yes. getting following error massage

The connection to the server 172.16.2.171:6443 was refused - did you specify the right host or port?

[neolit123]

any relevant errors from?
journalctl -xeu kubelet

[choudharykomal]

node@node1:$ journalctl -xeu kubelet
Jul 19 20:27:24 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jul 19 20:27:24 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jul 19 20:27:34 node1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Jul 19 20:27:34 node1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 481.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Automatic restarting of the unit kubelet.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Jul 19 20:27:34 node1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit kubelet.service has finished shutting down.
Jul 19 20:27:34 node1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is RESULT.
Jul 19 20:27:34 node1 kubelet[30675]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by 
Jul 19 20:27:34 node1 kubelet[30675]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by th
Jul 19 20:27:34 node1 kubelet[30675]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by 
Jul 19 20:27:34 node1 kubelet[30675]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by th
Jul 19 20:27:34 node1 kubelet[30675]: I0719 20:27:34.909672   30675 server.go:408] Version: v1.11.1
Jul 19 20:27:34 node1 kubelet[30675]: I0719 20:27:34.910073   30675 plugins.go:97] No cloud provider specified.
Jul 19 20:27:34 node1 kubelet[30675]: I0719 20:27:34.913803   30675 certificate_store.go:131] Loading cert/key pair from "/var/lib/kubelet/pki
Jul 19 20:27:34 node1 kubelet[30675]: I0719 20:27:34.966454   30675 server.go:648] --cgroups-per-qos enabled, but --cgroup-root was not specif
Jul 19 20:27:34 node1 kubelet[30675]: F0719 20:27:34.966998   30675 server.go:262] failed to run Kubelet: Running with swap on is not supporte
Jul 19 20:27:34 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Jul 19 20:27:34 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

[choudharykomal]

Thanks..

after swap off its working fine. is there any step for ignore swap ?

[neolit123]

is this an old error?

Jul 19 20:27:34 node1 kubelet[30675]: F0719 20:27:34.966998 30675 server.go:262] failed to run Kubelet: Running with swap on is not supporte...

having a swap enabled is not recommended, this can be force disabled with --fail-swap-on=false in the kubelet drop-in file.

what is your kubelet status before and after reboot:
sudo systemctl status kubelet

[neolit123]

after swap off its working fine. is there any step for ignore swap ?

please see above.

[choudharykomal]

Thank you so much