Kubespray: Cluster Version is not upgrading

[mhabicht]

Not sure this is a bug or feature.

Attempting to upgrade my Kuberenetes cluster from v1.11.3 to v1.11.5 using the upgrade-cluster.yml.

Is there something I need to add to the upgrade-cluster.yml?

After running I am still on v1.11.3

root@avinetarchkm-anp1:/etc/kubernetes/manifests# kubectl get nodes
NAME STATUS ROLES AGE VERSION
avinetarch-anp1 Ready node 69d v1.11.3
avinetarch-anp2 Ready node 69d v1.11.3
avinetarch-anp3 Ready node 69d v1.11.3
avinetarch-anp4 Ready node 69d v1.11.3
avinetarch-anp5 Ready node 69d v1.11.3
avinetarchkm-anp1 Ready master 69d v1.11.3
avinetarchkm-anp2 Ready master 69d v1.11.3

Environment:

• Bare metal dell servers

• **OS Ubuntu 16.04LTS

• root@avinetarchlb-anp1:~/kubespray# ansible --version
  ansible 2.6.4
  config file = /root/kubespray/ansible.cfg
  configured module search path = [u'/root/kubespray/library']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]

root@avinetarchlb-anp1:~/kubespray# git rev-parse --short HEAD
e3d562b

Network plugin used: Calico

Copy of your inventory file:
root@avinetarchlb-anp1:~/kubespray# more inventory/mycluster/hosts.ini
-# ## Configure 'ip' variable to bind kubernetes services on a
-# ## different ip than the default iface
-# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the v
alue, or can set the empty string value.
avinetarchkm-anp1 ansible_host=192.168.51.2 ip=192.168.51.2 etcd_member_name=etcd1
avinetarchkm-anp2 ansible_host=192.168.51.3 ip=192.168.51.3 etcd_member_name=etcd2
avinetarchkm-anp3 ansible_host=192.168.51.4 ip=192.168.51.4 etcd_member_name=etcd3
avinetarch-anp1 ansible_host=192.168.51.7 ip=192.168.51.7
avinetarch-anp2 ansible_host=192.168.51.8 ip=192.168.51.8
avinetarch-anp3 ansible_host=192.168.51.9 ip=192.168.51.9
avinetarch-anp4 ansible_host=192.168.51.10 ip=192.168.51.10
avinetarch-anp5 ansible_host=192.168.51.11 ip=192.168.51.11
-# ## configure a bastion host if your nodes are not directly reachable
-# bastion ansible_host=x.x.x.x ansible_user=some_user
[kube-master]
avinetarchkm-anp1
avinetarchkm-anp2
avinetarchkm-anp3
[etcd]
avinetarchkm-anp1
avinetarchkm-anp2
avinetarchkm-anp3
[kube-node]
avinetarch-anp1
avinetarch-anp2
avinetarch-anp3
avinetarch-anp4
avinetarch-anp5

[k8s-cluster:children]
kube-master
kube-node

Command used to invoke ansible:
ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.ini -e kube_version=v1.11.5 -u root --ask-pass -b --become-user=root

[micah]

I just attempted this same upgrade, but the upgrade fails with this message:

TASK [download : Download items] **********************************************************************************************************************************************************************************************************************************************************
Wednesday 05 December 2018  09:25:17 -0500 (0:00:00.040)       0:05:04.666 **** 
fatal: [kubeadmin]: FAILED! => {"msg": "'dict object' has no attribute u'1.11.5'"}

Did you get the same result when you ran the upgrade-cluster.yml playbook?

[micah]

You need to add the checksums to the download role.

Downloaded locally these two files:

https://storage.googleapis.com/kubernetes-release/release/v1.11.5/bin/linux/amd64/kubeadm
https://storage.googleapis.com/kubernetes-release/release/v1.11.5/bin/linux/amd64/hyperkube

Do sha256sum on each of them, and then add v1.11.5 lines to defaults/main.yml with those checksums.

Then try it again

[mhabicht]

Which default/main? This one /root/kubespray/roles/upgrade/pre-upgrade/defaults/main.yml

[mhabicht]

I just attempted this same upgrade, but the upgrade fails with this message:

TASK [download : Download items] **********************************************************************************************************************************************************************************************************************************************************
Wednesday 05 December 2018  09:25:17 -0500 (0:00:00.040)       0:05:04.666 **** 
fatal: [kubeadmin]: FAILED! => {"msg": "'dict object' has no attribute u'1.11.5'"}

Did you get the same result when you ran the upgrade-cluster.yml playbook?

Yes this is what I got.

[micah]

Which default/main? This one /root/kubespray/roles/upgrade/pre-upgrade/defaults/main.yml

roles/download/defaults/main.yml

[mhabicht]

Which default/main? This one /root/kubespray/roles/upgrade/pre-upgrade/defaults/main.yml
roles/download/defaults/main.yml

Made that change and downloaded the 2 files, is there a particular directory they need to be in or just /kubespray?

[micah]

No, you just need to download them to get their checksums, once you have their checksums in that file, you can remove them.

[micah]

Look for the Checksums section, you need to add a line for your version
for hypercube and kubeadm

[micah]

Did it work?

[mhabicht]

Sorry, my master1 crashed and I have been trying to recover. The playbook failed without master1.

[mhabicht]

Failed on the reload.
FAILED - RETRYING: Configure | Check if etcd cluster is healthy

I have not been able to figure out how to fix this. Might have to rebuild the whole system.

[micah]

mhabicht [e-mail] writes:

Failed on the reload.
FAILED - RETRYING: Configure | Check if etcd cluster is healthy

this seems like a different issue than #3819

[mhabicht]

Ok, my cluster is up, running v1.11.3.

NAME STATUS ROLES AGE VERSION
avinetarch-anp1 Ready node 3m v1.11.3
avinetarch-anp2 Ready node 3m v1.11.3
avinetarchkm-anp1 Ready master 4m v1.11.3
avinetarchkm-anp2 Ready master 4m v1.11.3
avinetarchkm-anp3 Ready master 4m v1.11.3

I followed the process above and changed the "Kube version to 1.11.5" in the kubespray/roles/download/defaults/main.yml.

NAME STATUS ROLES AGE VERSION
avinetarch-anp1 Ready node 20m v1.11.5
avinetarch-anp2 Ready node 20m v1.11.5
avinetarchkm-anp1 Ready master 21m v1.11.5
avinetarchkm-anp2 Ready master 21m v1.11.5
avinetarchkm-anp3 Ready master 21m v1.11.5