Kubespray: cluster upgrade FAILED

[kisahm]

i tried to upgrade my kubernetes v1.9.2 cluster (deployed with kubespray) to 1.12.3 but the upgrade fails because some certs are missing:

TASK [kubernetes/master : Copy old certs to the kubeadm expected path] ********************************************************************************************************************************************
Friday 07 December 2018  10:01:35 +0000 (0:00:00.470)       1:49:21.355 ******* 
changed: [k8s-0001] => (item={u'dest': u'apiserver.crt', u'src': u'apiserver.pem'})
changed: [k8s-0001] => (item={u'dest': u'apiserver.key', u'src': u'apiserver-key.pem'})
changed: [k8s-0001] => (item={u'dest': u'ca.crt', u'src': u'ca.pem'})
changed: [k8s-0001] => (item={u'dest': u'ca.key', u'src': u'ca-key.pem'})
failed: [k8s-0001] (item={u'dest': u'front-proxy-ca.crt', u'src': u'front-proxy-ca.pem'}) => {"changed": false, "item": {"dest": "front-proxy-ca.crt", "src": "front-proxy-ca.pem"}, "msg": "Source /etc/kubernetes/ssl/front-proxy-ca.pem not found"}
failed: [k8s-0001] (item={u'dest': u'front-proxy-ca.key', u'src': u'front-proxy-ca-key.pem'}) => {"changed": false, "item": {"dest": "front-proxy-ca.key", "src": "front-proxy-ca-key.pem"}, "msg": "Source /etc/kubernetes/ssl/front-proxy-ca-key.pem not found"}
changed: [k8s-0001] => (item={u'dest': u'front-proxy-client.crt', u'src': u'front-proxy-client.pem'})
changed: [k8s-0001] => (item={u'dest': u'front-proxy-client.key', u'src': u'front-proxy-client-key.pem'})
failed: [k8s-0001] (item={u'dest': u'sa.pub', u'src': u'service-account-key.pem'}) => {"changed": false, "item": {"dest": "sa.pub", "src": "service-account-key.pem"}, "msg": "Source /etc/kubernetes/ssl/service-account-key.pem not found"}
failed: [k8s-0001] (item={u'dest': u'sa.key', u'src': u'service-account-key.pem'}) => {"changed": false, "item": {"dest": "sa.key", "src": "service-account-key.pem"}, "msg": "Source /etc/kubernetes/ssl/service-account-key.pem not found"}
changed: [k8s-0001] => (item={u'dest': u'apiserver-kubelet-client.crt', u'src': u'node-k8s-0001.pem'})
changed: [k8s-0001] => (item={u'dest': u'apiserver-kubelet-client.key', u'src': u'node-k8s-0001-key.pem'})

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Environment:

• openstack

Linux 3.10.0-693.21.1.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Version of Ansible (ansible --version):
  ansible 2.7.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]

Kubespray version (commit) (git rev-parse --short HEAD): 225f765

Network plugin used: weave

Copy of your inventory file:

[all]
k8s-0001     ansible_host=192.168.0.113 ip=192.168.0.113
k8s-0002     ansible_host=192.168.0.110 ip=192.168.0.110
k8s-0003     ansible_host=192.168.0.103 ip=192.168.0.103
k8s-0005         ansible_host=192.168.0.230 ip=192.168.0.230

[kube-master]
k8s-0001     
k8s-0002     
k8s-0003

[kube-node]
k8s-0001     
k8s-0002     
k8s-0003     
k8s-0005

[etcd]
k8s-0001     
k8s-0002     
k8s-0003     

[k8s-cluster:children]
kube-node    
kube-master      

[calico-rr]

[vault]
k8s-0001     
k8s-0002     
k8s-0003     

Command used to invoke ansible: ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.ini -e kube_version=v1.12.3

[sdodsley]

Similar issue upgrading from 1.10.4. The original cluster was built using kubespray from the incubator github. The upgrade is being performed with the latest build from the sigs github.

TASK [kubernetes/master : Copy old certs to the kubeadm expected path] ********************************************************************************************************************************************
Monday 17 December 2018  06:18:38 -0800 (0:00:00.808)       0:47:54.694 *******
changed: [sn1-c07-caas-01] => (item={u'dest': u'apiserver.crt', u'src': u'apiserver.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'apiserver.key', u'src': u'apiserver-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'ca.crt', u'src': u'ca.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'ca.key', u'src': u'ca-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-ca.crt', u'src': u'front-proxy-ca.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-ca.key', u'src': u'front-proxy-ca-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-client.crt', u'src': u'front-proxy-client.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-client.key', u'src': u'front-proxy-client-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'sa.pub', u'src': u'service-account-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'sa.key', u'src': u'service-account-key.pem'})
failed: [sn1-c07-caas-01] (item={u'dest': u'apiserver-kubelet-client.crt', u'src': u'node-sn1-c07-caas-01.pem'}) => {"changed": false, "item": {"dest": "apiserver-kubelet-client.crt", "src": "node-sn1-c07-caas-01.pem"}, "msg": "Source /etc/kubernetes/ssl/node-sn1-c07-caas-01.pem not found"}
failed: [sn1-c07-caas-01] (item={u'dest': u'apiserver-kubelet-client.key', u'src': u'node-sn1-c07-caas-01-key.pem'}) => {"changed": false, "item": {"dest": "apiserver-kubelet-client.key", "src": "node-sn1-c07-caas-01-key.pem"}, "msg": "Source /etc/kubernetes/ssl/node-sn1-c07-caas-01-key.pem not found"}

[kisahm]

does somebody know about a workaroud or fix for this issue?

[Atoms]

you should use released versions of kubespray to upgrade (git checkout v2.x.x) and not to skip versions, so if your cluster was installed let's say with 2.5.0 kubespray release, to get latest release you need to execute upgrade-cluster.yml from 2.6.0 then from 2.7.0, and then from 2.8.0 or 2.8.1, we do not support skipping releases

[caruccio]

There are any way to detect the current installer version of the cluster so we can alert users not to upgrade skipping versions? Maybe a configmap or file at the first master?

[giafar]

Hi all,
it it possibile to upgrade from a tag to a release and then upgrare just release ?
I mean:
tag v2.10.4 --> release 2.11 --> release 2.12 --> release 2.13 --> release 2.14 --> tag v2.14.1