Kubespray: /etc/kuberentes/ssl certs do not update expire date with upgrade

[mhabicht]

Master and worker upgrade to the new patch, but the certificates still have the old expiration date. I have tested going from 14.1 to 14.2 and 14.2 to 14.3 same problem, the certs still have the old expiration date.

Environment:
Dell R440 bare metal

Linux 4.4.0-171-generic x86_64
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

ansible 2.7.10
config file = /root/kubespray-2.10.0/ansible.cfg
configured module search path = [u'/root/kubespray-2.10.0/library']
ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.12 (default, Oct 8 2019, 14:14:10) [GCC 5.4.0 20160609]

**Kubespray version - 2.10.0

Network plugin used: Calico

Command used to invoke ansible:
ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.yaml -e kube_version=v1.14.3 -u root --ask-pass -b --become-user=root

*14.2 before upgrade
-kubectl get nodes
NAME STATUS ROLES AGE VERSION
kats-1 Ready 3d1h v1.14.2
kats-2 Ready 3d1h v1.14.2
kats-3 Ready 3d1h v1.14.2
kats-4 Ready 3d1h v1.14.2
kats-5 Ready 3d1h v1.14.2
kats-km1 Ready master 3d1h v1.14.2
kats-km2 Ready master 3d1h v1.14.2
kats-km3 Ready master 3d1h v1.14.2

-cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text | less
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2175565984948876246 (0x1e3129bc964cf3d6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Jan 14 14:17:20 2020 GMT
Not After : Jan 13 14:17:20 2021 GMT <<<<----------
Subject: CN=kube-apiserver

*14.3 after upgrade
-kubectl get nodes
NAME STATUS ROLES AGE VERSION
kats-1 Ready 3d2h v1.14.3
kats-2 Ready 3d2h v1.14.3
kats-3 Ready 3d2h v1.14.3
kats-4 Ready 3d2h v1.14.3
kats-5 Ready 3d2h v1.14.3
kats-km1 Ready master 3d2h v1.14.3
kats-km2 Ready master 3d2h v1.14.3
kats-km3 Ready master 3d2h v1.14.3

-cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text | less
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2175565984948876246 (0x1e3129bc964cf3d6)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Jan 14 14:17:20 2020 GMT
Not After : Jan 13 14:17:20 2021 GMT <<<<----------
Subject: CN=kube-apiserver

[gjemp]

Hei,
identified similar issue with version 2.12.0 deploy(upgrade from 2.11) on RH 7.6 .
in my case it is really odd. 3 crt files are now outdated: apiserver.crt, apiserver-kubelet-client.crt, front-proxy-client.crt

before upgrade:
Issuer: CN=kubernetes Validity Not Before: Dec 18 08:23:44 2018 GMT Not After : Aug 22 12:07:49 2020 GMT Subject: CN=kube-apiserver

after upgrade :
_CN=kubernetes Validity Not Before_ Dec 18 08_23_44 2018 GMT Not After _ Dec 15 21_21_56 2020 GMT Subject_ CN=kube-apiserver

[LuckySB]

this is kubeadm bug, https://github.com/kubernetes/kubeadm/issues/1818

we need add --certificate-renewal=true to kubeadm-upgrade.yml

[mhabicht]

I am running 1.14, this bug appears to be in 1.15, am I correct?

I attempted to make the change anyways and received the following error.

```
'--->>>> "stderr": "Error: unknown flag: --certificate-renewal\nUsage:\n'

*Here is my Config File, I added the line in 2 places.

** vi /kubespray-2.10.0/roles/kubernetes/master/tasks/kubeadm-upgrade.yml

---

- name: kubeadm | Upgrade first master
command: >-
timeout -k 600s 600s
{{ bin_dir }}/kubeadm
upgrade apply -y {{ kube_version }}
--config={{ kube_config_dir }}/kubeadm-config.yaml
--ignore-preflight-errors=all
--allow-experimental-upgrades
--allow-release-candidate-upgrades
--etcd-upgrade=false
--force
--certificate-renewal=true <<<<------------
register: kubeadm_upgrade
* Retry is because upload config sometimes fails
retries: 3
when: inventory_hostname == groups['kube-master']|first
failed_when: kubeadm_upgrade.rc != 0 and "field is immutable" not in kubeadm_upgrade.stderr
notify: Master | restart kubelet
*- name: kubeadm | Upgrade other masters
command: >-
timeout -k 600s 600s
{{ bin_dir }}/kubeadm
upgrade apply -y {{ kube_version }}
--config={{ kube_config_dir }}/kubeadm-config.yaml
--ignore-preflight-errors=all
--allow-experimental-upgrades
--allow-release-candidate-upgrades
--etcd-upgrade=false
--certificate-renewal=true <<<<------------
register: kubeadm_upgrade
when: inventory_hostname != groups['kube-master']|first
failed_when:
- kubeadm_upgrade.rc != 0
- '"field is immutable" not in kubeadm_upgrade.stderr'
- kubeadm_upgrade.stdout_lines | length > 1
notify: Master | restart kubelet

[svistoi]

Not sure if I have the same issue/related or completely different certificate issue.

I ran into expiring certificate issue as well. After upgrading 1.14.3 (2.10.3 kubespray) to 1.15.3 (2.11.0 kubespray) the kubelet certificate on the master nodes was not renewed. That kubelet certificate on master is expiring in 4 days in my case.

I noticed that the master nodes kubelet.conf is configured to use client-certificate-data base64 cert, whereas regular nodes are configured to use client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem.

On the master node:

# awk -F': ' '/client-certificate-data/{print $2}' /etc/kubernetes/kubelet.conf | base64 -d | openssl x509 -enddate -noout -in -
notAfter=Feb  8 19:38:46 2020 GMT

And on the master node checking an existing (but looks like unused) file /var/lib/kubelet/pki/kubelet-client-current.pem

# openssl x509 -enddate -noout -in /var/lib/kubelet/pki/kubelet-client-current.pem 
notAfter=Jun  5 21:49:00 2020 GMT

Any comments/advice on:

• is the difference in kubelet configuration between master and regular nodes expected?
• should I insert base64 encoded contents into kubelet.conf on the master or I could change the line to reflect client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem found on regular nodes? But I want to remain kubespray compatible for future upgrades.
• should I try kubeadm alpha certs renew --config=/etc/kubernetes/kubeadm-config.yaml or some variation thereof to update the kubelet certificate?

Edit
Looks like modifying kubelet.conf is the suggested route to have this start working
https://github.com/kubernetes/kubeadm/issues/1361#issuecomment-549009785

I'm still unclear about differences in expiry dates, for example

# kubeadm alpha certs check-expiration --config=/etc/kubernetes/kubeadm-config.yaml
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Feb 01, 2021 03:50 UTC   363d            no      
apiserver                  Feb 01, 2021 03:49 UTC   363d            no      
apiserver-kubelet-client   Feb 01, 2021 03:49 UTC   363d            no      
controller-manager.conf    Feb 01, 2021 03:50 UTC   363d            no      
front-proxy-client         Feb 01, 2021 03:49 UTC   363d            no      
scheduler.conf             Feb 01, 2021 03:50 UTC   363d            no      

However as posted in my case /var/lib/kubelet/pki/kubelet-client-current.pem is expiring Jun 5 21:49:00 2020 GMT, so I'm not sure where kubeadm is getting its certificates from for apiserver-kubelet-client to check.

[LuckySB]

--certificate-renewal flag appeared in version 1.16

you may get kubeadm v1.16 and rotate certificate via command
kubeadm alpha certs renew all
and also rotate certificate for master-1 kubelet
kubeadm alpha kubeconfig user --client-name system:node:node-name.domain.com --org system:nodes > /etc/kubernetes/kubelet.conf