Kubeadm: Control plane certs not working with external etcd

[Joseph94m]

What keywords did you search in kubeadm issues before filing this one?

upload-certs
kubeadm join
control plane
master join
certificate

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):1.15

Environment:

• Kubernetes version (use kubectl version): 1.15.5
• Cloud provider or hardware configuration: physical machines
• OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)"
• Kernel (e.g. uname -a):Linux ns3150165 3.10.0-1062.4.1.el7.x86_64 # 1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

A good start:

After the first master node inits the control plane using sudo kubeadm init --config kubeadm-config-master.yaml --upload-certs, this generate a command that can be used to join nodes to the cluster:

kubeadm join my-url-com:5443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <cert-key>

This command works fine to join a master to the control plane, and removing the --control-plane --certificate-key also works fine to join a worker. So far, so good.

The problem:

After the certificate key expires (1 hour), we need to upload new certs to our external etcd if we want to join other nodes as masters. This is done using the following command: sudo kubeadm init phase upload-certs --upload-certs. This command prints a token that has the same format as the one following the --control-plane --certificate-key in the previous example.

Running the
kubeadm join my-url-com:5443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <new-token>
returns the following error:
error execution phase control-plane-prepare/download-certs: error downloading certs: the Secret does not include the required certificate or key - name: external-etcd-ca.crt, path: /etc/kubernetes/pki/etcd/ca.crt

The problem does not seem to be that the token is invalid or incorrect itself because I have tried replacing the generated token in the command with a gibberish token like uidgqyuiwegyuqgeyuqwe and the error returned was different and it stated that the key I was attempting to use was not valid.

What you expected to happen?

I would expect to be able to join a master to the control plane using certificate keys generated printed by the sudo kubeadm init phase upload-certs --upload-certs like in the following:

kubeadm join my-url-com:5443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <cert-key>

How to reproduce it (as minimally and precisely as possible)?

Pre-requisites:

Have external etcds up and running (3 of them).
Run sudo kubeadm init --config kubeadm-config-master.yaml --upload-certs with the correct configuration. Make sure that the control plane is initialized.
Try joining a master (and if you want a worker) to the cluster using the printed command. This should work.

Expected failure:

Upload a new certificate using sudo kubeadm init phase upload-certs --upload-certs and use the printed certificate key to join a master to the control plane. This should fail with the error:
error execution phase control-plane-prepare/download-certs: error downloading certs: the Secret does not include the required certificate or key - name: external-etcd-ca.crt, path: /etc/kubernetes/pki/etcd/ca.crt

Anything else we need to know?

Our (@cyril-schmitt and I) temporary solution is to manually copy certificates from the first master to subsequent masters in order to allow them to join the control plane, this includes the following files:
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

and then running kubeadm join my-url-com:5443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --node-name <node-name> instead of specifying the --certificate-key option.

On our side we believe that the problem is in relation with the fact that we are using external etcds.

[neolit123]

This is done using the following command: sudo kubeadm init phase upload-certs --upload-certs

i'm pretty sure you need to pass the --config to the phase command too.
please try that.

[neolit123]

an issue like that was logged in kubernetes/kubernetes the other day.

[Joseph94m]

Hello, thank you for your advice.
In fact using sudo kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml did fix our issue. Can't believe it was that easy.
In our defense, the official doc does not explicit it: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#uploading-control-plane-certificates-to-the-cluster

Is this the appropriate issue to ask for the doc to be updated? Or should i post on kubernetes/kubernetes or create a new issue on kubernetes/kubeadm?

[neolit123]

Is this the appropriate issue to ask for the doc to be updated?

yes, definitely.

Or should i post on kubernetes/kubernetes or create a new issue on kubernetes/kubeadm?

please log an issue in kubernetes/website and ping me on it., also if you send the PR yourself it would be much appreciated!

[hey-zhangxu]

Hello, thank you for your advice.
In fact using sudo kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml did fix our issue. Can't believe it was that easy.
In our defense, the official doc does not explicit it: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#uploading-control-plane-certificates-to-the-cluster

Is this the appropriate issue to ask for the doc to be updated? Or should i post on kubernetes/kubernetes or create a new issue on kubernetes/kubeadm?

using kubeadm init phase upload-certs --upload-certs --config kubeadm-config.yaml. I solved the problem！！thanks ！！