kubeadm init api-server ignores etcd-cafile etcd-certfile etcd-keyfile during manifest write.

[Nitecon]

BUG REPORT

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): ubuntu
• Kernel (e.g. uname -a): Linux 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

Kubeadm init does not provide CA / key / cert for etcd servers when an external server is used, which then causes the apiserver not to start up as it cannot connect to the external etcd servers.

What you expected to happen?

During kubeadm init the manifests should be written with the etcd ca / cert / key information as switches.

How to reproduce it (as minimally and precisely as possible)?

Create kubeadm-config.conf file with following data:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.0
certificatesDir: /etc/kubernetes/pki
api:
  controlPlaneEndpoint: "load-balancer-endpoint:443"
networking:
  podSubnet: 192.168.0.0/16
etcd:
  external:
    endpoints:
    - https://external-etcd-1.server.com:2379
    - https://external-etcd-2.server.com:2379
    - https://external-etcd-3.server.com:2379
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 1s1bvu.u1wll2dl2eo701ul

now run:
kubeadm init --ignore-preflight-errors=all --config kubeadm-config.conf

Content of /etc/kubernetes/manifests/kube-apiserver.yaml is now:

.... redacted ....
spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=REDACTED
    - --allow-privileged=true
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --disable-admission-plugins=PersistentVolumeLabel
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-servers=https://external-etcd-1.server.com:2379,https://external-etcd-1.server.com:2379,https://external-etcd-1.server.com:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    env:

Expecting:

    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/etcd/server.crt
    - --etcd-keyfile=/etc/kubernetes/pki/etcd/server.key

To be part of the flags in command setup.

Anything else we need to know?

Notice there is no config variables for setting apiserver etcd ca servers so the assumption is that it will use the certs that are pre-set under /etc/kubernetes/pki. Kubelet and other services do however use the existing certs correctly.

[neolit123]

@Nitecon
thanks for the issue report.

kind: MasterConfiguration
kubernetesVersion: v1.11.0
apiVersion: kubeadm.k8s.io/v1alpha2

^ as a side node, make sure you have apiVersion: kubeadm.k8s.io/v1alpha2 in your config.
i will verify what is the state for this in 1.11 and 1.12 and reply a bit later.

[Nitecon]

Sounds good I pulled that off the latest docs from kubernetes HA, beyond the apiserver issue everything else seems to function fine.

I double checked and apiVersion: kubeadm.k8s.io/v1alpha2 is set it just was cut off in the scripts above, i must have had that right next to the 3 tildes.

[neolit123]

@Nitecon

the code between 1.11 and current master hasn't changed for this:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L173-L179
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L173-L179

the way for the api-server flags to be applied is if the user explicitly defines the certs under: etcd: external: ..., please try to see if this fixes the issue for you.

should we consider this as a documentation flaw?

/assign @chuckha @timothysc
/cc @stealthybox

[neolit123]

I double checked and apiVersion: kubeadm.k8s.io/v1alpha2 is set it just was cut off in the scripts above, i must have had that right next to the 3 tildes.

ack, thanks. i just verified the same myself.

[Nitecon]

I'll double check tomorrow regarding the external settings for etcd certs and if that works we can just set this as a bug in documentation as I would expect consistent behavior throughout all the components, especially with something like kube-apiserver which is generally run inside a container and doesn't provide any logs when a failure like this usually occurs.

[neolit123]

/priority awaiting-more-evidence

[Nitecon]

Updated Kubeadm config file looks like this now:

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.0
certificatesDir: /etc/kubernetes/pki
api:
  controlPlaneEndpoint: "loadbalancer.server.endpoint.com:443"
networking:
  podSubnet: 192.168.0.0/16
etcd:
  external:
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key
    endpoints:
    - https://external-etcd-1.server.com:2379
    - https://external-etcd-2.server.com:2379
    - https://external-etcd-3.server.com:2379
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: cxvl1s.798jt3w72a535am5

With it set in that way the init completed successfully and apiserver config is set up correctly. We do need to make sure this information is in documentation and perhaps also fix the execution so it's consistent with all the other components.

[neolit123]

thanks for testing @Nitecon

@chuckha is currently refactoring a related document.
Chuck, do you know what would be the best place to include this addition for external etcd:

  external:
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key

[chuckha]

@neolit123 I don't quite understand the question, can you rephrase?

This information is already part of the HA-external etcd setup instructions: https://kubernetes.io/docs/setup/independent/high-availability/#set-up-the-first-control-plane-node. But if there is something lacking I'd be happy to add it.

I'm starting to think we need different pages for external vs stacked...

[neolit123]

@chuckha

do you know what would be the best place to include this addition for external etcd:

I don't quite understand the question, can you rephrase?

we seem to be lacking explanation that these additions:

 external:
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key

are required in a case where an external etcd server i used.

as @Nitecon wrote:

Kubeadm init does not provide CA / key / cert for etcd servers when an external server is used, which then causes the apiserver not to start up as it cannot connect to the external etcd servers.

so we either want to document this or investigate if this is a case where we should be populating the default cert values under external:.

[chuckha]

ohh, I see what you're saying. Thanks for the clarification @neolit123, much appreciated.

Yes, we need a validation check. If we're using an https connection to etcd in the external case, we need those files. If we are using an insecure connection then we don't need them. I think this is a bug.

Let's pick this default in the case no value is supplied when using a secure connection.

    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key

[neolit123]

@chuckha
ok, i'm looking into this.

/lifecycle active

[chuckha]

taking this one over

[chuckha]

/lifecycle active

[timothysc]

Closing on last merge.