kubeadm join can not mix '--config' with arguments [experimental-control-plane]

[marccarre]

What keywords did you search in kubeadm issues before filing this one?

• can not mix '--config' with arguments [experimental-control-plane]
• experimental
• certificate

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT or maybe neither and more a documentation issue?

Versions

kubeadm version:

# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:51:21Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

# uname -a
Linux node0 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

What happened?

When trying to setup a HA cluster with stacked control plane and etcd nodes, one needs to provide the following options to kubeadm join for master nodes:

• --experimental-control-plane
• --certificate-key $CERT_KEY

However if one is already passing configuration via --config, the following error is returned:

can not mix '--config' with arguments [experimental-control-plane]

According to #1251, and if I understood correctly, the below can be added to the JoinConfiguration object instead of passing --experimental-control-plane:

apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
controlPlane:
  localAPIEndpoint:
    advertiseAddress: <some-ip>
    bindPort: <some-port>

However, it is unclear where $CERT_KEY should be added.

• Can it be passed to kubeadm via JoinConfiguration too? If so, where?
• Can it be passed to another object? I couldn't find anything clearly referring to it in v1beta1's types. Maybe ExternalEtcd.KeyFile?

N.B.:

1. without --certificate-key $CERT_KEY, joining fails with:
   error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s
2. passing --certificate-key $CERT_KEY anyway doesn't fail with can not mix but the value seems to be ignored as it eventually also times out with:
   error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s

What you expected to happen?

The machine to join the cluster as a master node in order to form a HA cluster _with stacked control plane and etcd nodes_.

How to reproduce it (as minimally and precisely as possible)?

# kubeadm join <ip>:<port> \
    --config /path/to/config.yaml \
    --experimental-control-plane \
    --certificate-key $CERT_KEY

[neolit123]

@marccarre
--experimental-control-plane is CLI exclusive and cannot be part of the config.
but we are going to have to support the mixture in 1.15.

in terms of:

  localAPIEndpoint:
    advertiseAddress: <some-ip>
    bindPort: <some-port>

and

However if one is already passing configuration via --config

what is your use case for requiring the --config flag? kubeadm HA works by using a "control plane endpoint" which is the address of the load-balancer in front of your control plane nodes.

this is the address : port you specify here:
kubeadm join <ip>:<port> ...

$CERT_KEY should be passed to --certificate-key and the JoinConfiguration does not support that yet in 1.14.

so if you don't pass a --config
this should work for you:

kubeadm join <ip>:<port> \
    --experimental-control-plane \
    --certificate-key $CERT_KEY
...

[marccarre]

what is your use case for requiring the --config flag?

@neolit123, I currently pass the following via --config:

apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: $SEED_MASTER_IP:$PORT
    caCertHashes:
    - sha256:$CERT_HASH
    token: $TOKEN
nodeRegistration:
  kubeletExtraArgs:
    node-ip: $IP

Is there a CLI equivalent to the below?

nodeRegistration:
  kubeletExtraArgs:
    node-ip: $IP

I couldn't find any in cmd/kubeadm/app/cmd/join.go, but I might have missed something.

[neolit123]

Is there a CLI equivalent to the below?

https://kubernetes.io/docs/setup/independent/kubelet-integration/#the-kubelet-drop-in-file-for-systemd

KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet

although under normal circumstances you don't need to specify the node-ip for the kubelet.

[marccarre]

Many thanks for the pointers 👍

node-ip

Unfortunately, I have abnormal circumstances 🙂

[marccarre]

FWIW,

• not changing anything on my seed master node [1], and
• having the following on the 2nd master node:
  console # cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false --node-ip=$PRIVATE_IP"
  to compensate for:
  yaml nodeRegistration: kubeletExtraArgs: node-ip: $PRIVATE_IP

joining with the CLI flags [2] ~fails with~ (EDIT: works, I had another issue on top of this one):

error execution phase preflight: 
  couldn't validate the identity of the API Server: 
    abort connecting to API servers after timeout of 5m0s

I didn't change anything since I used to be able to have worker nodes joining.

# kubeadm join --token $TOKEN \
    --discovery-token-ca-cert-hash sha256:$CERT_HASH \ 
    --experimental-control-plane \
    --certificate-key $CERT_KEY \
    $SEED_MASTER_PRIVATE_IP:6443

[marccarre]

we are going to have to support the mixture in 1.15.

Out of curiosity, what is the target release date for 1.15 / this improvement?

[neolit123]

error execution phase preflight:
couldn't validate the identity of the API Server:
abort connecting to API servers after timeout of 5m0s

• what do you see if you add --v=2 to the join command?
• make sure your token has not expired; it does so after 24h.
• do you have connectivity from this node to the controlplane apiserver on this address:port using curl?

Out of curiosity, what is the target release date for 1.15 / this improvement?

1.14 was released 2 weeks ago so roughly 3 months minus 2 weeks.

[fabriziopandini]

/assign

[luxas]

a) Have you used --expermental-upload-certs in the beginning? If no, you don't need --certificate-key IIUC
b) I think you need to pass --apiserver-advertise-address=$PRIVATE_IP. Please try that like this, and tell us if it worked:

$ cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false --node-ip=$PRIVATE_IP"

```console
$ kubeadm join --token $TOKEN \
--discovery-token-ca-cert-hash sha256:$CERT_HASH \
--experimental-control-plane \
--apiserver-advertise-address=$PRIVATE_IP \
(--certificate-key $CERT_KEY) \
$SEED_MASTER_PRIVATE_IP:6443

Anyways, I think there is an issue in that the current code in `join.go` (if I read it right), doesn't allow for a config-only flow like this:
```yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: $SEED_MASTER_IP:$PORT
    caCertHashes:
    - sha256:$CERT_HASH
    token: $TOKEN
nodeRegistration:
  kubeletExtraArgs:
    node-ip: $IP
controlPlane:
  localAPIEndpoint:
    advertiseAddress: $IP

_That_ is a bug. Enabling the control plane mode via the config file (even by just doing controlPlane: {}), should definitely work

[marccarre]

@neolit123, @luxas thanks for your help & suggestions!

@neolit123:

1. --v=2 (thanks for the tip!) allowed me to find another issue hidden under this one: my way of using kubeadm was creating two kube-system/bootstrap-token-$TOKEN_ID Secrets, and the way I was programmatically reading these secrets back could end up with the incorrect bootstrap token, and eventually lead to couldn't validate the identity of the API Server when joining nodes. The workaround initially suggested by @neolit123, and made explicit below for convenience, therefore works for me. I've edited my previous comment accordingly as well.

```console
# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false --node-ip=$PRIVATE_IP"
```
```console
# kubeadm join --token $TOKEN \
    --discovery-token-ca-cert-hash sha256:$CERT_HASH \ 
    --experimental-control-plane \
    --certificate-key $CERT_KEY \
    $SEED_MASTER_PRIVATE_IP:6443
```

1. I'm joining immediately after initialising, so I'm 100% sure it hasn't expired.

2. I did have connectivity to the seed master (but as now established in 1., this was a bootstrap token issue anyway, so unrelated to connectivity):

   # curl -k https://$SEED_MASTER_PRIVATE_IP:6443
   {
     "kind": "Status",
     "apiVersion": "v1",
     "metadata": {
     },
     "status": "Failure",
     "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
     "reason": "Forbidden",
     "details": {
     },
     "code": 403
   }
   

@luxas, for the record & to answer your questions:

1. Yes, I do pass --experimental-upload-certs to kubeadm init.
2. I didn't need --apiserver-advertise-address=$PRIVATE_IP in the end, as per the above, but I'll keep this in mind (and will update here) if I face more issues in my experiments.

[neolit123]

thanks for the feedback @marccarre
FYI in 1.15:

• we added the support to have certificateKey as part of the kubeadm JoinConfiguration
  https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
  https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#JoinControlPlane
• the flag --experimental-control-plane is deprecated in favor of --control-plane.
• the mixture of --control-plane and --config is still not allowed, which is by design.

these changes cannot be backported to <1.15 as they fall under kind/feature.

for additional issues please open new tickets.
thanks.