Kubeadm: support storing oidc params for kubeapi in kubeadm-config cm

[davidkarlsen]

What keywords did you search in kubeadm issues before filing this one?

kubeadm oidc

FEATURE REQUEST

If this is a FEATURE REQUEST, please:
I have manually added oidc-params (oidc-client-id, oidc-groups-claim, oidc-issuer-url) to
/etc/kubernetes/manifests/kube-apiserver.yaml - when doing kubeadm upgrade apply these were removed. I want them to survive an upgrade, for instance by having them in the kubeadm-config configmap.

Versions

kubeadm version (use kubeadm version):

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
root@main:/etc/kubernetes/manifests# 

• Cloud provider or hardware configuration: x86 onprem
• OS (e.g. from /etc/os-release):

cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):

uname -a
Linux main.lan.davidkarlsen.com 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@main:/etc/kubernetes/manifests# 

• Others:

What happened?

existing parameters got lost after upgrade

What you expected to happen?

make the oidc parameters be left untouched after during upgrade

How to reproduce it (as minimally and precisely as possible)?

see above

Anything else we need to know?

[neolit123]

@davidkarlsen

I have manually added oidc-params (oidc-client-id, oidc-groups-claim, oidc-issuer-url) to
/etc/kubernetes/manifests/kube-apiserver.yaml - when doing kubeadm upgrade apply these were removed. I want them to survive an upgrade, for instance by having them in the kubeadm-config configmap.

what happens if you store the oidc* flags under the APIServerExtraArgs field in the kubeadm MasterConfiguration:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#MasterConfiguration

like so:

...
kind: MasterConfiguration
kubernetesVersion: ...
apiServerExtraArgs:
  oidc-client-id: ...
  ...

and re-attempt the upgrade.
(you can also use --dry-run BTW).

also what is the exact kubeadm upgrade... command line?

thanks.

[davidkarlsen]

I did kubeadm upgrade apply v1.10.4.
Already upgraded - so can't test until next release.
I guess this should go into kubectl -n kube-system edit cm/kubeadm-config under

apiVersion: v1
data:
  MasterConfiguration: |
 api:
      advertiseAddress: 192.168.3.1
      bindPort: 6443
      controlPlaneEndpoint: ""
 apiServerExtraArgs:
   oidc-client-id: ...

?

[neolit123]

Already upgraded - so can't test until next release.

you can still add these to your kubeadm file.
the one that is used via kubeadm init --config.

but you have to restart the cluster.

I guess this should go into kubectl -n kube-system edit cm/kubeadm-config under

yes, but i'm pretty sure it should be cm kubeadm-config instead of cmd/kubeadm-config.
mind that i haven't tried editing like that on a running cluster.

[davidkarlsen]

OK - added - let's see when 1.10.5 comes out!

[neolit123]

@davidkarlsen hi, 1.10.5 was released recently.
any update on the issue you were having with 1.10.x, or perhaps you moved to 1.11.x instead?

[davidkarlsen]

Hi - yes it worked well - sorry for not responding sooner. Should this issue be made a documenting issue or is it documented anywhere else than as code and should be closed?

[neolit123]

@davidkarlsen
thanks for the confirmation.

we've recently added this document:
https://kubernetes.io/docs/setup/independent/control-plane-flags/

the flags shouldn't be lost between upgrades if present under apiServerExtraArgs and the users don't have to edit the manifest files or use kubectl cm....

/close