Kubeadm: Cannot specify kubeconfig-dir in kubeadm config

[asauber]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: FEATURE REQUEST

When using the kubeadm init phase kubeconfig command, we can specify --kubeconfig-dir to choose the output directory for the generated kubeconfigs, as in the following example.

root@localhost:/tmp/cluster04/pki/kubeconfigs# kubeadm init phase kubeconfig all --kubeconfig-dir $(pwd)/kubeconfigs --cert-dir /tmp/cluster04/pki
[kubeconfig] Using kubeconfig folder "/tmp/cluster04/pki/kubeconfigs/kubeconfigs"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

If we already have a kubeadm config file, we might expect to be able to add the key kubeconfigDir to the ClusterConfiguration in order to simply pass --config to kubeadm init phase kubeconfig, such as in the following example kubeadm config file.

kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta1
...
certificatesDir: /tmp/cluster04/pki
kubeconfigDir: /tmp/cluster04/pki/kubeconfigs
clusterName: cluster04
dns:
  type: CoreDNS
imageRepository: k8s.gcr.io
kubernetesVersion: v1.13.3
scheduler: {}

However, this does not work, as kubeconfigDir is not a field on ClusterConfiguration

root@localhost:/tmp/cluster04/pki# kubeadm init phase kubeconfig all --config kubeadm.conf
W0211 22:17:53.619222   22560 strict.go:54] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta1", Kind:"ClusterConfiguration"}: error unmarshaling JSON: while decoding JSON: json: unknown field "kubeconfigDir"
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

This results in the kubeconfigs being written to the default directory /etc/kubernetes which may be unwanted behavior.

As with all other kubeadm commands, we cannot mix --config with command line parameters.

root@localhost:/tmp/cluster04/pki# kubeadm init phase kubeconfig all --config kubeadm.conf --kubeconfig-dir $(pwd)/kubeconfigs
can not mix '--config' with arguments [kubeconfig-dir]

Request:
Support a kubeconfigDir field in the kubeadm config API such that kubeadm init phase kubeconfig can be used with --config as expected.

[neolit123]

we might not want to add the field in the config but this seems like a bug to me:

kubeadm init phase kubeconfig all --config kubeadm.conf --kubeconfig-dir $(pwd)/kubeconfigs
can not mix '--config' with arguments [kubeconfig-dir]

given kubeconfig-dir is not something from the kubeadm API we should allow mixture of the flags --config and --kubeconfig-dir.

@fabriziopandini WDYT?

[asauber]

@neolit123 Why not make it a part of the kubeadm API?

[neolit123]

@asauber since the configuration API is beta, every change (even if small) has to be proposed in a kubeadm office hours meeting, discussed and voted there.

not allowing this flag mix seems more like a bug to me.

[fabriziopandini]

@asauber is there a specific reason for changing the Kubeconfig folder?

@neolit123 AFAIK as I now kubeadm standardized a set of well known file names and paths, so my take-away here is that --kubeconfig-dir should be considered only as an additional facilities that applies to the kubeconfig phase, and not as an option to be supported by the kubeadm config (that implies support by all the kubeadm workflows + test coverage)

In case the current UX considered confusing, my personal opinion is that we should deprecate the flag/make the help message more clear, but I'm open to discuss this if there is use care that potentially affect many users to be addressed

[neolit123]

@fabriziopandini
i agree that the -dir flag should be isolated to the phase only.

@yagonobre sent this PR:
https://github.com/kubernetes/kubernetes/pull/73998

please review if you think we should also limit the scope of -dir in the same PR?

[asauber]

@fabriziopandini The reason that I need to change the kubeconfig folder is that I use the kubeadm phases facility many times in succession to generate certs and kubeconfigs for a number of clusters. I need to output the kubeconfigs for each cluster in a separate directory.

@yagonobre Thanks for the PR

@neolit123 Thanks for reviewing

[fabriziopandini]

@asauber Thanks for the explanation
If I got this right, the PR just merged is enough for making kubeadm fit for your use case and changes to the kubeadm config file are not necessary. Great!

[asauber]

@fabianofranz It helps somewhat, because now I don't have to pass the flags that are in my config to the phase, I just have to pass two flags. But why not incorporate all flags for each phase into the config? I have a kubeadm config file in one location in my codebase, and flags that will get passed to each phase in another location. It would be much nicer to only have one source of truth for the parameters.

[neolit123]

@asauber

next to this comment:

since the configuration API is beta, every change (even if small) has to be proposed in a kubeadm office hours meeting, discussed and voted there.

i would add that this flag is specific to a sub-phase and adding it in the wider init-configuration is less-likely.

but again discussion about this field can certainly happen.
if you wish you can join the kubeadm office hours meeting next week and bring this as an agenda item.