Kubeadm: Overlap between kubeconfig phase and alpha kubeconfig command

[johnSchnake]

There are two code paths which are trying to accomplish the same work but there is some asymmetry that needs to be resolved and I'm not certain which direction to go.

There is the kubeconfigPhase which is a reusable phase of code which can take a --config flag and, as a result, can set the controlplaneEndpoint.

There is the kubeadm alpha kubeconfig user command which is completely unrelated to the "phase" and does not take a --config flag but instead allows you to specify the client name for the kubeconfig and the organization (both of which the kubeconfigPhase does not). Since this command lets you set the apiserver-advertise-address (and not the control plane endpoint) it actually requires IP addresses and won't allow CNAMEs.

The result of this is confusion since they are trying to accomplish the same thing but do so with slight differences that make it hard to tell what the _real_ differences are or why both exist.
https://github.com/kubernetes/kubeadm/issues/1057 is an example of this.

IMO it seems like we should be able to consolidate the logic. At the very least we could add the --config flag to the kubeadm alpha kubeconfig user command so that it has more parity with the phase but it seems like the ideal thing would be to expand the phases capability to customize the user and organization. That seems like it would allow us to delete the alpha command all together.

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:33:30Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

• Kernel (e.g. uname -a):
  Linux ip-10-0-2-205 4.15.0-1031-aws #33-Ubuntu SMP Fri Dec 7 09:32:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

What happened?

Two different flows for creating kubeconfig files with different capabilities:
kubeadm alpha kubeconfig user

• Doesnt take a config and so you cant specify CNAMEs
• Allows you to specify user
• Allows you to specify organization

kubeadm init phase kubeconfig admin

• Takes a config flag
• Can't specify user
• Can't specify organization

What you expected to happen?

Have a single flow for creating kubeconfig files with all the needed capabilities.

Anything else we need to know?

Here are links to the relevant code sections:

https://github.com/kubernetes/kubernetes/blob/b5b627d522ed74549e2583b258a7df0e3ffd33ea/cmd/kubeadm/app/cmd/phases/kubeconfig.go#L77

https://github.com/kubernetes/kubernetes/blob/b5b627d522ed74549e2583b258a7df0e3ffd33ea/cmd/kubeadm/app/cmd/alpha/kubeconfig.go#L74

[MalloZup]

@neolit123 is ok for you if i take this one?

[fabriziopandini]

@MalloZup great! however, let's agree on a possible solution before start implementing

[fabriziopandini]

back on help/wanted as per slack discussion

[johnSchnake]

Per slack, it seems like there isn't concensus about expanding the logic around the phase code so that it can also customize user/organization, is that right? A short look at the code did make this seem a little awkward to do (but I don't have a lot of experience with the phase code yet).

• Seems like a bad idea to try and expand config options since these would be optional, one-off things that arent related to the cluster configuration itself.
• Seems like adding another sibling phase that could could be called out itself would be awkward because the phase is already set up with an all option which will run all the sibling phases. As a result if we put an optional user phase it would also try to be run during the main bootstrapping logic which isnt the intent.

I know this isnt a priority (and the workaround is simply to change the control plane endpoint via sed after kubeadm alpha kubeconfig user), but I'm trying to start the discussion about what is the most desirable end state so that it is easier for someone to jump on this if they want.

The shortest/fastest solution is to just add the config flag to the command. I'm not sure if everyone is happy with that solution though so I'll defer to those that have been stewards of this code longer. If that is an OK solution then I'll happily take this ticket and get it knocked out.

[timothysc]

todo - remove the alpha kubeconfig command.

[hpandeycodeit]

@timothysc if this issue is still open, I would like to work on it?

[fabriziopandini]

@hpandeycodeit great!
/lifecycle active

[hpandeycodeit]

@fabianofranz are we looking to comment out the related code or remove it?

[fabriziopandini]

@hpandeycodeit remove it.

[seh]

Per #1435, I don't see any discussion here about whether we were trying to _lose_ the ability to create kubeconfig files for arbitrary users and groups; the discussion was about how to unify the code that does so. When @timothysc wrote the following, did he mean that we should no longer be able to create such files?

todo - remove the alpha kubeconfig command.

[neolit123]

since the removal of the user command is breaking users we should consider actions during this code freeze - i.e. did we regress badly here.

we did agree to remove kubeadm alpha kubeconfig user but we do not provide an alternative.

cc @seh @fabriziopandini @rosti

EDIT: well the alternative is kubectl.

[seh]

If it's something that the collective _kubeadm_ maintainers don't want to facilitate any longer, I can understand such a decision. However, in this case, it looks more like the regression was unintentional. I'd just to settle whether we want to continue to offer this capability. If so, we have a new task to restore it.

[neolit123]

let's move the discussion in https://github.com/kubernetes/kubeadm/issues/1435

it looks more like the regression was unintentional

i forgot that we are removing it because it's alpha (slightly out of scope for kubeadm) and there is the kubectl alternative.

[timothysc]

reverting this change and punting to 1.15 due to feedback.

[astrieanna]

This is marked help wanted, but I'm not sure what the needed task is. One implementation has been merged then reverted; is that expected to be merged again with more warning, for 1.15? Or is the solution to this issue an alternative implementation that supports more use-cases?

[neolit123]

we need to promote the functionality to generate new users. currently it's under alpha, but is quite useful.
the OP outlines available options.

marked as kind/design - i.e. needs discussion.

[BenTheElder]

The users[].name key being identical in kubeconfigs for different clusters has been a bit of a thorn for kind, we'll probably just fix that ourselves but it would be much easier to work with multiple clusters if one of the following were true:

• this was configurable in some way
• it by default included the cluster name and/or was otherwise mangled to make it somewhat unique between clusters
• recommend that users use kubeadm alpha kubeconfig user to generate a customized kubeconfig and promote it out of alpha

The contexts are already generated in a unique way with kubernetes-admin@${CLUSTER_NAME} but this doesn't carry over to the users entries. Not having this is problematic for merging configs etc.

It's not too difficult to do as a post-processing step on the standard admin configs but that's less than ideal.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[candlerb]

The ability to generate additional certificates for non-admin users is extremely helpful, and I don't think can be done by init phase kubeconfig admin

For example, there is a pre-defined "view" clusterrole, so you can create read-only users by:

kubeadm alpha kubeconfig user --client-name [email protected] --org noc

plus a single clusterrolebinding between group "noc" and clusterrole "view". This is a simple step up from everyone sharing the admin credentials (eurgh), without ripping and replacing the entire internal CA.

There are instructions for using an external CA, which say it's possible to give the external CA's key to k8s to allow it to continue to roll its own certs. This opens up more questions: what about certificate serial number conflicts? Shouldn't you really have one intermediate CA for k8s automated use, and a separate intermediate CA for signing user certs? If so, how do you install the intermediate CA certs?

Using the internal CA for both peer-to-peer credentials (automated) and end-user credentials (on demand) makes things simple.

[neolit123]

The ability to generate additional certificates for non-admin users is extremely helpful, and I don't think can be done by init phase kubeconfig admin

there are two options:

• make the phase flexible to support the same
• graduate the alpha kubeconfig command to a top level kubeconfig command

For example, there is a pre-defined "view" clusterrole

i cannot find where these are documented at k8s.io. do you know where?

This opens up more questions: what about certificate serial number conflicts?

the CA is responsible to pick a serial number from its S/N range.
if it ends up picking the same S/N i would consider this a CA issue and not a k8s or a kubeadm issue.

Shouldn't you really have one intermediate CA for k8s automated use, and a separate intermediate CA for signing user certs?

what are the actual benefits of having separate intermediate CAs?

If so, how do you install the intermediate CA certs?

the installation process is just copying them to /etc/kubernetes, but separate CAs are not supported.

Using the internal CA for both peer-to-peer credentials (automated) and end-user credentials (on demand) makes things simple.

indeed.

[candlerb]

For example, there is a pre-defined "view" clusterrole

i cannot find where these are documented at k8s.io. do you know where?

The predefined user-facing roles including "view" are documented here.

It does say "view" is supposed to be used with a namespaced RoleBinding, but I find it works for ClusterRoleBinding too (i.e. across all namespaces).

This opens up more questions: what about certificate serial number conflicts?

the CA is responsible to pick a serial number from its S/N range.
if it ends up picking the same S/N i would consider this a CA issue and not a k8s or a kubeadm issue.

The issue is: you have an external CA which you use to issue user certs, and k8s has its own CA which issues internal certs. The documentation says you can create your external CA and then hand over the private key to k8s CA for automated use. But there is no coordination of serial numbers.

I argue that if you continue to issue certs directly from the original CA (e.g. for user authentication) then you have a "split-brain" CA: neither one knows which serial numbers the other has issued, but both are signing with the same key and the same CA identity.

what are the actual benefits of having separate intermediate CAs?

To avoid two different certs being issued with the same serial number and signed with the same CA key / CA identity.

[neolit123]

The predefined user-facing roles including "view" are documented here.

i knew there were some default roles, but i didn't know they are documented.
thanks.

The issue is: you have an external CA which you use to issue user certs, and k8s has its own CA which issues internal certs. The documentation says you can create your external CA and then hand over the private key to k8s CA for automated use. But there is no coordination of serial numbers.

can you please state from which paragraph exactly are you reading this from?

external CA mode for kubeadm is documented here:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#external-ca-mode

the idea of "external CA" is to not give k8s your ca.key (private key), while providing manually signed certs. so not sure where you are see "hand over the private key to k8s"?

I argue that if you continue to issue certs directly from the original CA (e.g. for user authentication) then you have a "split-brain" CA: neither one knows which serial numbers the other has issued, but both are signing with the same key and the same CA identity.

yet, we have not seen issue reports about conflicts, or at least i haven't.

@randomvariable do you happen to have comments about this?

[candlerb]

can you please state from which paragraph exactly are you reading this from?

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#custom-certificates where it says:

"If a given certificate and private key pair exists before running kubeadm init, kubeadm does not overwrite them. This means you can, for example, copy an existing CA into /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key, and kubeadm will use this CA for signing the rest of the certificates."

I interpret this as meaning that it's fine to create an external CA, and hand over the keys to k8s. The issue then is whether that external CA continues to issue certs, such as user certs, independently. That doesn't seem to be discussed. (If it doesn't, then all you've done is prime the k8s CA with a key of your choice rather than letting it generate its own, which seems pretty pointless)

To be clear: the issue about duplicate serial numbers is not something I found in the doc, it's just something I have inferred as a consequence of issuing certs both from k8s and externally. I believe a separate intermediate CA for k8s would be the "right" way of dealing with this.

FWIW, I don't see any mention of certificate revocation, CRLs or OCSP in the documentation. Therefore, the issue of certificate serial numbers may be moot.

Without being able to use a CRL, recovery from a compromised user key is going to be a problem - which probably requires rekeying the whole cluster - but that's a separate issue.

external CA mode for kubeadm is documented here:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#external-ca-mode

This states:

"It is also possible to provide just the ca.crt file and not the ca.key file (this is only available for the root CA file, not other cert pairs)."

(my emphasis on "also"). So this is another mode of operation, requiring you to perform all signing externally. There's no technical issue here, just more work involved.

[neolit123]

we decided to refactor and graduate the alpha kubeconfig command, despite the overlap with the "kubeadm admin" phase:
https://github.com/kubernetes/kubeadm/issues/1381

closing, but this discussion is linked in the above issue.
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

we decided to refactor and graduate the alpha kubeconfig command, despite the overlap with the "kubeadm admin" phase:
https://github.com/kubernetes/kubeadm/issues/1381

closing, but this discussion is linked in the above issue.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.