Kubeadm: API server authorization-mode order not as specified in kubeadm config

[ghouscht]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:11:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  none, on-prem
• OS (e.g. from /etc/os-release):

NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

• Kernel (e.g. uname -a):

Linux myhost 3.10.0-957.21.3.el7.x86_64 #1 SMP Fri Jun 14 02:54:29 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

When setting up a cluster with kubeadm the order of arguments for the kube-apiserver flag authorization-mode is not preserved.
In our case, the authorization-mode (apiServer->extraArgs) is set to Node,Webhook,RBAC (see sample config below). The generated kube-apiserver manifest (/etc/kubernetes/manifests/kube-apiserver.yaml) does not reflect this configuration and the flag is set to --authorization-mode=Node,RBAC,Webhook.

The order of the arguments is crucial as it's also the order in which api calls are authenticated by the apiserver. In our case (for cluster admins) RBAC allows nearly every operation, so the webhook is never called, but we require the webhook to be called first for our companys own so called "priviledged access management".

What you expected to happen?

The authorization-mode flag in the kube-apiserver manifest to look like this:
--authorization-mode=Node,Webhook,RBAC

How to reproduce it (as minimally and precisely as possible)?

kubeadm configuration kubeadm.yaml - stripped to the absolute minimum to reproduce the problem:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.15.3
apiServer:
  extraArgs:
    "authorization-mode": "Node,Webhook,RBAC"

1. run kubeadm init --config kubeadm.yaml
2. run grep authorization-mode /etc/kubernetes/manifests/kube-apiserver.yaml
3. Observe, that the order of the authorization mode is Node,RBAC,Webhook and not Node,Webhook,RBAC as specified in the config.

Anything else we need to know?

The configmap in the kube-system namespace is also wrong.

[ghouscht]

Found the code which is responsible for this behaviour manifests.go#L197.
Not what I expected as a user and probably also not documented (at least found nothing about that...). In such cases I expect if there is a knob to tune a setting that the tool around respects my settings and assumes the user knows what he is doing.

My approach to "fix" this would be to preserve the default behaviour which adds Node,RBAC and as soon as the user specifies another mode like Webhook to assume he knows what he is doing and preserve the order given by the user. I can do the work on the code to change this if you think thats ok, just let me know.

[neolit123]

the logic in https://github.com/kubernetes/kubernetes/blob/efcb62abff0ee6511cb5f82b9f25d28b17d81912/cmd/kubeadm/app/phases/controlplane/manifests.go#L197
is wrong.

if the user passes authorization-mode it should completely overwrite the the kubeadm defaults.

if you have the option to soon use 1.16 we are enabling an alpha feature to customize the static pods with patches. https://github.com/kubernetes/kubeadm/issues/1379

[neolit123]

/kind bug
/priority important-longterm

[ghouscht]

thx, I'll open a PR to fix this

[neolit123]

thx, I'll open a PR to fix this

it probably needs discussion by the maintainers first as it's a breaking change to existing users.
but you can send the PR if you want to showcase the problem.

[neolit123]

he had a talk about this during the meeting today and we decided to allow the user to completely override the auth-mode, but print a warning ( klog.Warning ), if it's not the default kubeadm values.

we are already doing this for kubelet security defaults in other places.

@ghouscht LMK, if you send a PR for that so that we can help with reviews and get it merged.
thanks

[ghouscht]

PR with a possible fix is now open.
I did some more work around this and added a bit of code to validate the users input trying to provide an "easy to use" user experience. Simpler solution would be to just take what the user provided in the configuration, but could be odd if there is something which the apiserver does not understand.

[rosti]

The main problem that we have here is that we try to interpret the authorization-mode extra arg in the ClusterConfiguration, patch it and then store it back as an extra arg.
This is wrong and breaks the assumption, that extra arg values completely override the kubeadm specified values.
Hence, we have a couple of possibilities in here:

1. Allow users to completely override authorization-mode via extraArgs for the API server in the ClusterConfiguration. We should also warn them if Node and RBAC modes are not specified in the override as this may break the cluster.
2. Introduce a config option in the API Server settings in the ClusterConfiguration. Something like this can be done:

apiVersion: kubeadm.x-k8s.io/v1beta3
type: ClusterConfiguration
apiServer:
  AuthorizationMode:
  - Node
  - WebHook
  - RBAC

Whichever option we choose, however, we must implement it only for a new config version. The behavior of the existing config versions should not be changed. Failure to do so, can cause issues for existing users upon upgrades or replicating old deployments with new kubeadm versions.

/kind api-change

[ghouscht]

I can understand that you're concerned about breaking existing users - which is of course totally valid.

As I already tried to point out in my PR, the key question for me here is, is the current behavior a bug, or desired? If it is a bug we should fix it despite probably breaking some users. If this behavior was/is desired we should at least document it.

The documentation in the code of a ControlPlaneComponent says:

ExtraArgs is an extra set of flags to pass to the control plane component.

--> so there is no hint about an interpretation of those args. For me (as a non native English speaker) the word "pass" lets me assume that the args. are not interpreted.

The documentation about control plane flags here (which also references the kube-apiserver cmd reference) also states nothing about the interpretation of the extraArgs, moreover it says:

The extraArgs field consist of key: value pairs. To override a flag for a control plane component:

    1. Add the appropriate fields to your configuration.
    2. Add the flags to override to the field.
    3. Run kubeadm init with --config <YOUR CONFIG YAML>.

The key point here for me is the word "override", and so comes my assumption that I'm in full control when it comes to providing extraArgs - which actually is only the case for some args.

A try to address your concern about breaking existing users:

1. Why should a user come up with a config like the one you posted in the referenced PR

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    "authorization-mode": "Webhook"

and not the following config

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    "authorization-mode": "Node,RBAC,Webhook"

when there is not a single hint in the docs that the first configuration will work?

1. Naturally a user would expect that all extraArgs behave in the same way which also speaks for the second config.

In conclusion, probably most of the users who specified an extra authorization-mode (which is not very common in my opinion, because this a "more advanced configuration" not relevant for most people using kubeadm) already have a correct configuration which would also work when we address this issue without a new config version.

Of course this is only an assumption and my personal (opinionated) view.

Long story short: I think this is a bug and should be addressed despite the possibility to break existing users.

[neolit123]

i'm +1 on the pending PR, as is and the breaking change.
also i think we should not the expose the auth mode in the kubeadm config (saw the discussion during the planning).

[Mattias-]

+1 for addressing this.
I'm looking for a way to set Webhook as the first authorization-mode.
I have no opinion regarding how that would be implemented though as I only use kubeadm with kind.

[rosti]

@ghouscht I can see your point. However, code is also a form of documentation. Specifically, that piece of code was designed to work in that way and users may be aware of that through reading the code.
Another point is accidental use. Users may not even be aware how this works, they just supplied it like that and it works.
And my third point is, that we stamp configs with versions. These versions guarantee both format and behavior. Hence, if something was labeled v1beta1 or v1beta2, then it should work as such regardless of the kubeadm version.

However, I am fine with going with the breaking change and an "action required" release note as long as the rest of the approvers are happy with it.