Kubeadm: Cannot join node if addon-kubeproxy phase has not been run

[DennisGlindhart]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.13.2

kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:33:30Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): 1.13.2
• Cloud provider or hardware configuration: Bare metal
• OS (e.g. from /etc/os-release): CentOS 7.6
• Kernel (e.g. uname -a): 3.10.0
• Others:

What happened?

When bootstrapping a master using phases, leaving out kube-proxy addon, nodes are unable to join with kubeadm join.

Error: unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kube-proxy" is forbidden: User "system:bootstrap:jgozlb" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

What you expected to happen?

Be able to join node without installing kube-proxy addon.

How to reproduce it (as minimally and precisely as possible)?

1. Bootstrap a master using (manual) phases, leaving out kube-proxy addon

$ kubeadm init phase certs all
$ kubeadm init phase kubeconfig all
$ kubeadm init phase control-plane all
$ kubeadm init phase etcd local
$ kubeadm init phase kubelet-start
$ kubeadm init phase upload-config all
$ kubeadm init phase mark-control-plane
$ kubeadm init phase bootstrap-token

1. Try to join a node

$ kubeadm join [2001:0db8::1]:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "[2001:0db8::1]:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://[2001:0db8::1]:6443"
[discovery] Requesting info from "https://[2001:0db8::1]:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "[2001:0db8::1]:6443"
[discovery] Successfully established connection with API Server "[2001:0db8::1]:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
unable to fetch the kubeadm-config ConfigMap: failed to get component configs: configmaps "kube-proxy" is forbidden: User "system:bootstrap:jgozlb" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Anything else we need to know?

Running kubeadm init phase addon kube-proxy on the master and re-run the join command on the node again, join is successful.

[neolit123]

hi,

Be able to join node without installing kube-proxy addon.

kube-proxy is an "essential" addon.
what is your use case such that you wish to omit applying it?

[DennisGlindhart]

@neolit123 Primarily using IPv6-only cluster with BGP/Calico routing where kube-proxy is, for now at least, not needed and I'm using Ingress instead of Services in general.
(https://opsnotice.xyz/kubernetes-ipv6-only/ talks about it a bit).

There is no problem in installing it for me, except I would like not to have anything not used hanging around - I like simple :)

Everything I need seems to work without kube-proxy in my case, so not "essential" in my world . Maybe there are other/better ways to do my setup nowadays, or more advanced features I'm not using that will maybe give me problems if using them, but I'm happy without for now.

I also don't install CoreDNS phase using kubeadm, but deploys it manually afterwards because I wish to adjust something in the deployment-spec - I imagine someone would maybe like to do the same with kube-proxy too, where this would probably come up as a problem.

What does kubeadm join do with the kube-proxy info/config it fetches? Are there anything that actually needs to be set up on the node? I imagine this is could be solved with something like if (configmap/kube-proxy exists) then do something; else, but I very might be wrong.

[neolit123]

thank you for the explanation. your use case seems quite plausible.

we seem to have a bug, or rather something we didn't anticipate related to phases.

when join is performed in 1.13 we now have a way to retrieve the init configuration from the control plane node to the worker node. as part of this process we also fetch the "component config" for kubelet and kube-proxy, yet if one of them is missing (e.g. kube-proxy phase omitted) we simply error out.

ref:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/config/cluster.go#L222
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/componentconfigs/config.go#L63

one possible solution would be to show warnings and not error out in such cases.

[fabriziopandini]

Related issue: https://github.com/kubernetes/kubeadm/issues/1318

The underlying topic IMO is what are "the officially supported" cluster variants Vs custom variants you can achieve using/skipping phases. Considering the potential huge number of the custom variants, my personal opinion is that it is not possible for the Sig to provide support/guarantees about the join/upgrade workflows in the custom use cases.

This is why I proposed following items in the v1.14 planning discussion

• support a new cluster variant - without kube-proxy - (Rif. Kube router use case)
• implement phases in the kubedm join workflow
  @neolit123 might be we should reconsider the low priority assigned to those tasks and raise a call for volunteers

one possible solution would be to show warnings and not error out in such cases.

This is a possible workaround for the join workflow, but after that the users will be stuck during the upgrade workflow. I'm in favour of a more structured solution as explained above (and with a dedicated CI signal)

[neolit123]

@fabriziopandini

implement phases in the kubedm join workflow
@neolit123 might be we should reconsider the low priority assigned to those tasks and raise a call for volunteers

we can discuss the list of join phases with @timothysc on Wednesday and then i can create a tracking issue for volunteers, so that someone works on this problem in parallel.

does that sound good?

[alejandrox1]

Is there any chance that this is not a bug per se?
As I understand it, Calico uses kube-proxy to manage filtering rules. So in order for pods to know where the dns service lives the cluster needs the kube-proxy component to be running in all cluster nodes. In this sense the blog post referenced by @DennisGlindhart , Kubernetes in IPv6 only fails to take this into account when claiming that "With Kubernetes in IPv6 only, Kube-Proxy become totally useless."

More related to the question in this issue would be that network plugins seem to make use of the kube-proxy component to set up pod-to-pod communications and as such it is and should be kept as an essential addon (unless replace by the kube-router?).

[DennisGlindhart]

@alejandrox1 What do you mean by filtering rules? I can use NetworkPolicy with Calico just fine without having kube-proxy in the cluster.

DNS server(s) is set to routable IP-address instead of a ServiceIP so the DNS-config could be either a list of static Pod-IP's, a Headless Service (in which case kube-proxy is not used) or could even be external.

(Newest Calico releases also supports BGP-routed ServiceIP's - I don't know how it works under the hood yet, tho).

Pod-to-Pod communication is also just routable IP's. The point is not relying on NAT which is what kube-proxy mainly does AFAIK but instead having a simple L3 network between pods.

I understand that it's a lot of combinations with every phase possibly being left out, but would at least supporting the "add-ons" to be optional be an acceptable compromise?

[pytimer]

Could kubeadm support custom selection addons?
kubeadm init use configuration file, we can enable or disable the specified addons. For example, if we use kube-router, we can disable kube-proxy.

[fabriziopandini]

@pytimer This was discussed last kubeadm office hours
As of today kubeadm does not support custom selection addons

• you can choose between CoreDNS or kube-dns only
• there are no alternatives to kube-proxy
  Using kubeadm Init phases you can create clusters without addons, but then you are on your own for join and upgrades because those workflows are not yet flexible as init (with phases)

However this doesn't means that things cannot chang, but to make this happen help from the community is needed

[neolit123]

i will try to send a proposal patch for this in this cycle, but it will be in the lines of:

• show a warning instead of error-ing out.
• notify the user that this cluster (that lacks kube-proxy) is unsupported in upgrades etc,.

/assign

[timothysc]

TODO: document support level on these changes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

we decided to close my PR here as it's far from ideal:
https://github.com/kubernetes/kubernetes/pull/77744

and punt this issue to the kubeadm upgrade apply phase support.
this is already covered in the OP here:
https://github.com/kubernetes/kubeadm/issues/1318

[neolit123]

looks like we closed this with the idea that upgrade phases will solve it but this is also a problem in join as the title indicated:

see this new similar ticket:
https://github.com/kubernetes/kubeadm/issues/1733