Kubeadm: Specify node label on node join

[mcapuccini]

Hi! After reading the docs for kubeadm, and asking in the Slack channel, I figured out that there isn't yet a node join option that allows to specify node labels. This would be very useful when using terraform with kubeadm (e.g. https://github.com/kubenow/KubeNow/blob/master/bootstrap/node.sh#L11), as we could scale the cluster with a single command, without having to run kubectl afterwards to label the new nodes.

P.S. if you could point me to the parts that need to be modified, I can try to contribute to get this in kubeadm 🙂

[fabriziopandini]

I'm working on a PR on this and #209

[mcapuccini]

Nice!! We are working it around in KubeNow at the moment https://github.com/kubenow/KubeNow/blob/master/bootstrap/node.sh

[luxas]

@fabriziopandini How do you envision fixing this?

We have to take authorization/authentication aspects into account, and they will be/are very tricky.

I'd just say that still the best approach:

cat > /etc/systemd/system/kubelet.service.d/20-labels-taints.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--node-labels=label1=value1,label2=value2 --register-with-taints=foo=bar:NoSchedule"

Also, right now, kubeadm join is unblocking (just creates the credentials for the kubelet and writes to a file). This would be a major change in behavior since it would make kubeadm join blocking to wait for the kubelet to register.

I'd like to keep the line between kubeadm join and kubelet clean.

cc @jbeda @timothysc @pipejakob @mikedanese thoughts?

[fabriziopandini]

@luxas, I started to explore an approach similar to taint and labeling for master nodes, but of course I can switch to the drop-in approach if the second one is preferred.

Apart from athentication and authorizatio, another potentially tricky area I see so far is the idenfication of the node name picked up by kubelet...

[fabriziopandini]

@luxas, I'm checking the drop-in option, and it seems that label and taints created by kubelet are applied only at node join (and thus not enforced at every node/kubelet restart). This behaviour is more than fine for me. what do you think about?

The last point to be verified on this approach is how to issue a systemctl daemon-reload command if the kubelet service is already running. Any suggestion here? (possible solution improve util\initsystem package, which is outside of cmd\kubeadm folder...)

[timothysc]

@fabianofranz Is there a PR that you wanted to get for 1.7?

[fabianofranz]

@fabianofranz Is there a PR that you wanted to get for 1.7?

I think that was for @fabriziopandini ;)

[fabriziopandini]

@timothysc this PR for me is eligible for 1.7, but it mostly depends by the agreement on the discussion above (I implemented @luxas suggested approach, but I'm not sure ifor there is convergence on it)

[fabriziopandini]

I was informed about a new PER dynamic kubelet configuration, currently in alpha - rif #46254. This will provide a cleaner solution than relying on file system/dropins. I will have a look and this and discuss the proposal in sig meetings

[jamiehannaford]

@fabriziopandini Did we reach consensus on the design for this? It seems that if we add this functionality to kubeadm join it would either need to:

1. stay non-blocking and generate systemd drop-ins and restart the unit. What about for distros that don't use systemd?
2. wait for the node to join and then send labels to the apiserver
3. use dynamic kubelet config

so maybe this is just a case of waiting for #28 to land?

[fabriziopandini]

@jamiehannaford from my understanding the solution based on the drop-in file was discarded in favor to option 3, use dynamic kubelet configuration (this is why I closed #46506 back in May).

However, now that I'm going through this thread after some times, I don't se nothing against 2 as well (in fact this is what we are already doing for mark-master). WDYT?

PS. this PR overlaps with #486

[jamiehannaford]

@fabriziopandini I've added it to tomorrow's meeting notes, let's discuss with everyone then :)

[jbeda]

I think that there is work ongoing in SIG-Auth to lock this down a bit. As a security measure, nodes may not be able to set their own labels. This might be something we can bundle into the token but that is getting a bit messy. I don't have all the details paged in here though.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[mcapuccini]

/remove-lifecycle stale

[mcapuccini]

Any news on this?

[jamiehannaford]

Should be possible once https://github.com/kubernetes/kubernetes/pull/57224 lands I think

[timothysc]

kubernetes/kubernetes#57224 landed, closing.

Also as mentioned there exists multiple operator work-arounds for this.

[cjbottaro]

I'd just say that still the best approach:

cat > /etc/systemd/system/kubelet.service.d/20-labels-taints.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--node-labels=label1=value1,label2=value2 --register-with-taints=foo=bar:NoSchedule"

@luxas what if my system (official AWS EKS image) doesn't have /etc/systemd/system/kubelet.service.d/? I'm not familiar with systemd. If I just make that dir/file, will it automatically be picked up when I start kubelet?

Thanks.

[colegatron]

@cjbottaro
For AWS EKS image kubelet-extra-args you just need to add them into the eks bootstrap.sh.
Important; remember to use single quotes to surround the args

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --use-max-pods false --kubelet-extra-args '--node-labels=kubernetes.io/trovimap-nodename=${NODENAME}-${CLUSTER_NAME}-Node' ${CLUSTER_NAME}
systemctl enable kubelet
systemctl start kubelet

[fredleger]

@cjbottaro
For AWS EKS image kubelet-extra-args you just need to add them into the eks bootstrap.sh.
Important; remember to use single quotes to surround the args

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --use-max-pods false --kubelet-extra-args '--node-labels=kubernetes.io/trovimap-nodename=${NODENAME}-${CLUSTER_NAME}-Node' ${CLUSTER_NAME}
systemctl enable kubelet
systemctl start kubelet

For reference if you need to do this on unmanaged eks node pool (created wit a cloudformation template), you will need to update the BootstrapArguments parameters with the suggested values : --kubelet-extra-args '--register-with-taints=key=value:NoSchedule --node-labels=key=value'