Rke: Specify taints to nodes

[superseb]

Ability to specify taints to nodes in cluster.yml

[moelsayed]

Add extra_arge: to the nodes to be passed to the kubelet maybe ?

[superseb]

Could be, need to test if multiple --node-labels arguments get merged. But I think labels/taints param to node would be nice.

[moelsayed]

They get merged, we do this to assign multiple labels to nodes with multiple roles.

I was thinking of a generic solution, but explicit params for this do sound better!

[galal-hussein]

@superseb @moelsayed node-lables is still an alpha feature in k8s 1.8, so i don't think we should add an explicit option for it in rke, and the user may still add it through extra_args, and you can assign it multiple times so it will have no problem as @moelsayed pointed out.

[rossedman]

@superseb @moelsayed @galal-hussein Is there a way to do this that is recommended? Use case would be I want nodes labeled so my addons can utilize taints and tolerations when standing up clusters.

[japaniel]

I would also like to taint the nodes during cluster creation.

[mei-tech]

I would also like to taint the nodes during cluster creation.

Looking forward to see this also!

[tlvenn]

Hi, is there any update on this please ?

[nicknezis]

I also want this feature. I think I have a solution that I'll try to submit a merge request for. But I first need to update rancher/types repo. I'm having some Go dependency problems though.

[nicknezis]

I've updated RKEConfigNode and RKEConfigNodePlan to include Taints similar to how Labels exist. But I'm having issues with go generate or make failing. Would someone be able to show me the steps to update rancher/types?

[nicknezis]

Ok, I might need help getting rancher/types to generate the code that's causing the CI build to fail, but I believe the following two Pull Requests would satisfy this feature request (unless I'm missing something). I'll do some testing to make sure. (https://github.com/rancher/types/pull/829 & https://github.com/rancher/rke/pull/1349)

[mkjpryor-stfc]

I would also like to see this feature.

The ability to add custom taints to nodes, in the same way that custom labels can be applied currently, would be really useful. A use-case would be having some nodes dedicated to infrastructure, such as running the ingress controller, custom registry or a storage system like rook.

Of course the built-in addons, like ingress, would also need to support the specification of tolerations as well as node_selector.

[alena1108]

Can be tested with v0.3.0-rc8

[nicknezis]

@alena1108 Should I close my merge requests which also added this feature? (rancher/types#829 & #1349)

[sowmyav27]

Verified with rke v0.3.0-rc8

Scenario#1

• Add taints on one node - worker node in an existing cluster.

Cluster.yml file:

nodes:
  - address: ip-1
    internal_address: int-ip-1
    user: ubuntu
    role: [etcd]
    ssh_key_path: <path>
  - address: ip-2
    internal_address: int-ip-2
    user: ubuntu
    role: [controlplane]
    ssh_key_path: <path>
  - address: ip-3
    internal_address: int-ip-3
    user: ubuntu
    role: [worker]
    ssh_key_path: <path>
    taints:
      - key: test-key
        value: test-value
        effect: NoSchedule

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

• On running an rke up, Taints set on nodes - kubectl get nodes <node-name> -o json

"taints": [
            {
                "effect": "NoSchedule",
                "key": "test-key",
                "value": "test-value"
            }
        ]

Scenario#2

• Add taints on another node in an existing cluster in the cluster.yml file, and do an rke up.
• Taint gets added on the 2nd node. The taint on the initial node is intact. Verified using kubectl on the node.

Scenario#3

• Remove taints from nodes with existing taints from the cluster.yml file.
• do an rke up.
• Taint is removed from the node. Checked using kubectl get nodes <node-name> -o json command on the node.

Scenario#4

• Add taints on nodes, while creating the cluster, on all nodes.

    taints:
      - key: test-key-etcd
        value: test-value-etcd
        effect: NoExecute

• Do an rke up.
• Taints are Available on all the nodes.
• But the rke up resulted in warnings

INFO[0053] [addons] Executing deploy job rke-coredns-addon 
WARN[0084] Failed to deploy addon execute job [rke-coredns-addon]: Failed to get job complete status for job rke-coredns-addon-deploy-job in namespace kube-system 
INFO[0084] [dns] DNS provider coredns deployed successfully 
INFO[0084] [addons] Setting up Metrics Server           
INFO[0084] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0084] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0084] [addons] Executing deploy job rke-metrics-addon 
WARN[0116] Failed to deploy addon execute job [rke-metrics-addon]: Failed to get job complete status for job rke-metrics-addon-deploy-job in namespace kube-system 
INFO[0116] [ingress] Setting up nginx ingress controller 
INFO[0116] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0116] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0116] [addons] Executing deploy job rke-ingress-controller 
WARN[0148] Failed to deploy addon execute job [rke-ingress-controller]: Failed to get job complete status for job rke-ingress-controller-deploy-job in namespace kube-system 
INFO[0148] [addons] Setting up user addons              
INFO[0148] [addons] no user addons defined              
INFO[0148] Finished building Kubernetes cluster successfully 

Note: The above scenario#4, when tested with taint: NoSchedule on the nodes initially, no warnings seen while doing an rke up.

Scenario#5

• On existing nodes with taints, change the taint effect. E.x. NoExecute on all nodes to NoSchedule.
• Do an rke up.
• Taints are Available as expected on all the nodes.
• But the rke up resulted in warnings

INFO[0047] [addons] Executing deploy job rke-metrics-addon 
WARN[0108] Failed to deploy addon execute job [rke-metrics-addon]: [k8s] Job [rke-metrics-addon-deploy-job] deletion timed out. Consider increasing addon_job_timeout value 
INFO[0108] [ingress] Setting up nginx ingress controller 
INFO[0108] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0108] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0108] [addons] Executing deploy job rke-ingress-controller 
WARN[0169] Failed to deploy addon execute job [rke-ingress-controller]: [k8s] Job [rke-ingress-controller-deploy-job] deletion timed out. Consider increasing addon_job_timeout value 
INFO[0169] [addons] Setting up user addons              
INFO[0169] [addons] no user addons defined              
INFO[0169] Finished building Kubernetes cluster successfully 

[sowmyav27]

Verified with v0.3.0-rc9.

• Scenario 4 and Scenario 5 works fine.
• No warnings/errors seen during rke up.
• Taints are available on nodes
• metric server and ingress controller are deployed.