Containers-roadmap: [EKS] [request]: Managed Node Groups support for node taints

[mikestef9]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Add support for tainting nodes through managed node groups API

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Managed nodes supports adding Kubernetes labels as part of node group creation. This makes it easy for all nodes in a node group to have consistent labels. However, taints are not supported through the API.

Are you currently working around this issue?
Manual kubectl commands after new nodes in node group come up.

[TBBle]

When this was raised in #585, #507 was tagged as an existing request for this feature, but I think that was confusion... #507 seems to be about Container Insights correctly monitoring tainted nodes, while what we want here (and in #585) is to support _setting_ the taints on Managed Nodegroups as part of a rollout, e.g. with eksctl.

The comment in #585 had nine thumbs-up, on top of the three currently here.

[mikestef9]

@TBBle correct, I wanted to open a separate issue to explicitly track tainting node groups through the EKS API

[karstenmueller]

@mikestef9 we would like to see "tainting node groups through the EKS API" progressing and bumped it from #12 👍 to #37 as of now.

[aviau]

It looks like the bootstrap script used by eks nodes already support taints. My understanding is that it would be a small feature to implement because it would only require to modify the userdata in the launch template to add extra args, just like its done for labels currently.

[AlbertoPeon]

We would love to have this!

[jhcook-ag]

"When nodes are created dynamically by the Kubernetes autoscaler, they need to be created with the proper taint and label.
With EKS, the taint and label can be specified in the Kubernetes kubelet service defined in the UserData section of the AWS autoscaling group LaunchConfiguration."

https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/eks-auto-scaling-nodes

[TBBle]

@jhcook-ag You can't specify the UserData for Managed Node Groups when you create them.

You can modify the UserData in the Launch Configuration in the AWS console after creation, but then the Managed Node Groups feature will refuse to touch your Launch Configuration again, and you're effectively now using unmanaged Node Groups, although eksctl will still try to use the Managed Node Groups API and fail.

[jhcook-ag]

@mhausenblas we really need this 👍

[borisputerka]

Absolutely would love the idea.

[Lincon-Freitas]

It is a must-have feature!

[vcucereanu]

👍

[martinoravsky]

This is a must-have feature for us as well. We can't use managed node groups because of this. When would you expect this to be released? (just roughly) 👍

[Dudssource]

Hi @martinoravsky, I believe this feature is available now.

https://aws.amazon.com/blogs/containers/introducing-launch-template-and-custom-ami-support-in-amazon-eks-managed-node-groups/

We did it by customizing the userdata on the custom launch template and specifying the taints for the kubelet (using the register-with-taints argument).

[martinoravsky]

Hi @Dudssource ,

are you using custom AMIs? I'm using launch templates with EKS optimized AMIs which include UserData that bootstraps the node to the cluster automatically (with --kubelet-extra-args empty). This userdata is not editable for us, we can only add our own UserData as MIME multipart file which has no effect on bootstrapping the cluster. I'm curious if you were able to get this to work without custom AMIs.

[Dudssource]

@martinoravsky, yes unfortunately we had to use a custom AMI for this to work.
But we used the same optimized AMI that EKS uses, we use terraform so we used a datasource to get the latest AMI for our cluster version. I know that this is possible with Cloudformation and parameter store too.

[mikestef9]

The approach that @Dudssource used here is certainly an option, but we do plan to add taints directly to the EKS API (similar to labels), so that a custom AMI is not required.

[lwimmer]

I've found a solution (admittedly quite hackish) to allow setting taints with the offical AMIs:

Set the userdata for the Launch Template similar to this:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==7561478f-5b81-4e9d-9db6-aec8f463d2ab=="

--==7561478f-5b81-4e9d-9db6-aec8f463d2ab==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
sed -i '/^KUBELET_EXTRA_ARGS=/a KUBELET_EXTRA_ARGS+=" --register-with-taints=foo=bar:NoSchedule"' /etc/eks/bootstrap.sh

--==7561478f-5b81-4e9d-9db6-aec8f463d2ab==--\

This script is run before the bootstrap script, which is managed by EKS, patching the /etc/eks/bootstrap.sh to inject the necessary --register-with-taints in the KUBELET_EXTRA_ARGS variable.
This solution is not perfect and might break if AWS changes the bootstrap script, but it works for now and can be used until there is proper support for taints.

[dwilliams782]

@lwimmer That is superbly hacky! Good work.

I'm really surprised this feature is missing, and overall I'm shocked how feature incomplete node groups are.

[markthebault]

+1

[DovAmir]

+1

[ArchiFleKs]

Thanks @lwimmer , I tried to implement your logic here in the official terraform module: https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1138

[andre-lx]

Thanks @lwimmer. I made it based on your solution. This is my full node group with custom taints:

resource "aws_eks_node_group" "some_nodegroup" {
  node_group_name = "some_nodegroup"

  cluster_name  = aws_eks_cluster.eks_cluster.name
  node_role_arn = aws_iam_role.eks_nodegroup_role.arn
  subnet_ids    = aws_subnet.public_subnet.*.id

  instance_types = [
    ...,
  ]

  scaling_config {
    desired_size = ...
    max_size     = ...
    min_size     = ...
  }

  launch_template {
    id      = aws_launch_template.some_launch_template.id
    version = aws_launch_template.some_launch_template.latest_version
  }

  labels = {
    type = "some-label"
  }

  depends_on = [
    aws_iam_role_policy_attachment.iam-eks-nodegroup-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.iam-eks-nodegroup-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.iam-eks-nodegroup-AmazonEC2ContainerRegistryReadOnly,
  ]

  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

resource "aws_launch_template" "some_launch_template" {
  name = "some_lauch_template"

  block_device_mappings {
    device_name = "/dev/xvda"

    ebs {
      volume_size = 20
      volume_type = "gp2"
    }
  }

  user_data = base64encode(<<-EOF
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==7561478f-5b81-4e9d-9db6-aec8f463d2ab=="

--==7561478f-5b81-4e9d-9db6-aec8f463d2ab==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
sed -i '/^KUBELET_EXTRA_ARGS=/a KUBELET_EXTRA_ARGS+=" --register-with-taints=some_taint1=true:NoSchedule,some_taint2=true:NoSchedule"' /etc/eks/bootstrap.sh

--==7561478f-5b81-4e9d-9db6-aec8f463d2ab==--\
  EOF
  )

  tag_specifications {
    resource_type = "instance"

    tags = {
      Name = "..."
    }
  }
}

For multiple taints:

<key1>=<value1>:<effect1>,<key2>=<value2>:<effect2>