Terraform-aws-eks: EKS 1.16 Support

[sc250024]

Opening this issue to handle all changes needed for EKS 1.16, which was released on April 30th:

• https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-eks-now-supports-kubernetes-version-1-16/
• https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#1-16-prequisites

I'm submitting a...

• [ ] bug report
• [X] feature request
• [ ] support request - read the FAQ first!
• [ ] kudos, thank you, warm fuzzy

[gertjangaillet]

I just upgraded our clusters to 1.16, and I spent quite some time debugging an issue related to kubelet arguments and annotations that are apparently no longer valid, resulting in the nodes no longer joining the cluster.

We were creating worker launch templates with the following kubelet argument, exactly like described on https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/spot-instances.md

kubelet_extra_args  = "--node-labels=kubernetes.io/lifecycle=spot"

These kubernetes.io annotations are no longer accepted. I updated them in our clusters to node.kubernetes.io/lifecycle=spot which works but not 100% sure if that is a "correct"/conventional value.

[sc250024]

I just upgraded our clusters to 1.16, and I spent quite some time debugging an issue related to kubelet arguments and annotations that are apparently no longer valid, resulting in the nodes no longer joining the cluster.

We were creating worker launch templates with the following kubelet argument, exactly like described on /docs/spot-instances.md@master

kubelet_extra_args  = "--node-labels=kubernetes.io/lifecycle=spot"

These kubernetes.io annotations are no longer accepted. I updated them in our clusters to node.kubernetes.io/lifecycle=spot which works but not 100% sure if that is a "correct"/conventional value.

I ran into the same issue; I ended up using node.kubernetes.io/lifecycle=spot, but not sure if that's the convention either. I was a bit upset because it seems you also cannot use node-role.kubernetes.io since that's also a protected prefix, meaning you cannot see what roles your nodes have by running kubectl get nodes, which is frustrating.

I know --node-labels=kubernetes.io/lifecycle=spot is present in many examples, so it should probably be updated also in https://github.com/terraform-aws-modules/terraform-aws-eks/pull/857

[sc250024]

@gertjangaillet It seems the node-role label has some changed in 1.16 => https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/2019-07-16-node-role-label-use.md

[sc250024]

@gertjangaillet Also good discussion here: https://github.com/kubernetes/kubernetes/issues/75457. It seems being able to apply certain node label via the kubelet is no longer supported for security reasons.

UPDATE: Also this issue => https://github.com/sighupio/fury-kubernetes-aws/issues/42. Seems like they're having the same issue as well related to losing the output of kubectl get nodes.

[igaskin]

I just upgraded our clusters to 1.16, and I spent quite some time debugging an issue related to kubelet arguments and annotations that are apparently no longer valid, resulting in the nodes no longer joining the cluster.

We were creating worker launch templates with the following kubelet argument, exactly like described on https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/spot-instances.md

kubelet_extra_args  = "--node-labels=kubernetes.io/lifecycle=spot"

These kubernetes.io annotations are no longer accepted. I updated them in our clusters to node.kubernetes.io/lifecycle=spot which works but not 100% sure if that is a "correct"/conventional value.

Thanks for sharing your workaroud. This helped put me on the right track for solving the same problem. Additionally, other kubernetes.io node labels are no longer supported in 1.16 and will prevent nodes from joining the cluster. In my case I had to remove the alpha.service-controller.kubernetes.io/exclude-balancer=true label.

[max-rocket-internet]

Great information @gertjangaillet 😃