Kops: Mixed instance types and labels

[coderanger]

One issue we've had with using mixed instance policies has been that there doesn't seem to be a great way to tell if the node is spot or not. What would you all say about checking the instance metadata in nodeup and setting a label for it? I know we removed the automatic label setting stuff back in 1.16 because it moved to the kubelet, but this would be a different kind of thing?

[rifelpet]

I think this sounds reasonable. The two approaches I see are having nodeup check the instance metadata and add it to kubelet's --node-labels, or extend kops-controller to query the EC2 API in response to new nodes and label them through kube-apiserver. I suppose it depends on what the label key would be and whether kubelet has permission to set it, as you mentioned:

Labels in the 'kubernetes.io' namespace must begin with an allowed prefix (kubelet.kubernetes.io, node.kubernetes.io) or be in the specifically allowed set (beta.kubernetes.io/arch, beta.kubernetes.io/instance-type, beta.kubernetes.io/os, failure-domain.beta.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.kubernetes.io/region, failure-domain.kubernetes.io/zone, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/instance-type, kubernetes.io/os)

Is there a standard label for this purpose? I found this EKS Blog Post which uses Lifecycle=spot and this terraform eks module which uses kubernetes.io/lifecycle=spot but otherwise don't see anything "official".

Perhaps node.kubernetes.io/lifecycle=spot?

[johngmyers]

A common label is node-role.kubernetes.io/spot-worker

[hakman]

NodeUp should be as fast as possible to get the instance running. I wouldn't complicate it more for this. kops-controller seems a good place for this.

The label itself, maybe node.kubernetes.io/instance-lifecycle=spot.
As @johngmyers said, node-role.kubernetes.io/spot-worker is/was also used, but not sure how much, except for k8s-spot-termination-handler.

[rifelpet]

The downside of using kops-controller is more EC2 API requests, in this case ec2.DescribeInstances, rather than being able to use the compute metadata endpoint.

My mistake, it turns out that spot information is not available from metadata, so the EC2 API will be required which makes kops-controller the most sensible solution. I think it would require adding a new field to nodeidentity.Info, setting it in the aws IdentifyNode and then setting the appropriate label in node_controller.go.

Google has a fair amount of search results for node-role.kubernetes.io/spot-worker so I'd be inclined to go with that.

[atmosx]

NodeUp should be as fast as possible to get the instance running. I wouldn't complicate it more for this. kops-controller seems a good place for this.

The label itself, maybe node.kubernetes.io/instance-lifecycle=spot.
As @johngmyers said, node-role.kubernetes.io/spot-worker is/was also used, but not sure how much, except for k8s-spot-termination-handler.

Is anyone working on this? I'd be interested in working on this.

[MoShitrit]

Hey @atmosx ,
Doesn't seem like anyone's on that issue right now. Feel free to assign it to yourself by commenting /assign.
Thanks!

[atmosx]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[atmosx]

/close

[k8s-ci-robot]

@atmosx: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.