Amazon-eks-ami: Indicate instance ID & Lifecycle

[tsmgeek]

Can the next release add in additional node labels by default.
This would prevent the need to have custom user data setup.

• Instance ID
• Lifecycle (spot/ondemand/scheduled)

[micahhausler]

I think AMI-ID and InstanceID are good additions, but I don't know a great way to determine "Lifecycle". There is the spot/instance-action metadata key, but it 404's for non-spot instances.

[tsmgeek]

Not sure how viable this is, nodes have an IAM role that allows ec2:describeinstances.

https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html

curl http://169.254.169.254/latest/meta-data/instance-id
curl http://169.254.169.254/latest/meta-data/iam/security-credentials
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role>
---extract AWS setup--
AWS_ACCESS_KEY_ID=....
AWS_SECRET_ACCESS_KEY=....
AWS_SESSION_TOKEN=....
aws ec2 describe-instances --instance-ids=<instanceid>
---there should be a flag `instance-lifecycle` which if set will be populated with spot/scheduled, if blank then default of ondemand---

This could also be used to get a lot more info into the tags.

[micahhausler]

We can't rely on the instance having any permissions. If you want to add additional labels, you don't have to use a custom user-data, you can use the BootstrapArguments CloudFormation parameter and --kubelet-extra-args '--node-labels=lifecycle=spot'

[max-rocket-internet]

A lot of people are excited to use the new Launch Templates but this also brings the complexity of potentially having mixed on-demand and spot instances in the same ASG. Then it becomes very important to have taints and/or labels so work loads can be scheduled correctly.

So I would also like to push for some default labels applied for lifecycle here also.

Is there any convention for this existing already in the k8s projects?

@runningman84 made a nice PR for our EKS Terraform module, and I know we can add custom user-data etc, but it would be great to have something to address this in bootstrap.sh or something else from AWS 🙂

[max-rocket-internet]

@micahhausler

We can't rely on the instance having any permissions.

The policy arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy contains ec2:DescribeInstances

[micahhausler]

It is true that the instance has this permission currently, but we're working toward not giving instances that permission. The greater node label story is probably best left to a controller that can have a specific permission.

Instance ID is already a field on the node status, but a label could be added by a separate controller.

We're closing this issue in favor of having a stand-alone controller to manage these node labels

[max-rocket-internet]

We're closing this issue in favor of having a stand-alone controller to manage these node labels

OK cool @micahhausler but I would like some clarification here as I don't really understand 😅

1. Who makes the controller? Is this an AWS thing or are we (the community) supposed to do this?
2. How does this even work? The controller runs in the cluster and watches for new nodes, queries AWS for info and then adds relevant labels??

[micahhausler]

1. Who makes the controller? Is this an AWS thing or are we (the community) supposed to do this?

This is something we're working to open source, its been an ask from multiple customers

1. How does this even work? The controller runs in the cluster and watches for new nodes, queries AWS for info and then adds relevant labels??

That is the idea!

[skang0601]

@micahhausler Is the timeline for this change something you can share? This is something that would influence whether I should go forward with a similar solution proposed above or wait until this long term solution lands.

[palsivertsen]

@micahhausler I have a similar use case (#410) that could benefit from such a controller.