Kops: Launch Templates support

[dzoeteman]

1. Describe IN DETAIL the feature/behavior/change you would like to see.

AWS has released the Launch Templates feature a while ago. This seems to allow better control of launching instances, and recently allows for quite a new features with ASGs. Features like Combine purchase models are of particular interest, at least to us at SAA. I'd say best option would be to migrate away from Launch Configurations to Launch Templates.

2. Feel free to provide a design supporting your feature request.

Obviously quite a few changes will need to be done to kops. It seems from first glance that all features from Launch Configurations are available in Launch Templates.
It would have to be researched how we can go about moving this.

I'm happy to do more research on this topic (what will need to be done, any possible drawbacks, some testing) and help integrate this into kops if there's consensus that it would be a good idea to move to Launch Templates.

I'd think that integration with the new features of ASGs like Combine purchase models would have to be a seperate issue after this one.

[mikesplain]

Hi @dzoeteman! I agree, and I think many Kops users would be interested in launch templates. Since most of us don't have too much time for development time, it could be some time till we have it implemented until a contributor is really interested in getting it going! (We love contributors!)

That said, I think there a few points to make:

• Per IG config: this should probably be a per Instance Group config and default to LC's to start
• We need a "migration path" from LCs to LTs... so a config that exists in LCs should be fully compatible with LTs. We should error if a user attempts to go from LT to LC where a given value doesn't exist.

That said, if anyone want to work on this, feel free to reach out to me or any other owners for pointers!

[dzoeteman]

@mikesplain I would definitely be interested to work on this, as my company could really make use of the new features with LTs.

You mentioned migration path. Do you think this would be a good way to go? (bullet point per minor version (1.1x)):

• Add functionality with option per IG (i.e. launchTemplates: true), but keep LCs default
• Make LTs default, deprecate LCs
• Remove LCs

Other than that, I definitely think we need something to automatically convert LCs to LTs. I don't think that should be very difficult; from first glance it seems like all options in LCs are available in LTs. There's even a copy option in the AWS console, though I don't see this documented in the APIs, so I'm not sure if we could make use of that functionality.

Interested to hear your thoughts.

[fmarceau90]

+1 for this!

[mikesplain]

For the record, this is being worked on in https://github.com/kubernetes/kops/pull/6277!

[disha1104]

@mikesplain Any idea, which release is this (#6277) planned for?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[rifelpet]

6277 has been released in Kops 1.12.0 stable.

/close

[k8s-ci-robot]

@rifelpet: Closing this issue.

In response to this:

6277 has been released in Kops 1.12.0 stable.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[geekofalltrades]

This was closed, but I don't think it's complete. This ticket specifies that Launch Configurations should be completely deprecated and removed in favor of Launch Templates. I just built a cluster using Kops 1.12.1 and the InstanceGroups were still using Launch Configurations.

Switching completely off of LCs is a good idea because the describe-launch-templates API call is more fully-featured than the describe-launch-configurations call - it allows filtering! Right now we have a cluster with > 200 InstanceGroups, and this results in kops update cluster getting into some heinous API backoffs describing launch configurations. It appears that, because the describe-launch-configurations API doesn't do intelligent filtering, Kops is making n^2 API calls in the number of InstanceGroups: for every autoscaling group, it describes all launch configurations, then manually filters them on the client side. Very soon, we're going to hit a point where kops update cluster times out before the backoffs resolve.

I have a coworker who's currently working on patching kops to cache several AWS API calls to get around the backoffs (they're a major operational pain point for us), but switching to Launch Templates, and then using the filtering built into that API, would also fix it.

[Deepak1100]

even kops 1.14.1 is using launch configuration. and I wanted to use M5a series but it's not supported in launch configuration. Can we reopen this ticket?

[rifelpet]

At the moment Kops will automatically choose to use Launch Templates instead of Launch Configurations if certain conditions are met, mainly if the instance group uses a mixed instance types policy:

https://github.com/kubernetes/kops/blob/298f79659a2bd9c1d6548c6c95b335b5e29e6466/pkg/model/awsmodel/context.go#L36-L42

Alternatively there is a feature flag that should inform Kops to always use Launch Templates:

https://github.com/kubernetes/kops/blob/298f79659a2bd9c1d6548c6c95b335b5e29e6466/pkg/featureflag/featureflag.go#L55

you can enable it with export KOPS_FEATURE_FLAGS=+EnableLaunchTemplates but i'm now realizing the documentation is not up to date so I will get that updated. Eventually we will flip the feature flag default so that Kops will create launch templates by default. Does this allow you to create M5a series instance groups?

[mikesplain]

Instance types aren't dependent on launch configuration vs launch templates as far as I know. We use m5a's pretty heavily and only use launch configurations.

@Deepak1100, please open an issue with details of what you're experiencing because I think it's unrelated to this thread. Thanks!

[Deepak1100]

@mikesplain it's not supported in Mumbai region at least. I have talked with aws support as well and they confirm it. from AWS console also its giving error at the end of creation with launch configuration.

```We confirmed that the m5a instance family is indeed supported in this region. Testing via the EC2 console I was successfully able to launch an instance from this family.

After our chat, I had tested the same via other configuration methods and isolated the issue specifically to Auto Scaling Group Launch Configuration.

Escalating to the Auto Scaling Launch Configuration team they have informed me that this is expected behaviour. Unfortunately, at this time the support for m5a instance family is not fully integrated by the Auto Scaling Launch Configuration service.
```