Eksctl: Spot instance without mixed instance group

[kimxogus]

Why do you want this feature?
It seems spot instances are only supported with mixed instance groups .
As aws cluster-autoscaler doesn't support mixed instance group with spot instances yet and seems to take some time, this feature would be helpful.
Spot termination may make asg with single instance type of spot instance unstable, but it's still useful for batch job worker and ci worker nodes.

What feature/behavior/change do you want?
Spot Instance support without mixed instance group.

[overdrive3000]

I think CA is able to scale ASG using Launch Templates with mixed instance distribution the trick is to use a set of Instance Types that shares the same amount of vCPU and Memory check this out https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-autoscalinggroup-mixedinstancespolicy

[errordeveloper]

Yes, my understanding was that it currently works under certain conditions, and improvements are work-in-progress.

/cc @ruecarlo @Jeffwan

[ranshn]

The PR for support for Mixed ASGs in cluster-autoscaler has been merged and will hopefully be released soon: https://github.com/kubernetes/autoscaler/pull/1886

[danielchalef]

Another use case: There are some instance types that are somewhat unique in capabilities and size/shape. For example, our team uses p3.2xlarge instances interactively (Jupyter notebooks) and non-interactively (polyaxon experiments). Other instances in the p3 family are large and expensive (dissimilar size/shape) or p2 family are older, less capable gpus (dissimilar capabilities).

Being able to scale out groups of spot p3.2xlarge instances would be very useful.

[Jeffwan]

Spot with MixedInstancePolicy is already supported in CA with some limitations.

I am talking about better support for MixInstancePolicy in this issue. https://github.com/kubernetes/autoscaler/issues/2057

There's some technically challenges since CA assumes single instance type per node group and MixedInstancePolicy is against core assumption

[Jeffwan]

@danielchalef

If training and notebook instances are in same cluster, normally we suggest users to use nodeAffinity. But the problem MixedInstancePolicy can not be used along with NodeAffinity.

Your case has different usage of two instance types. Use more ASG makes more sense, if there's single instance type in MixedInstancePolicy overrides, node Affinity can work

[kimxogus]

CA before version 1.14 doesn't support launch templates because of aws-sdk version. https://github.com/kubernetes/autoscaler/pull/2018 https://github.com/kubernetes/autoscaler/pull/2019

CA 1.14.x currently seems to work fine in my company's k8s(kops) 1.12 cluster, but as it's not tested in k8s 1.12.x which is latest supported version in eks, I would prefer spot instances without mixed instance policy.

[Jeffwan]

@kimxogus Your requirement makes sense. I also see group of users in CA have similar case, but user have to manage spot instance (specially the price and multiple ASGs to increase availability).

We don't run compatibility tests across different versions. Even it's working fine based on real world experience, there's no guarantee everything works perfect using a latest CA version on 1.12

[albertmichaelj]

@danielchalef @Jeffwan
I'm trying to follow the discussion here about trying to use a specific instance type with spot instances. I also would like to specify a particular instance type in an ASG, but use spot instances. My understanding is that this isn't possible right now with eksctl (since spot instances require a mixed instance group), but @Jeffwan seems to indicate this is possible.

@danielchalef Are you currently using ASGs with a spot instance of a particular instance?

[TarekAS]

@Jeffwan don't regular Launch Templates (without mixed policy) already support automatic spot pricing? I already run my spot nodegroups like this, so why is this a challenge to integrate with eksctl?

[Jeffwan]

@TarekAS eksctl already has the support for MixedInstancePolicy. It doesn't come with the challenge.

I think originally we are discussing Spot without MixedInstancePolicy? The reason I am talking about spot within MixedInstancePolicy is because it has several limitations. That's the reason some user wants spot instance ASG. This will give you only single instance type per ASG which is more reliable in some of the case but user has to manage spot pricing them selves.

[TarekAS]

@Jeffwan

eksctl already has the support for MixedInstancePolicy.

I am referring to Launch Templates without MixedInstancePolicy, with the option of enabling spot. We need that as a feature for multiple use-cases, other than avoiding the CA limitation.

[...] user has to manage spot pricing them selves.

Launch Templates (without MixedInstancePolicy) already support automatic spot pricing, why would the user need to manage it? In CloudFormation, it's as easy as:

InstanceMarketOptions:
          MarketType: spot

There is no reason for eksctl not to have this feature.

[Jeffwan]

@TarekAS faire enough, agree. I think we are on the same page. I didn't notice spot instance auto pricing is well supported now. eksctl could support it since it's supported in cloud formation. The use case makes sense.

[scottyhq]

Just seconding @TarekAS that this would be a convenient feature for the nodegroups we are deploying that can be entirely on spot instances (see linked issue above).

[jwalton]

Bit of a n00b question, but why does eksctl require you to specify two different instance types in the first place? All the docs say are "# At least two instance types should be specified".

[TarekAS]

@jwalton because they are using MixedInstancePolicy underneath, which requires at least 2 instance types.

The regular instance policy which supports 1 instance type is not yet integrated in eksctl, hence my suggestion above

[schmutze]

Note that MixedInstancesPolicy no longer requires at least instance 2 types and now supports between 1 and 20 instance types:

https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_LaunchTemplate.html#API_LaunchTemplate_Contents

[Jeffwan]

@schmutze Super exciting news. This help a lot in cluster-autoscaler project.

[TarekAS]

@schmutze A related question, if I specify 0% or 1% On-Demand nodes in my MixedInstancesPolicy, would the node group fallback to On-Demand in case Spot requests cannot be fulfilled?

[schmutze]

@schmutze A related question, if I specify 0% or 1% On-Demand nodes in my MixedInstancesPolicy, would the node group fallback to On-Demand in case Spot requests cannot be fulfilled?

Hi @TarekAS - no. MixedInstancesPolicy will maintain the ratio between On-Demand and Spot as specified and doesn't do any kind of failover.

[martina-if]

I believe this was solved via https://github.com/weaveworks/eksctl/pull/1772 Feel free to reopen this issue if you believe there's still any action needed.