Kops: Using autoscaler with instancegroups, spot instances, that can go down to zero

[chrislovecnm]

We should be able to have an instancegroup in kops that is set up to go down to zero. Setup the instance group with a maxPrice, and have auto-scaler control the group.

https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-0.6.1

With that combination, you can have auto-scaler control a very affordable group of k8s nodes, and significantly reduce the price of the nodes.

cc: @Rishka

[rishka]

related docs: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0

[chrislovecnm]

@andrewsykim can you comment on this? Do we have a setup that we can reference that works?

[andrewsykim]

This is supported in GCE at the moment: https://github.com/kubernetes/autoscaler/issues/82. I believe AWS support is pending: https://github.com/kubernetes/autoscaler/issues/131.

cc @mumoshu

[andrewsykim]

Unfortunately, I've never tried it and can't provide a setup that works :(

[rishka]

As far as I can tell, this works. I spun up a temporary cluster to test it. It will scale the Instance Group to 0, and will allow scaling up from 0

It does require CA 0.6.1

You need to use cloudLabels for the ASG tags in the IG spec like:

cloudLabels:
    k8s.io/cluster-autoscaler/node-template/label/ig: nodes

And you need to add permissions for autoscaler to describe ASGs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

I can open a PR to add this to the docs, just not sure it belongs in kops. This is all documented in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md

[rishka]

@andrewsykim AWS support is pending for price based autoscaling. Scale to 0 was added in 0.6.1

[Globegitter]

Only tangiably related and I already posted it on another issue here, but it would be great if an instancegroup of kops can work somehow with https://github.com/cristim/autospotting, further improving costs.

[mikn]

And you need to add permissions for autoscaler to describe ASGs:
[...]

Just want to chip in here, it turns out that this will be reset on each cluster update right now, as you can see here:

  IAMRolePolicy/masters.[...]
    PolicyDocument      

                                  {
                                    "Effect": "Allow",
                                    "Action": [
                                      "autoscaling:DescribeAutoScalingGroups",
                                      "autoscaling:DescribeAutoScalingInstances",
                      -               "autoscaling:DescribeAutoScalingInstances",
                      -               "autoscaling:DescribeTags",
                      -               "autoscaling:DescribeLaunchConfigurations",
                                      "autoscaling:SetDesiredCapacity",
                                      "autoscaling:TerminateInstanceInAutoScalingGroup"

                                    ],

[stijndehaes]

In the latest version of kops 1.8.0 the autoscaling:DescribeLaunchConfigurations is added to the master permissions. autoscaling:DescribeTags is not added but according to the documentation of cluster autoscaler this is only needed when you use --node-group-auto-discover.

I have been testing with 0 as a min for autoscaling and it works.

[whereisaaron]

With cluster-autoscaler 1.0.2 I can get regular ASGs to scale to zero, however I can't get spot price ASGs to scale to zero, even when the min-size is set to zero. Anyone else achieved zero for spot price ASGs?

[rishka]

It worked with 0.6.1 when I originally tested it, not sure if 1.0.* still does

[wskinner]

Has anything changed in the last 3 months? I am looking at running cluster using AWS spot instances, and the ability to scale all the way to zero is important. Based on this thread, it looks like 1.8.0 is the latest version that scaling to zero for spot ASGs works with.

[whereisaaron]

I have been able to scale down to zero with spot-price ASGs. In some cases it was working, but you have to be patient because there is a limit to how fast ASGs will change and to how fast the auto-scaler will act. You need "autoscaling:DescribeLaunchConfigurations" in order to scale back up from zero.

[cristim]

@Globegitter as the author of AutoSpotting I've heard of people successfully running it with kops, you just need to tag your groups to what it expects or configure it to use some tags you already have.

Please let me know if you run into any issues with it.

[bhack]

I think that also a spot rescheduler is generally useful. Check introduction and motivation sections in https://github.com/pusher/k8s-spot-rescheduler/

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close