Eksctl: Allow enabling autoscaler on a nodegroup without needing to set the IAM policy

[rrtj3]

Why do you want this feature?
We apply custom Instance roles and profiles to our nodegroups on creation. We also want to enable autoscaling on them, but trying to set the needed config options on the same nodegroup fails:

    iam:
        instanceProfileARN: "arn:aws:iam::888888888888:instance-profile/1010-eks-edge-node-role"
        instanceRoleARN: "arn:aws:iam::888888888888:role/1010-eks-edge-node-role"
        withAddonPolicies:
            autoScaler: true

with this error:

[✖] nodeGroups[0].iam.instanceProfileARN and nodeGroups[0].iam.withAddonPolicies.autoScaler cannot be set at the same time

What feature/behavior/change do you want?
We need the autoscaling group to automatically have the needed autoscaling flags that get set on the ASG when the autoScaler IAM policy is added:

https://github.com/weaveworks/eksctl/commit/4d186fe68f04758da213af92bbfc9fcf8370a614

but not have to apply the IAM policy. (We can add the IAM policy ourselves with our custom iam policies we're already using.)

Do not hesitate, when appropriate, to share the exact commands or API you would like, and/or to share a diagram (e.g.: asciiflow.com): "a picture is worth a thousand words".

[rndstr]

edit: see post by Adam below, it is already possible to do manually

~there are a couple options to implement this~

~1
allow both iam.withAddonPolicies.autoScaler == true and an instance ARN set, create the tags and let the user know policies haven't been added~

~_this is somewhat unexpected since the field is in a setting named 'IAM Policies' but no policies are being created_~

~2
add a new field somewhere, like nodeGroups[*].autoScalerTags: true|false~

~3
same as 2) but take a list of tags, this has the benefit to support other options such as node selectors (with the tag k8s.io/cluster-autoscaler/node-template/label/foo=bar)~

[adamjohnson01]

@rrtj3

This can be done using tags in the nodegroup config.

nodeGroups:
  - name: edge-nodes
    instanceType: m5.xlarge
    desiredCapacity: 1
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/yourClusterName: "owned"
    iam:
        instanceProfileARN: "arn:aws:iam::888888888888:instance-profile/1010-eks-edge-node-role"
        instanceRoleARN: "arn:aws:iam::888888888888:role/1010-eks-edge-node-role"

[artemkv]

@rrtj3

This can be done using tags in the nodegroup config.

nodeGroups:
  - name: edge-nodes
    instanceType: m5.xlarge
    desiredCapacity: 1
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/yourClusterName: "owned"
    iam:
        instanceProfileARN: "arn:aws:iam::888888888888:instance-profile/1010-eks-edge-node-role"
        instanceRoleARN: "arn:aws:iam::888888888888:role/1010-eks-edge-node-role"

I tried to set tags, but they were not applied.

UPD: actually there were applied, there just weren't in the generated CloudFormation template. I expected to see them there, but they were not. But the actual nodes that were created had the tags applied. So, I am not sure how those tags are actually passed to nodes, but it seems that it's not through the CloudFormation.

Below the original comment.

Here is my config:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: xxxxx
  region: eu-west-1

nodeGroups:
  - name: ng-workers-114
    instanceType: m5a.large
    minSize: 1
    maxSize: 20
    desiredCapacity: 4
    tags:
      # EC2 tags required for cluster-autoscaler auto-discovery
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/xxxxx: "owned"
    ssh:
      publicKeyName: eks-ng-workers-key
    labels:
      autoscaling: enabled
      purpose: staging
    iam:
      instanceProfileARN: arn:aws:iam::5780xxxxxxxxxxxx...
      instanceRoleARN: arn:aws:iam::5780xxxxxxxxxxxx...

The only tags that I can see in the generated CloudFormation template are:

      Tags:
      - Key: Name
        PropagateAtLaunch: 'true'
        Value: xxxxx-ng-workers-114-Node
      - Key: kubernetes.io/cluster/xxxxx
        PropagateAtLaunch: 'true'
        Value: owned

I am not a specialist in GO, so I might be wrong, but looking at the code, it seems that the tags are not passed from the config, but hard-coded, controlled by asg-access flag:

    tags := []map[string]interface{}{
        {
            "Key":               "Name",
            "Value":             fmt.Sprintf("%s-%s-Node", n.clusterSpec.Metadata.Name, n.nodeGroupName),
            "PropagateAtLaunch": "true",
        },
        {
            "Key":               "kubernetes.io/cluster/" + n.clusterSpec.Metadata.Name,
            "Value":             "owned",
            "PropagateAtLaunch": "true",
        },
    }
    if api.IsEnabled(n.spec.IAM.WithAddonPolicies.AutoScaler) {
        tags = append(tags,
            map[string]interface{}{
                "Key":               "k8s.io/cluster-autoscaler/enabled",
                "Value":             "true",
                "PropagateAtLaunch": "true",
            },
            map[string]interface{}{
                "Key":               "k8s.io/cluster-autoscaler/" + n.clusterSpec.Metadata.Name,
                "Value":             "owned",
                "PropagateAtLaunch": "true",
            },
        )
    }

    asg := nodeGroupResource(launchTemplateName, &vpcZoneIdentifier, tags, n.spec)

This matches the observed result. But you seem to say that somehow the tags should still be applied. What am I missing?

The reason why we use custom instance roles and profile:

• We use kube2iam, so we have create a custom role for each microservice, e.g. "my-service-role"
• The role "my-service-role" needs to have a policy allowing to assume role that the node runs with
• So, in essense, every microservice references the node role
• When we upgrade the k8s cluster, the suggested sequence is to create a new nodegroup, then delete the old nodegroup
• This creates a new node role and profile, and deletes the old ones
• So if we don't use custom role for the node, every role for every microservice needs to be updated to allow assuming the new node role created with the new node group
• Morover, every service needs to be re-deployed for the changes to take effect
• This has a very disruptive effect on our services

So it seems all we need is to be able to pass tags using config file.

[adamjohnson01]

@artemkv

It looks like this is not longer working in the newer versions of eksctl, it is working for us but we use an older verison of eksctl. I will have a look and see if I can figure out when this changed.

[artemkv]

@artemkv

It looks like this is not longer working in the newer versions of eksctl, it is working for us but we use an older verison of eksctl. I will have a look and see if I can figure out when this changed.

Hi @adamjohnson01 ,
thank you for such a prompt response, I updated my comment with the latest info.

I turns out the tags actually were applied, they just weren't in the generated CloudFormation template. I expected to see them there, but they were not. But the actual nodes that were created had the tags applied. So, I am not sure how those tags are actually passed to nodes, but it seems that it's not through the CloudFormation.

So looks like everything is working, I am just curious how the tags are passed, because I cannot see them in CloudFormation and I can't find anything in the code that does that. But it's not that important. We use the version "0.6.0"

[adamjohnson01]

@artemkv ah ok, the tags are added to the cloudformation stack and propogated to anything created by the stack. They will not show up in the template itself.

[artemkv]

@adamjohnson01 OK... Indeed, I see them on the stack. The mystery solved :) Thank you again!

[rrtj3]

Haven't tested this yet but thank you for letting us know how this can be done!