Autoscaler: What to apply AWS Policy to?

[chapati23]

Excuse the stupid question please but I'm confused about the README a bit.
What do I need to apply the suggested IAM policy to?

I've set this up with helm and it's not working. No error messages but nodes are not scaling up when running against capacity limit and i suspect the policy might be the culprit:

echo "Installing helm chart 'aws-cluster-autoscaler'…"
envsubst < autoscaling.template.yml > autoscaling.yml
helm install stable/aws-cluster-autoscaler -f autoscaling.yml
rm autoscaling.yml

With this autoscaling.template.yml:

autoscalingGroups:
  - name: ${AUTO_SCALING_GROUP_NAME}
    maxSize: 12
    minSize: 4

awsRegion: ${AWS_REGION}

image:
  repository: gcr.io/google_containers/cluster-autoscaler
  tag: v0.5.4
  pullPolicy: IfNotPresent

For further debugging purposes, here's kubectl describe pod aws-cluster-autoscaler:

Name:       loping-armadillo-aws-cluster-autoscaler-3021239351-mwxgp
Namespace:  default
Node:       xxx.eu-central-1.compute.internal/xxx.xxx.xxx.xxx
Start Time: Wed, 07 Jun 2017 19:38:22 +0200
Labels:     app=aws-cluster-autoscaler
        pod-template-hash=3021239351
        release=loping-armadillo
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"loping-armadillo-aws-cluster-autoscaler-3021239351","uid":"19ea6...
        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container aws-cluster-autoscaler
Status:     Running
IP:     xxx.xxx.xxx.xxx
Controllers:    ReplicaSet/loping-armadillo-aws-cluster-autoscaler-3021239351
Containers:
  aws-cluster-autoscaler:
    Container ID:   docker://8449a1d6fbc09a2e52d71f2cc67b520720125743f2f0384887b94cafddb6a44f
    Image:      gcr.io/google_containers/cluster-autoscaler:v0.5.4
    Image ID:       docker-pullable://gcr.io/google_containers/cluster-autoscaler@sha256:abe1ed1410c6ea58a80afec69e2b4397740cfa4ffc02484eb0cfbe96d3e81984
    Port:       8085/TCP
    Command:
      ./cluster-autoscaler
      --cloud-provider=aws
      --nodes=4:12:nodes.my-domain.com
      --scale-down-delay=10m
      --skip-nodes-with-local-storage=false
      --skip-nodes-with-system-pods=true
      --v=4
    State:      Running
      Started:      Wed, 07 Jun 2017 19:38:22 +0200
    Ready:      True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      AWS_REGION:   eu-central-1
    Mounts:
      /etc/ssl/certs/ca-certificates.crt from ssl-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jdpp7 (ro)
Conditions:
  Type      Status
  Initialized   True
  Ready     True
  PodScheduled  True
Volumes:
  ssl-certs:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/ssl/certs/ca-certificates.crt
  default-token-jdpp7:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-jdpp7
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:     <none>

[mtcode]

The policy should be added to the IAM role belonging to the node that the cluster-autoscaler is running in. The CA will use that policy to adjust the desired value of the ASG that it is configured to control.

You can see the CA logs with kubectl logs loping-armadillo-aws-cluster-autoscaler-3021239351-mwxgp. If you see access errors that CA can't adjust the size of the ASG, then this is the issue.

[mwielgus]

cc: @mumoshu @andrewsykim

[mumoshu]

@chapati23 Hi, thanks for trying CA on AWS 👍

As @mtcode suggested, you should create and attach the minimum policy described in README to your instance profile/IAM role for k8s node(s)=EC2 instances on which CA is scheduled.

Suppose CA is being scheduled on one of worker nodes(which is common), you can browser your EC 2 instances backing the worker nodes in the AWS console to see which IAM role(s) is associated to the instances.

[chapati23]

thanks @mtcode @mumoshu, for other people struggling with this: here's the section of my setup script i ended up writing in the end

echo "7️⃣  Set up Autoscaling"

echo "   First, we need to update the minSize and maxSize attributes for the kops instancegroup."
echo "   The next command will open the instancegroup config in your default editor, please save and exit the file once you're done…"
KOPS_CONFIG_BUCKET=${PREFIX}.kops-${CLUSTER_NAME}.config
kops edit ig nodes --state s3://${KOPS_CONFIG_BUCKET} --name ${CLUSTER_URL}
kops update cluster --yes --state s3://${KOPS_CONFIG_BUCKET} --name ${CLUSTER_URL}
printf "\n"


printf "   a) Creating IAM policy to allow aws-cluster-autoscaler access to AWS autoscaling groups…"
# Unfortunately AWS does not support ARNs for autoscaling groups yet so you must use "*" as the resource.
cat > asg-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}
EOF

ASG_POLICY_NAME=aws-cluster-autoscaler
chronic aws iam list-policies | jq -r '.Policies[] | select(.PolicyName == "aws-cluster-autoscaler") | .Arn'
if [ $? -eq 0 ]
then
  printf " ✅  Policy already exists\n"
  ASG_POLICY_ARN=$(aws iam list-policies | jq -r '.Policies[] | select(.PolicyName == "aws-cluster-autoscaler") | .Arn')
else
  ASG_POLICY=$(aws iam create-policy --policy-name $ASG_POLICY_NAME --policy-document file://asg-policy.json)
  ASG_POLICY_ARN=$(echo $ASG_POLICY | jq -r '.Policy.Arn')
  printf " ✅ \n"
fi


printf "   b) Attaching policy to nodes role…"
ASG_NAME="nodes.$CLUSTER_URL"
chronic aws iam attach-role-policy --policy-arn $ASG_POLICY_ARN --role-name $ASG_NAME
printf " ✅ \n"


printf "   c) Installing aws-cluster-autoscaler…"
CLOUD_PROVIDER=aws
IMAGE=gcr.io/google_containers/cluster-autoscaler:v0.5.4
MIN_NODES=3
MAX_NODES=12
SSL_CERT_PATH="/etc/ssl/certs/ca-certificates.crt"

addon=cluster-autoscaler.yml
chronic curl -o ${addon} https://raw.githubusercontent.com/kubernetes/kops/master/addons/cluster-autoscaler/v1.6.0.yaml

sed -i -e "s@{{CLOUD_PROVIDER}}@${CLOUD_PROVIDER}@g" "${addon}"
sed -i -e "s@{{IMAGE}}@${IMAGE}@g" "${addon}"
sed -i -e "s@{{MIN_NODES}}@${MIN_NODES}@g" "${addon}"
sed -i -e "s@{{MAX_NODES}}@${MAX_NODES}@g" "${addon}"
sed -i -e "s@{{GROUP_NAME}}@${ASG_NAME}@g" "${addon}"
sed -i -e "s@{{AWS_REGION}}@${AWS_REGION}@g" "${addon}"
sed -i -e "s@{{SSL_CERT_PATH}}@${SSL_CERT_PATH}@g" "${addon}"

chronic kubectl apply -f ${addon}
printf " ✅ \n"