Amazon-eks-ami: Support for instance draining?

[igrayson]

Hi all,

Is it within scope for this AMI to drain pods from a node during:

1. scaling events (i.e., listen to the ASG lifecycle events)
2. manually triggered reboot/terminations

For context, here's how kube-aws approaches this.

[max-rocket-internet]

That's deployed as a DaemonSet:

if the auto scaling group signals some instance must be terminated, the script will detect the change and will perform the node draining

I doubt they would include it in this repo but you could perhaps create the daemonset yourself.

As a side note though, normally the cluster-autoscaler would perform this task. i.e. the ASG does nothing and the cluster-autoscaler is selecting nodes to terminate and doing the draining. If a node was unhealthy, i.e. EC2 healthcheck failed and then I guess the ASG would trigger the termination though.

[micahhausler]

As @max-rocket-internet said, node draining would be managed outside this repo likely in a separate project.

[thomasjungblut]

@igrayson / @max-rocket-internet did you find a concrete way to approach this?

[tckb]

Anyone found a way to achieve this ?

[thomasjungblut]

We implemented it via cloudformation LifecycleHook (autoscaling:EC2_INSTANCE_TERMINATING) and in k8s we have a daemonset that listens for the termination state and starts to drain the respective instance. After that is done, it completes the lifecycle action in CF.

So kinda like the kube-aws PR from the OP does. Would love to eventually see this in https://github.com/aws/containers-roadmap/issues/139

[tckb]

@thomasjungblut interesting to see that. I have a CF template for rolling updates and had imagined to use lambdas to trigger this. But, this is more effective, any chance the deamonset is OSS? I'd like to implement it similar to the one from OP

[thomasjungblut]

I sadly can't easily share it with you (or not that fast), however take a look at the kube-aws PR, I took most of the building blocks from there, eg. the deployment and daemonset:
https://github.com/danielfm/kube-aws/blob/master/core/controlplane/config/templates/cloud-config-controller#L2463
https://github.com/danielfm/kube-aws/blob/master/core/controlplane/config/templates/cloud-config-controller#L2550

I did some minor adjustments around notifications and rbac, but it worked pretty much out of the box. Cool thing is that it also supports spot instances with the same logic.

here are the cloudformation pieces:
https://github.com/kubernetes-incubator/kube-aws/pull/674/files#diff-7824eb2dc187893e9c52944adc99cebcR223

The lifecycle hook is also quite well documented, the example section even describes this usecase:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-as-lifecyclehook.html

[tckb]

Thanks for the tips! As of now I have most the building blocks but, the missing part is detecting the instance shutdown. It would have been straight forward with spot instance as you can poll for termination notice.

With normal ec2, it’s not looking easy without using lambdas

[thomasjungblut]

Yep, I agree. This is indeed what this deployment does:
https://github.com/danielfm/kube-aws/blob/master/core/controlplane/config/templates/cloud-config-controller#L2527-L2539

So it grabs the autoscaling group with the right tag (we had to add it to our terraform/cloudformation scripts) and just writes the ones that are in terminating state to a configmap. The daemonset then checks whether the current node is in that list and proceeds to drain.

It still felt a bit complex, since the deployment could drain it as well, but there are also some advantages of having it as a daemonset.

[tckb]

@thomasjungblut thanks for the great tips! I just rewrote complete functioning node-drainer for EKS 🎉 🎉 🎉 the RBAC was a bit tricky though. After removing the helm stuff out, the entire thing worked out of the box with some minor adjustments.

It works so:

The node-drainer kicks in as soon as the ASG starts _terminating_ the nodes and the daemonset pod running on that node takes care of draining itself and then informs the ASG that it can "continue" terminating this node.

[wintergren]

@tckb that sounds cool, any possibility you could share it? Also looking for ways to handle just this.

[tckb]

@anderssoder may be you could try the tips from @thomasjungblut if things don't work out, I will help you out.

[wintergren]

@tckb thanks, I'll check it out later!

[wintergren]

@tckb So got around to try get this working, but ran into this with the status-update pod:

Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=configmaps", GroupVersionKind: "/v1, Kind=ConfigMap"
Name: "kube-node-drainer-status", Namespace: "kube-system"
Object: &{map["apiVersion":"v1" "kind":"ConfigMap" "metadata":map["name":"kube-node-drainer-status" "namespace":"kube-system" "annotations":map["kubectl.kubernetes.io/last-applied-configuration":""]] "data":map["asg":""]]}
from server for: "STDIN": configmaps "kube-node-drainer-status" is forbidden: User "system:serviceaccount:kube-system:default" cannot get configmaps in the namespace "kube-system"

Basically this is not allowed:

# Update ConfigMap to reflect current ASG state
echo "{\"apiVersion\": \"v1\", \"kind\": \"ConfigMap\", \"metadata\": {\"name\": \"kube-node-drainer-status\"}, \"data\": {\"asg\": \"$${instances_to_drain}\"}}" | kubectl -n kube-system apply -f -

The node drainer pod is running fine it seems, just the updater that's not. I must have missed something. Also running EKS. Any idea what could be missing?

[wintergren]

Nvm, I think I got the code for the wrong node-drainer DaemonSet. Mine was missing some parts I see now. Will update and try again.

[wintergren]

Nope, still same issue and the pod is in a CrashLoopBackOff Status :(

[wintergren]

I'll need to check the rbac things and how it would work for EKS in this case.

[tckb]

@anderssoder the problem is with your rbac access. You need to allow the pod to create and update ConfigMaps . Check the access privileges

[wintergren]

@tckb ok, I got the rbac working I think. Both drainer and drainer-status-updater pods are running now without errors. 3 drainer and 1 updater. However, not sure it works as intended. The rolling update I configured seems not take place anymore, instead now 3 new instances were launched before the old ones were taken down. Looking at the Cloudformation log output I saw that it considered instances terminated before they actually were/had even begun terminating. So seems it got a complete signal before it should have.

I use Terraform but setup my ASG through the aws_cloudformation_stack resource. Rolling update worked before I added the lifecycle hook and implemented all this with the node draining. Have you got it working all the way?

[tckb]

@anderssoder I have a batch size of 1, that means rolling updates are done one-node at a time. once the node is set of termination, the node drainer "drains" the node and then sends the signal to ASG that it can further continue shutting down the instance.

The life cycle hook is only needed to for the instance to "Wait" until the draining is performed.

this is what I have in my CF

 LifecycleHookSpecificationList:
        - LifecycleTransition: 'autoscaling:EC2_INSTANCE_TERMINATING'
          LifecycleHookName: 'WorkerNodeDrainingHook'
          HeartbeatTimeout: 300
          DefaultResult: 'CONTINUE'

and in the drainer you watch for this hook

##... snip ...
##... snip ...

# Instance termination detection loop
while sleep ${POLL_INTERVAL}; do
if [ -e /etc/kube-node-drainer/instances_to_drain ] && grep -q "${INSTANCE_ID}" /etc/kube-node-drainer/instances_to_drain; then
   termination_source=asg
   break
fi
done
##... snip ...
##... snip ...

 if [ "${termination_source}" == asg ]; then
   echo Notifying AutoScalingGroup that instance ${INSTANCE_ID} can be shutdown
   ASG_NAME=$(asg describe-auto-scaling-instances --instance-ids "${INSTANCE_ID}" | jq -r '.AutoScalingInstances[].AutoScalingGroupName')
   HOOK_NAME=$(asg describe-lifecycle-hooks --auto-scaling-group-name "${ASG_NAME}" | jq -r '.LifecycleHooks[].LifecycleHookName' | grep -i WorkerNodeDrainingHook)
   asg complete-lifecycle-action --lifecycle-action-result CONTINUE --instance-id "${INSTANCE_ID}" --lifecycle-hook-name "${HOOK_NAME}" --auto-scaling-group-name "${ASG_NAME}"
fi

[wintergren]

@tcbk I have the same setup with a batch size of 1 and the same code in the drainer. Have now get everything working as good as it gets by now, can see in the pod logs that the drainer does drain the node and send the complete-lifecycle-action to the ASG. However, I still find that the rolling update policy does not respect the lifecycle hook, it seems to run in parallel and do not wait on the hook before proceeding. When studying the cloudformation output, asg activity history, ec2 instances, as well as watching the logs of the different pods during an update process I see that (in order):

1. CF starts by temp updating asg size
2. Adds new instance to ASG
3. Get SUCCESS signal from the new instance
4. Terminating old instances, replacing with 1 new instance (<< says the event output in CF console)
5. Launching new instance (<< asg activity log)
6. Successfully terminated instance Progess 33% (<< according to CF events, note that it is not terminated
7. New instance added
8. Terminating old instances, replacing with 1 new instance
9. First instance is terminated (reported in ASG activity log)

Snippet of CF output with timestamps:

16:07:28 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | New instance(s) added to autoscaling group - Waiting on 1 resource signal(s) with a timeout of PT15M.
-- | -- | -- | -- | --
  | 16:07:28 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Successfully terminated instance(s) [i-0401d02b02cd30bc1] (Progress 33%).
  | 16:06:35 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Terminating instance(s) [i-0401d02b02cd30bc1]; replacing with 1 new instance(s).
  | 16:06:34 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Received SUCCESS signal with UniqueId i-0cca70488c3038c85
  | 16:05:52 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | New instance(s) added to autoscaling group - Waiting on 1 resource signal(s) with a timeout of PT15M.
  | 16:04:57 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Temporarily setting autoscaling group MinSize and DesiredCapacity to 4.
  | 16:04:57 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Rolling update initiated. Terminating 3 obsolete instance(s) in batches of 1, while keeping at least 3 instance(s) in service. Waiting on resource signals with a timeout of PT15M when new instances are added to the autoscaling group.

Output from ASG activity log with timestamps:

Successful | Terminating EC2 instance: i-07c83312cbe983c58 | 2019 March 18 16:08:12 UTC+1 | 2019 March 18 16:11:51 UTC+1
-- | -- | -- | --
  | Successful | Launching a new EC2 instance: i-03d3e1d48e342a9fb | 2019 March 18 16:06:44 UTC+1 | 2019 March 18 16:07:16 UTC+1
  | Successful | Terminating EC2 instance: i-0401d02b02cd30bc1 | 2019 March 18 16:06:35 UTC+1 | 2019 March 18 16:08:58 UTC+1
  | Successful | Launching a new EC2 instance: i-0cca70488c3038c85 | 2019 March 18 16:05:10 UTC+1 | 2019 March 18 16:05:43 UTC+1

Notice the end time of activities in the activity log and compare with the timestamps of events in the CF log you can see what I'm saying is actually happening. The expected output would have been the rolling update would have launched a new instance and terminate old one, wait until termination is complete before taking the next set and replace (launch new + terminate old). Right now it just goes on launching new ones without waiting. Most telling is the CF event that says instance is terminated when in fact it is is not - this is reported the same second as a new instance is added to the group oddly enough.

Are you really sure that you don't have the same behavior? If you don't then there must be something else that is different in our setups.

My lifecycle:

TerminateHook:
    Condition: DrainerEnabled 
    Type: AWS::AutoScaling::LifecycleHook
    Properties:
      AutoScalingGroupName: !Ref ASG
      DefaultResult: CONTINUE
      HeartbeatTimeout: "${var.drainer_heartbeat_timeout}"
      LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING"
      LifecycleHookName: "nodedrainer"

Updatepolicy:

UpdatePolicy:
      # Ignore differences in group size properties caused by scheduled actions
      AutoScalingScheduledAction:
        IgnoreUnmodifiedGroupSizeProperties: 
          !If [IsIgnoreUnmodified, true, false]
      AutoScalingRollingUpdate:
        MaxBatchSize: "${var.cfn_update_policy_max_batch_size}"
        MinInstancesInService: "${var.min_size}"
        MinSuccessfulInstancesPercent: "${var.cfn_update_policy_min_successful_instances_percent}"
        PauseTime: "${var.cfn_update_policy_pause_time}"
        SuspendProcesses: ["${join("\",\"", var.cfn_update_policy_suspended_processes)}"]
        WaitOnResourceSignals:
          !If [IsWaitOnResourceSignals, true, false]

batch is 1, waitonresourcesignals true etc etc

[wintergren]

Ok, looking again I see one difference. You have defined your lifecycle within the Autoscaling block using the LifecycleHookSpecificationList, while I have it created separate and referencing the ASG. I'll change to do the same as you and try again.

[thomasjungblut]

YAML is hard, I actually had the same issue :)

[wintergren]

@thomasjungblut yeah, good thing is you learn a lot! :)
Sounds promising at least, I must be on the right track. Have made the changes, but need to leave work now so will have to wait to test it until tomorrow.

[tckb]

@anderssoder I don't have extra signals in my CF. point to note is, I have all params in the CF as parameters, this is the only way it worked for me.
``` ClusterWorkerASG:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AvailabilityZones: !Ref AvailabilityZones
LaunchConfigurationName: !Ref ClusterWorkerLC
MaxSize: !Ref MaximumCapacity
MinSize: !Ref MinimumCapacity
DesiredCapacity: !Ref MinimumCapacity
VPCZoneIdentifier: !Ref VPCZoneIdentifier
TerminationPolicies:
- OldestLaunchConfiguration
- OldestInstance
HealthCheckType: EC2
HealthCheckGracePeriod: 300
LifecycleHookSpecificationList:
- LifecycleTransition: 'autoscaling:EC2_INSTANCE_TERMINATING'
LifecycleHookName: 'WorkerNodeDrainingHook'
HeartbeatTimeout: 300
DefaultResult: 'CONTINUE'
Tags:
- Key: Name
Value: !Ref ClusterWorkerName
PropagateAtLaunch: true
- Key: !Sub kubernetes.io/cluster/${ClusterName}
Value: owned
PropagateAtLaunch: true
UpdatePolicy:
AutoScalingRollingUpdate:
MinInstancesInService: !Ref MinimumCapacity
MaxBatchSize: !Ref MaxBatchSize
PauseTime: !Ref UpdatePauseTime

in my terraform:

resource "aws_cloudformation_stack" "XXXXXXX" {
....
template_body = "${file("aws/worker-asg-lc-cf.yaml")}"
.....
parameters {
....

MaximumCapacity   = "${var.instance-count * 2}"
MinimumCapacity   = "${var.instance-count}"
AvailabilityZones = "${join(",",aws_subnet.tooling-cluster.*.availability_zone)}"
VPCZoneIdentifier = "${join(",",aws_subnet.tooling-cluster.*.id)}"

# we don't wait for too long
UpdatePauseTime = "PT1M"

# we would roll the updates 1-at-a-time
MaxBatchSize = "1"

}
```

[wintergren]

Thanks @tckb, I have already some as CF params, will convert the rest as well. And remove the cfn-signal + waitonsignal stuff. Simplify a bit and hope it works.

[wintergren]

Soon give up on this :(

@tckb Removed signals and changed so everything is used as CF parameters as you did. Also decreased the PauseTime to 1 min as you have. Still see the same behavior as before. New instances are launched without waiting for the previous batch to complete/being fully terminated.

Here's my terraform for the stack:

resource "aws_cloudformation_stack" "default" {
  count = "${var.enabled == "true" ? 1 : 0}"

  name = "terraform-${module.label.id}"
  tags = "${module.label.tags}"

  parameters = {
    AutoScalingGroupName   = "${module.label.id}"
    VPCZoneIdentifier      = "${join(",", var.subnet_ids)}"
    LaunchTemplateId       = "${join(",", aws_launch_template.default.*.id)}"
    LaunchTemplateVersion  = "${aws_launch_template.default.latest_version}"
    MinSize                = "${var.min_size}"
    MaxSize                = "${var.max_size}"
    LoadBalancerNames      = "${join(",", var.load_balancers)}"
    TargetGroupARNs        = "${join(",", var.target_group_arns)}"
    ServiceLinkedRoleARN   = "${var.service_linked_role_arn}"
    PlacementGroup         = "${var.placement_group}"
    IgnoreUnmodified       = "${var.cfn_update_policy_ignore_unmodified_group_size_properties}"
    WaitOnResourceSignals  = "${var.cfn_update_policy_wait_on_resource_signals}"
    NodeDrainEnabled       = "${var.node_drain_enabled}"
    UpdatePolicyPauseTime  = "${var.cfn_update_policy_pause_time}"
    HeartbeatTimeout       = "${var.drainer_heartbeat_timeout}"
    HealthCheckType        = "${var.health_check_type}"
    HealthCheckGracePeriod = "${var.health_check_grace_period}"
    TerminationPolicies    = "${join(",", var.termination_policies)}"
    MetricsGranularity     = "${var.metrics_granularity}"
    Metrics                = "${join(",", var.enabled_metrics)}"
    Cooldown               = "${var.default_cooldown}"
    MaxBatchSize           = "${var.cfn_update_policy_max_batch_size}"
  }

  on_failure = "${var.cfn_stack_on_failure}"

  template_body = <<STACK
Description: "${var.cfn_stack_description}"
Parameters:
  AutoScalingGroupName:
    Type: String
  VPCZoneIdentifier:
    Type: CommaDelimitedList
    Default: ""
  LaunchTemplateId:
    Type: String
    Default: ""
  LaunchTemplateVersion:
    Type: String
    Default: ""
  MinSize:
    Type: String
    Default: ""
  MaxSize:
    Type: String
    Default: ""
  LoadBalancerNames:
    Type: CommaDelimitedList
    Description: The load balancer names for the ASG
    Default: ""
  TargetGroupARNs:
    Type: CommaDelimitedList
    Description: Target group ARNs for the ASG
    Default: ""
  ServiceLinkedRoleARN:
    Type: String
    Description: ARN of the role the ASG uses to call other AWS services with
    Default: ""
  PlacementGroup:
    Type: String
    Description: The name of an existing cluster placement group into which you want to launch your instances.
    Default: ""
  IgnoreUnmodified:
    Type: Number
    Default: 0
  WaitOnResourceSignals:
    Type: Number
    Default: 0
  NodeDrainEnabled:
    Type: String
    Default: 0
  UpdatePolicyPauseTime:
    Type: String
    Default: PT5M
  HeartbeatTimeout:
    Type: Number
    Default: 300
  HealthCheckType:
    Type: String
    Default: EC2
  HealthCheckGracePeriod:
    Type: Number
    Default: 300
  TerminationPolicies:
    Type: CommaDelimitedList
    Default: ""
  MetricsGranularity:
    Type: String
  Metrics:
    Type: CommaDelimitedList
    Default: ""
  Cooldown:
    Type: String
    Default: 300
  MaxBatchSize:
    Type: Number
    Default: 1
Conditions:
  DrainerEnabled: !Equals [ !Ref NodeDrainEnabled, "true"]
  HasLoadBalancers: !Not [ !Equals [ !Join [ "", !Ref LoadBalancerNames], ""]]
  HasTargetGroupARNs: !Not [ !Equals [ !Join [ "", !Ref TargetGroupARNs], ""]]
  HasServiceLinkedRoleARN: !Not [ !Equals [ !Ref ServiceLinkedRoleARN, ""]]
  HasPlacementGroup: !Not [ !Equals [ !Ref PlacementGroup, ""]]
  IsIgnoreUnmodified: !Equals [ !Ref IgnoreUnmodified, 1]
  IsWaitOnResourceSignals: !Equals [ !Ref WaitOnResourceSignals, 1]
Resources:
  ASG:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: !Ref AutoScalingGroupName
      VPCZoneIdentifier: !Ref VPCZoneIdentifier
      LaunchTemplate:
        LaunchTemplateId: !Ref LaunchTemplateId
        Version: !Ref LaunchTemplateVersion
      MinSize: !Ref MinSize
      MaxSize: !Ref MaxSize
      LoadBalancerNames: 
        !If [HasLoadBalancers, !Ref LoadBalancerNames, !Ref "AWS::NoValue"]
      LifecycleHookSpecificationList:
        - LifecycleTransition: "autoscaling:EC2_INSTANCE_TERMINATING"
          DefaultResult: CONTINUE
          HeartbeatTimeout: !Ref HeartbeatTimeout
          LifecycleHookName: "nodedrainer"
      HealthCheckType: !Ref HealthCheckType
      HealthCheckGracePeriod: !Ref HealthCheckGracePeriod
      TerminationPolicies: !Ref TerminationPolicies
      ServiceLinkedRoleARN:
        !If [HasServiceLinkedRoleARN, !Ref ServiceLinkedRoleARN, !Ref "AWS::NoValue"]
      MetricsCollection:
        -
          Granularity: !Ref MetricsGranularity
          Metrics: !Ref Metrics
      PlacementGroup:
        !If [HasPlacementGroup, !Ref PlacementGroup, !Ref "AWS::NoValue"]
      TargetGroupARNs:
        !If [HasTargetGroupARNs, !Ref TargetGroupARNs, !Ref "AWS::NoValue"]
      Cooldown: !Ref Cooldown
    # CreationPolicy:
    #   AutoScalingCreationPolicy:
    #     MinSuccessfulInstancesPercent: "${var.cfn_creation_policy_min_successful_instances_percent}"
    #   ResourceSignal:
    #     Count: "${var.cfn_signal_count}"
    #     Timeout: "${var.cfn_creation_policy_timeout}"
    UpdatePolicy:
      # Ignore differences in group size properties caused by scheduled actions
      # AutoScalingScheduledAction:
      #   IgnoreUnmodifiedGroupSizeProperties: 
      #     !If [IsIgnoreUnmodified, true, false]
      AutoScalingRollingUpdate:
        MaxBatchSize: !Ref MaxBatchSize
        MinInstancesInService: !Ref MinSize
        # MinSuccessfulInstancesPercent: "${var.cfn_update_policy_min_successful_instances_percent}"
        PauseTime: !Ref UpdatePolicyPauseTime
        # SuspendProcesses: ["${join("\",\"", var.cfn_update_policy_suspended_processes)}"]
        # WaitOnResourceSignals:
        #   !If [IsWaitOnResourceSignals, true, false]
    # DeletionPolicy: "${var.cfn_deletion_policy}"
Outputs:
  AsgName:
    Value: !Ref ASG
  STACK
}

ASG activity history:

Successful | Terminating EC2 instance: i-0d3ac81b220ee6b08 | 2019 March 19 16:44:38 UTC+1 | 2019 March 19 16:48:12 UTC+1
-- | -- | -- | --
  | Successful | Launching a new EC2 instance: i-066d3eeafc070a516 | 2019 March 19 16:42:57 UTC+1 | 2019 March 19 16:43:29 UTC+1
  | Successful | Terminating EC2 instance: i-0f5b44155a6d9b099 | 2019 March 19 16:42:43 UTC+1 | 2019 March 19 16:46:14 UTC+1
  | Successful | Launching a new EC2 instance: i-021f721dc7011996b | 2019 March 19 16:40:58 UTC+1 | 2019 March 19 16:41:31 UTC+1
  | Successful | Terminating EC2 instance: i-0c7a593f160d12c4c | 2019 March 19 16:40:49 UTC+1 | 2019 March 19 16:45:02 UTC+1
  | Successful | Launching a new EC2 instance: i-0380f3cab93b0645a | 2019 March 19 16:39:00 UTC+1 | 2019 March 19 16:39:32 UTC+1

We can see that start and end of termination activity can take roughly ~3-5 minutes.

CF output:

16:44:39 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Successfully terminated instance(s) [i-0d3ac81b220ee6b08] (Progress 100%).
-- | -- | -- | -- | --
  | 16:44:36 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Terminating instance(s) [i-0d3ac81b220ee6b08]; replacing with 0 new instance(s).
  | 16:43:36 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | New instance(s) added to autoscaling group - Pausing for PT1M.
  | 16:43:35 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Successfully terminated instance(s) [i-0f5b44155a6d9b099] (Progress 67%).
  | 16:42:42 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Terminating instance(s) [i-0f5b44155a6d9b099]; replacing with 1 new instance(s).
  | 16:41:42 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | New instance(s) added to autoscaling group - Pausing for PT1M.
  | 16:41:41 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Successfully terminated instance(s) [i-0c7a593f160d12c4c] (Progress 33%).
  | 16:40:48 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Terminating instance(s) [i-0c7a593f160d12c4c]; replacing with 1 new instance(s).
  | 16:39:48 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | New instance(s) added to autoscaling group - Pausing for PT1M.
  | 16:38:54 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Temporarily setting autoscaling group MinSize and DesiredCapacity to 4.
  | 16:38:54 UTC+0100 | UPDATE_IN_PROGRESS | AWS::AutoScaling::AutoScalingGroup | ASG | Rolling update initiated. Terminating 3 obsolete instance(s) in batches of 1, while keeping at least 3 instance(s) in service. Pausing for PT1M when new instances are added to the autoscaling group.

[tckb]

@anderssoder may be I misunderstood, the launching of the new instance and the termination of the older one will happen in-parallel. These are not connected; the asg will _wait_ and will proceed for termination of the instance either on wait timeout or on notification. This is how a rolling update works

[tckb]

@anderssoder for the benefit of others stuck at the same issue, Here's the cloud formation script that worked for me.

https://gist.github.com/tckb/4f93ca216af09e285a9df5601076bb34

And the terraform snippet:

resource "aws_cloudformation_stack" "tooling-cluster-worker-asg" {
  name          = "tooling-cluster-workers-asg-stack"
  template_body = "${file("worker_asg.yaml")}"

  parameters {
    ClusterName        = "${var.cluster-name}"
    ClusterWorkerName  = "${var.cluster-worker-name}"
    ClusterRegion      = "${data.aws_region.current.name}"
    ClusterCAAuth      = "${aws_eks_cluster.tooling-cluster.certificate_authority.0.data}"
    ClusterApiEndPoint = "${aws_eks_cluster.tooling-cluster.endpoint}"
    MaxPodsPerNode     = "${var.cluster-worker-max-pods}"

    NodeInstanceProfile = "${aws_iam_instance_profile.tooling-cluster-worker.name}"
    NodeImageId         = "${data.aws_ami.eks-worker-ami.id}"
    NodeInstanceType    = "${var.cluster-worker-instance-type}"
    NodeSecurityGroup   = "${aws_security_group.tooling-cluster-worker.id}"

    MaximumCapacity   = "${var.cluster-worker-instance-count * 2}"
    MinimumCapacity   = "${var.cluster-worker-instance-count}"
    AvailabilityZones = "${join(",",aws_subnet.tooling-cluster.*.availability_zone)}"
    VPCZoneIdentifier = "${join(",",aws_subnet.tooling-cluster.*.id)}"

    # we don't wait for too long
    UpdatePauseTime = "PT1M"

    # we would roll the updates 1-at-a-time
    MaxBatchSize = "1"
  }
}

hope this helps!

[wintergren]

Thanks @tckb, then I guess mine probably "works" as well if that is how it should work. I was assuming it would wait with the next batch until the first one has been replaced(new instance in service + old one terminated), instead of it as now launching new instances one by one but ignoring the termination of old ones. Also, even if it supposed to work like that with the rolling update I still find it very odd that Cloudformation reports an instance to be successfully terminated long before it has actually terminated.

But maybe it's just me who thinks the behavior is a bit off. @thomasjungblut what's your thought's on it?