Aws-cdk: ecs container instance draining handler unable to get container instance arn

[jakebanks]

Hi all,

I'm not sure if this is the right place to raise, as this is an issue in CDK and potentially the ECS API also. I will try to keep both of these separate in my description.

Summary of the issue:
The container instance draining hook Lambda included with aws-ecs' Cluster class currently does not work, as it fails to get the container instance ARN to drain.

The issue with the Lambda code is that it is calling list_container_instances, which does not return the ec2InstanceId property being searched/filtered on. I found it was necessary to follow that call with describe_container_instances . I have attempted a fix with #6864

In testing my fix to that I realised that as soon as an instance is terminated* and enters the Terminating:Wait stage of the ASG lifecycle hook, it disappears from the ECS console and is no longer returned in calls to list_container_instances, meaning looking up its ARN is impossible.

*FWIW I am terminating instances from the console

Reproduction Steps

1. Deploy a cluster and add an ASG to it. I am doing this with cluster.addCapacity(). Set the mincapacity to 2. The defaults are fine for optional values.
2. Define a service and task definition. Run a copy of the task on each instance in the ASG.
3. Ensure the ASG has a lifecycle hook on it. Follow that through to find the lambda it will call.
4. Open the service on the ECS console
5. Terminate an instance with a task running on it. It is expected that this instance is set to DRAINING
6. Note the instance disappears immediately
7. Check lambda logs

Error Log

In the Cloudwatch logs:
Instance i-0aefbc4a591d4ed1c has container instance ARN None

Environment

• CLI Version :
• Framework Version:
• OS :
• Language :

Other

For the CDK fix check out #6864, however this relies on the ability to get the container instance arn for an ec2 instance id, which would mean that the ec2 instance needs to remain as a 'container instance' in ECS' eyes even during the Terminating:Wait stage of the lifecycle hook.

This is :bug: Bug Report

[kohidave]

Hey there @jakebanks sorry for the delay! Let me take a look.

[jakebanks]

This looks fixed now actually! I am using 1.31.0 of aws-autoscaling. Thanks for looking.

Instance xxxxxx has container instance ARN arn:aws:ecs:xxxxxxxxxx

[efekarakus]

Closing the issue as it appears to be fixed.