Amazon-ecs-agent: Agent fails to propagate tags from EC2 Instance

[PettitWesley]

Summary

When using ECS_CONTAINER_INSTANCE_PROPAGATE_TAGS_FROM=ec2_instance the Agent can sometimes fail to add tags to the container instance.

In my case, I have an autoscaling group which propagates tags to EC2 instances. Because of the nature of distributed services, it seems that when the Agent calls ec2:DescribeTags there are no tags on the EC2 instance. So the Agent registers the container instance without tags. However, when I query the tagging API a few minutes later, I see that my EC2 instance does have tags. This seems to be a sort of race condition, when the Agent calls ec2:DescribeTags it gets nothing; key line in the Agent log:

2019-03-11T17:57:56Z [INFO] Retrieved Tags from EC2 DescribeTags API:
[]

See: https://github.com/aws/amazon-ecs-cli/pull/745#issuecomment-471692550

Expected Behavior

Container Instance has tags

Observed Behavior

Container Instance does not have tags

Environment Details

Latest ECS Optimized Amazon Linux 2 AMI with ECS Agent 1.26.0 and Docker version 18.06.1-ce

[petderek]

Are the ec2 instances created with tags or are the tags added after creation?

[PettitWesley]

@petderek They get added in the autoscaling group: https://github.com/aws/amazon-ecs-cli/blob/master/ecs-cli/modules/clients/aws/cloudformation/cluster_template.go#L740

And we set PropagateAtLaunch to true: https://github.com/aws/amazon-ecs-cli/blob/master/ecs-cli/modules/clients/aws/cloudformation/cluster_template.go#L50

[haikuoliu]

@PettitWesley

When you use PropagateAtLaunch, the tags are not guaranteed to be propagated to instance when the instance is started, it may be added to the instance after launch, in which case Agent will not register the instance tags as ECS tags. For now you can use ECS_CONTAINER_INSTANCE_TAGS as a work around (looks like you already did).

I will mark this as a bug, the fix could probably be Agent updates the tags with backend periodically or ASG supports attaching instance tags on create when using PropagateAtLaunch.

Thanks,
Haikuo

[fenxiong]

I tested this with a cfn template that has a cluster and an autoscaling group with 3 instances and a tag (https://gist.github.com/fenxiong/8ec9529bb1c23e510bc4dea0d3ed9955) and a test program that creates the stack, checks tag for each container instance and deletes the stack.

After running the test 100 times, 300 out of 300 container instances all have the tag propagated from the autoscaling group. So seems like this isn't an issue anymore. Closing

[PettitWesley]

After running the test 100 times, 300 out of 300 container instances all have the tag propagated from the autoscaling group. So seems like this isn't an issue anymore.

👏