Terraform: Terraform should ignore 'aws:' prefixed tags as they are reserved and cannot be updated or deleted

[pdecat]

Terraform should ignore aws: prefixed tags as they are reserved and cannot be updated or deleted through the API or Console.

A similar Chef issue was worked around by removing the aws: prefixed tags before invoking the AWS API: https://tickets.opscode.com/browse/COOK-1560

Use case

My use case is to manage AWS resources created by CloudFormation with Terraform.

The goal is to abandon CloudFormation usage, never touching the CloudFormation Stack again after the transition.

Resources created by CloudFormation have the following tags:

• aws:cloudformation:logical-id
• aws:cloudformation:stack-id
• aws:cloudformation:stack-name

cf. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html

Issue

When trying to apply a new Terraform configuration to such AWS resources, it fails:

$ terraform apply                                                                                                                                                                                        
module.sg.aws_security_group.sg-xxxxxxxx-cf-vpc-prod-ap-northeast-1-ResourceVPCSecurityGroupNAT-UNQXXXXXXX: Refreshing state... (ID: sg-xxxxxxxx)
module.sg.aws_security_group.sg-xxxxxxxx-default: Refreshing state... (ID: sg-xxxxxxxx)
module.vpc.aws_vpc.vpc-prod-ap-northeast-1: Refreshing state... (ID: vpc-xxxxxxxx)
module.subnet.aws_subnet.sub-priv-c-prod: Refreshing state... (ID: subnet-xxxxxxxx)
module.subnet.aws_subnet.sub-priv-a-prod: Refreshing state... (ID: subnet-xxxxxxxx)
module.subnet.aws_subnet.sub-pub-a-prod: Refreshing state... (ID: subnet-xxxxxxxx)
module.subnet.aws_subnet.sub-pub-c-prod: Refreshing state... (ID: subnet-xxxxxxxx)
module.vpc.aws_vpc.vpc-prod-ap-northeast-1: Modifying...
  tags.#:           "4" => "5"
  tags.Environment: "" => "prod"
module.subnet.aws_subnet.sub-priv-c-prod: Modifying...
  tags.#:           "6" => "7"
  tags.Environment: "" => "prod"
module.subnet.aws_subnet.sub-pub-c-prod: Modifying...
  tags.#:           "6" => "7"
  tags.Environment: "" => "prod"
module.subnet.aws_subnet.sub-pub-a-prod: Modifying...
  tags.#:           "6" => "7"
  tags.Environment: "" => "prod"
module.subnet.aws_subnet.sub-priv-a-prod: Modifying...
  tags.#:           "6" => "7"
  tags.Environment: "" => "prod"
Error applying plan:

5 error(s) occurred:

* aws_vpc.vpc-prod-ap-northeast-1: InvalidParameterValue: Tag keys starting with 'aws:' are reserved for internal use
        status code: 400, request id: 
* aws_subnet.sub-pub-c-prod: InvalidParameterValue: Tag keys starting with 'aws:' are reserved for internal use
        status code: 400, request id: 
* aws_subnet.sub-priv-a-prod: InvalidParameterValue: Tag keys starting with 'aws:' are reserved for internal use
        status code: 400, request id: 
* aws_subnet.sub-priv-c-prod: InvalidParameterValue: Tag keys starting with 'aws:' are reserved for internal use
        status code: 400, request id: 
* aws_subnet.sub-pub-a-prod: InvalidParameterValue: Tag keys starting with 'aws:' are reserved for internal use
        status code: 400, request id: 

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Workaround

Update: the following workaround does not work as CloudFormation leaves 'aws:' prefixed tags on resources even after their CloudFormation stack is deleted.

Detaching AWS resources from a CloudFormation is possible but not trivial, if not risky if a resource is missing the 'Retain' DeletionPolicy:

DeletionPolicy Updates

During a stack update, you cannot add or update a DeletionPolicy by itself. You can add or update a DeletionPolicy only when you include changes that add, modify, or delete resources. If you need to add or modify a DeletionPolicy and don't want to make any changes to a resource, you can use a dummy resource, such as AWS::CloudFormation::WaitConditionHandle.

cf. http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html

Excerpts from the AWS documentation regarding 'aws:' prefixed tags

Load balancers

Do not use the aws: prefix in your tag names or values because it is reserved for AWS use. You can't edit or delete tag names or values with this prefix. Tags with this prefix do not count against your tags per resource limit.

cf. http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/add-remove-tags.html

Autoscaling Groups

Do not use the aws: prefix in your tag names or values, because it is reserved for AWS use. You can't edit or delete tag names or values with this prefix, and they do not count against toward your limit of tags per Auto Scaling group

cf. http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASTagging.html

[phinze]

Sounds reasonable to me - thanks for the thorough write up! cc @catsby for tacking onto your list somewhere.

[pdecat]

The workaround I suggested above does not work as CloudFormation leaves 'aws:' prefixed tags on resources even after their CloudFormation stack is deleted.

[rokka-n]

:+1:

This is a very annoying aws "feature"

[grenzr]

Just hit this problem now trying to get Terraform to inherit an existing CloudFormation deployment. I would love to see an ability to ignore aws: tags in Terraform.

[rokka-n]

Peeps migrating to terraform from CF :)

Is this problem hard to patch?
Is the code that's creating/updating tags in one place or it's resource dependent?

[Soprano74]

Any update on this issue?

[stack72]

Closed via #7454

[rulatir]

Some of those tags can contain references to other resources, and those references should not be entirely "lost" to terraform. Ideas?

[pdecat]

Hi @rulatir, maybe Cloudformation Exports could help, see
https://www.terraform.io/docs/providers/aws/d/cloudformation_export.html and
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-exports.html

If not, please open a new issue describing your use case as this issue was closed a long time ago.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.