Terraform-provider-aws: Overwrite tags to an EBS attached to an EC2 instance when volume_tags is specified

[hashibot[bot]]

_This issue was originally opened by @notuscloud as hashicorp/terraform#18860. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Terraform Version

Terraform v0.11.7
+ provider.aws v1.30.0

Terraform Configuration Files

I use https://github.com/terraform-aws-modules/terraform-aws-ec2-instance/blob/v1.9.0/main.tf which rely on the volume_tags directive.

Debug Output

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.db.aws_instance.this_t2
      volume_tags.%:         "5" => "4"
      volume_tags.Env:       "MyEnv" => "MyEnv"
      volume_tags.Name:      "company-stack-db-ec2" => "company-stack-db-ec2"
      volume_tags.Selector:  "company-stack-db-data" => ""
      volume_tags.Stack:     "stack" => "stack"
      volume_tags.Terraform: "true" => "true"

Expected Behavior

The tag volume_tags.Selector belongs to an EBS provisioned by terraform beforehand, by an another team with its own separate set of files. The volume is then attached to an EC2 instance thanks to an aws_volume_attachment.

When applying modifications to the instance, it should not update the tags of the attached EBS, only the tags of the volumes created directly by the aws_instance resource.

Actual Behavior

Terraform changes the tags of every attached disk, even those not created when you spawned the instance.

Steps to Reproduce

1. Create an EBS with terraform
2. Create an EC2 instance with the volume_tags directive.
3. Attach the previously created EBS to the EC2 instance
4. Update the volume_tags value then terraform apply

[karnauskas]

Just hit same thing.

[HontoNoRoger]

Seems to be a different behaviour than before.
I hit the same with the provider at version v1.43.2.

With version <= v1.40.0 that wasn't the case for me and behaved as expected (root EBS gets the tags from volume_tags specified in the aws_instance resource, additionally created and attached EBS get their own specified tags)

[bmathieu33]

Here is my observations, in the hope it could help to write a patch. Please note that I don't know much of terraform internals, and not much about go, so it's unfortunate that I cannot provide a patch and a test case.

readVolumeTags and setVolumeTags are using getAwsInstanceVolumeIds to get the list of volumes. Theses are the functions that respectively collect tags on all attached volumes, and update tags. The former is called by resourceAwsInstanceRead, the latter by resourceAwsInstanceUpdate. On can see at line 909 that it's not called when d.HasChange("volume_tags") is false, which is the case when there is no diff or that it's not specified at all (in resource config and in current state).

getAwsInstanceVolumeIds is likely the culprit: it queries all volumes attached to running instance, whereas it should probably rely on IDs of discovered block devices in d object (schema.ResourceData type).

In resourceAwsInstanceRead one can see that readVolumeTags is called before readBlockDevices; it should be called after so that getAwsInstanceVolumeIds can use the list of discovered block devices.

From theses observations I believe it would be enough to fix the problem with attached volumes.

How about volumes declared in instance resource? From what I understand, any change here would trigger a resource recreation.

EDIT
Actually readBlockDevices will collect all volumes ids. It still has the advantage to distinguish the root block device from other volumes. So getAwsInstanceVolumeIds should check if ebs_block_device is set in config to decide if it should use theses volume ids:

• if it is set, use all volume ids discovered
• if it is not set, use only root volume id: we are in the case of attached volumes. The case were ebs_block_device was declared cannot happen as an update operation because in such case a resource recreation is triggered

[lttmtins]

Same here. +1 to upvote this.

[raypettersen]

This is a problem for us, and is affecting DLM snapshot lifecycle as the disk we want backed up are not visible due to sporadic tag-changes.

[caparisi]

Idem +1

[MMollyy]

Same here. Still an issue.
Currently it is forming a (somewhat) large unidentified amount of costs on our organisations billing, because resources can't be tagged properly.
Similar to https://github.com/terraform-providers/terraform-provider-aws/issues/5609
or https://github.com/terraform-providers/terraform-provider-aws/issues/5080

[klebediev]

+1 for the issue

an obvious workaround:

  lifecycle {
    ignore_changes = ["volume_tags"]
  }

[MMollyy]

I might agree with that if it applied, but yes. We did use that workaround in the past too just to get around the terraform state difference every time.

[tdorrodt]

Same as of v2.14.0. Don't see anything in v.2.15.0 that addresses this. +1 to upvote.

[klebediev]

The issue was addressed in #12226
Please, upvote