Terraform-provider-aws: aws_instance dynamic ebs_block_device forces replacement

[acdha]

I ran into an issue with changes being forced on every run while attempting to deploy servers using a third-party AMI while enabling storage encryption. I used a dynamic block to mimic the block_device_mappings from the AMI, which produced a working encrypted system but Terraform detects the system as changed on every run even though the listed plan shows the same values for the removed and added blocks. Unlike the condition noted in the AWS instance docs, these instances do not mix usage with aws_ebs_volume.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.24
+ provider.aws v2.59.0
+ provider.http v1.1.1

Affected Resource(s)

• aws_instance

Terraform Configuration Files

  dynamic "ebs_block_device" {
    for_each = data.aws_ami.cis_amazon_linux2.block_device_mappings
    iterator = device
    content {
      device_name = device.value["device_name"]
      encrypted   = true
      kms_key_id  = aws_kms_key.app_server.arn
      iops        = device.value["ebs"]["iops"]
      snapshot_id = device.value["ebs"]["snapshot_id"]
      volume_size = device.value["ebs"]["volume_size"]
      volume_type = device.value["ebs"]["volume_type"]
    }
  }

Expected Behavior

No changes were attempted

Actual Behavior

      - ebs_block_device { # forces replacement
          - delete_on_termination = true -> null
          - device_name           = "/dev/sdb" -> null
          - encrypted             = true -> null
          - iops                  = 100 -> null
          - kms_key_id            = "arn:aws:kms:us-east-1:…:key/4c134819-e3f1-48e1-a9f3-078c51e50247" -> null
          - snapshot_id           = "snap-0141e981aeb73231b" -> null
          - volume_id             = "vol-03883d920c410e30a" -> null
          - volume_size           = 2 -> null
          - volume_type           = "gp2" -> null
        }
      + ebs_block_device { # forces replacement
          + delete_on_termination = true
          + device_name           = "/dev/sdb"
          + encrypted             = true
          + iops                  = (known after apply)
          + kms_key_id            = "arn:aws:kms:us-east-1:…:key/4c134819-e3f1-48e1-a9f3-078c51e50247"
          + snapshot_id           = "snap-0141e981aeb73231b"
          + volume_id             = (known after apply)
          + volume_size           = 2
          + volume_type           = "gp2"
        }

Steps to Reproduce

1. terraform apply

[acdha]

A little bit of testing confirms that the problem is specific to how the dynamic block is evaluated — using the output of terraform state show avoids the constant churn.

[sudomateo]

I was able to reproduce this behavior with the following Terraform configuration.

provider "aws" {
  region = "us-east-1"
}

data "aws_ami" "centos7" {
  most_recent = true
  owners      = ["aws-marketplace"]
  filter {
    name   = "product-code"
    values = ["aw0evgkw8e5c1q413zgy5pjce"]
  }
}

data "aws_vpc" "default" {
  default = true
}

resource "aws_instance" "testing" {
  ami           = data.aws_ami.centos7.id
  instance_type = "t2.micro"

  dynamic "ebs_block_device" {
    for_each = data.aws_ami.centos7.block_device_mappings
    iterator = device
    content {
      device_name = device.value["device_name"]
      iops        = device.value["ebs"]["iops"]
      snapshot_id = device.value["ebs"]["snapshot_id"]
      volume_size = device.value["ebs"]["volume_size"]
      volume_type = device.value["ebs"]["volume_type"]
    }
  }
}

output "ami_details" {
  value = data.aws_ami.centos7
}

The first apply went through successfully but the subsequent apply reported the instance needed to be destroyed and recreated. If I use a static value in my for_each I do not run into this issue.

The output of terraform version.

Terraform v0.12.24
+ provider.aws v2.60.0

[anGie44]

The fix for this has been merged and will release with version 2.65.0 of the Terraform AWS Provider, expected in this week's release.

[acdha]

@anGie44 Thank you! I just saw your update earlier and was about to test it.

[anGie44]

awesome, yes please report report back with your findings @acdha 😄

[hashibot[bot]]

This has been released in version 2.65.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[acdha]

@anGie44 I'm not seeing a change with 2.65. Here's the full module in question. I also tried switching between using a set or a map in the for expression but that didn't change it.

Terraform v0.12.26
+ provider.aws v2.65.0
+ provider.http v1.1.1

data "aws_ami" "cis_amazon_linux2" {
  /*
    This is the same lookup on the CLI:

    aws ec2 describe-images --owners aws-marketplace --filters 'Name=name,Values="CIS Amazon Linux 2 Benchmark * Level 2*"' --query "Images[].[ImageId,ImageLocation]" --out=text
  */
  most_recent = true

  owners = ["679593333241"] # The official CIS account

  filter {
    name = "name"
    values = [
      "CIS Amazon Linux 2 Benchmark * Level 2*",
    ]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }
}

locals {
  cis_ami_block_devices = {
    for i in data.aws_ami.cis_amazon_linux2.block_device_mappings :
    i["device_name"] => i
  }
}

resource "aws_instance" "this" {
  count = var.instance_count

  tags        = merge(var.tags, { Name = "${var.name_prefix}-${count.index + 1}" })
  volume_tags = merge(var.tags, { Name = "${var.name_prefix}-${count.index + 1}" })

  instance_type        = var.instance_type
  ami                  = data.aws_ami.cis_amazon_linux2.id
  iam_instance_profile = var.iam_instance_profile
  user_data_base64     = base64encode(var.user_data)

  # Networking:
  subnet_id                   = var.subnet_ids[count.index % length(var.subnet_ids)]
  vpc_security_group_ids      = var.vpc_security_group_ids
  associate_public_ip_address = var.associate_public_ip_address

  metadata_options {
    # Require usage of the newer v2 metadata API:
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
  }

  root_block_device {
    encrypted  = true
    kms_key_id = var.kms_key_id
  }

  dynamic "ebs_block_device" {
    for_each = local.cis_ami_block_devices

    content {
      device_name           = ebs_block_device.value["device_name"]
      delete_on_termination = true
      encrypted             = true
      kms_key_id            = var.kms_key_id
      iops                  = ebs_block_device.value["ebs"]["iops"]
      snapshot_id           = ebs_block_device.value["ebs"]["snapshot_id"]
      volume_size           = ebs_block_device.value["ebs"]["volume_size"]
      volume_type           = ebs_block_device.value["ebs"]["volume_type"]
    }
  }
}

[sudomateo]

@acdha ~I believe the for expression in your locals block is causing an issue.~ See my edit below.

I tested both aws provider version 2.64.0 and 2.65.0 using the following configuration.

provider "aws" {
  region = "us-east-1"
}

data "aws_ami" "centos7" {
  most_recent = true
  owners      = ["aws-marketplace"]
  filter {
    name   = "product-code"
    values = ["aw0evgkw8e5c1q413zgy5pjce"]
  }
}

data "aws_vpc" "default" {
  default = true
}

resource "aws_instance" "testing" {
  ami           = data.aws_ami.centos7.id
  instance_type = "t2.micro"

  dynamic "ebs_block_device" {
    for_each = data.aws_ami.centos7.block_device_mappings
    iterator = device
    content {
      device_name = device.value["device_name"]
      iops        = device.value["ebs"]["iops"]
      snapshot_id = device.value["ebs"]["snapshot_id"]
      volume_size = device.value["ebs"]["volume_size"]
      volume_type = device.value["ebs"]["volume_type"]
    }
  }
}

output "ami_details" {
  value = data.aws_ami.centos7.block_device_mappings
}

When using aws provider version 2.64.0, applying the above configuration back to back showed the aws_instance.testing needed to be replaced.

When using aws provider version 2.65.0, applying the above configuration back to back showed no changes.

Edit: The AMI that I am using for my testing only has one item in its block_device_mappings. When I used the same CIS AMI that has multiple items in its block_device_mappings, I find that the aws_instance _does_ want to be replaced on back to back apply operations. This is true whether I use locals or use the data source directly in the dynamic block.

[acdha]

Ah, thank you for confirming that you see it with multiple devices – I was starting to wonder what I could be missing. The CIS Linux baseline requires separate partitions for a number of common paths so we have to have a ton of them.

[anGie44]

thank you @acdha and @sudomateo for following-up on this issue! I see the code change addressed only the case of 1 dynamic ebs_block_device 😞 My best guess atm is that the underlying structure of the ebs_block_device which is a TypeSet reports a Diff even if the blocks are calculated in the same order..but interesting that for 1 block there's no diff but for multiple blocks there is a diff 🤔

[acdha]

Sorry about not testing earlier — this week has been really hectic.

[8bitlunch]

I was able to get around a very similar issue relating to this by using lifecycle and ignore_changes function against the block device. I was using root_block_device but this might work in this case too.

[anGie44]

Hi @acdha 👋 thank you again for the time you've invested in this issue! I wanted to comment back here and note that for the time being, we've determined the best course of action for working around the provider's behavior is to do this at the config-level as changes proposed in the linked PR may be breaking for some users and existing configs. Given your use-case for example, to work-around the perpetual diff when dynamically creating ebs_block_device blocks, one should pass all the EBS blocks defined in an AMI's block_device_mappings except for the one block defined as the root and this can be done with the filtering you see below. As the provider stands atm, when passing all device_mappings, for example in the CIS AMI, at plan-time it determines there are a total of 6 dynamic ebs_block_device mappings regardless of how they are arranged (i.e. 5 are solely EBS blocks, 1 is an EBS block but used as the root); however, after the apply, terraform returns in state 5 ebs_block_device blocks (excluding the block known to be the "root"), thus resulting in the diff after a state refresh.

resource "aws_instance" "test" {
  ami           = data.aws_ami.ami.id
  instance_type = "m3.medium"
  dynamic "ebs_block_device" {
    for_each = [for bdm in data.aws_ami.ami.block_device_mappings : bdm if bdm.device_name !=  data.aws_ami.ami.root_device_name]
    iterator = device
    content {
      device_name = device.value["device_name"]
      iops        = device.value["ebs"]["iops"]
      snapshot_id = device.value["ebs"]["snapshot_id"]
      volume_size = device.value["ebs"]["volume_size"]
      volume_type = device.value["ebs"]["volume_type"]
    }
  }
}

If any further questions arise or this work-around doesn't behave as expected, please let me know :)

[acdha]

That workaround seems to be working well

[anGie44]

closing this for now, given the above config example can be used as a reference when creating an instance with dynamic EBS blocks and an additional code change in the provider is not required.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!