Terraform-provider-aws: Cannot terraform apply aws_instances into consistent state 0.6.5 (EBS)

[hashibot]

_This issue was originally opened by @gtmtech as hashicorp/terraform#3654. It was migrated here as part of the provider split. The original body of the issue is below._

---

Using terraform 0.6.5

I am unable to get terraform to apply into a consistent state - that is , an apply followed by a plan always wants to blow away all my instance resources and start over. Every time. Despite the fact terraform apply says everything was successful. Obviously this renders terraform an unusable tool for this usecase

It seems the offensive part relates to each instance having an EBS block volume, forcing a new resource despite the fact pre and post are the same values. Note the force new resource sections below when doing a subsequent plan, after a successful apply

-/+ aws_instance.foo_2
    ...
    associate_public_ip_address:                       "false" => "0"
    ...
    ebs_block_device.#:                                "2" => "1"
    ebs_block_device.158414664.delete_on_termination:  "1" => "0"
    ebs_block_device.158414664.device_name:            "/dev/xvda" => ""
    ebs_block_device.3905984573.delete_on_termination: "true" => "1" (forces new resource)
    ebs_block_device.3905984573.device_name:           "/dev/xvdb" => "/dev/xvdb" (forces new resource)
    ebs_block_device.3905984573.encrypted:             "false" => "<computed>"
    ebs_block_device.3905984573.iops:                  "30" => "<computed>"
    ebs_block_device.3905984573.snapshot_id:           "" => "<computed>"
    ebs_block_device.3905984573.volume_size:           "10" => "10" (forces new resource)
    ebs_block_device.3905984573.volume_type:           "gp2" => "gp2" (forces new resource)
    ephemeral_block_device.#:                          "0" => "<computed>"
   ...

As you can see from above, its pretty screwed, and terraform is now unusable.

This is my code for an instance (obviously a fair amount of variables in there)

resource "aws_instance" "foo_2" {
    ami                         = "${var.foo.ami_image_2}"
    availability_zone           = "${var.foo.availability_zone_2}"
    instance_type               = "${var.foo.instance_type_2}"
    key_name                    = "${var.foo.key_name_2}"
    vpc_security_group_ids      = ["${aws_security_group.base.id}",
                                   "${aws_security_group.foo.id}"]
    subnet_id                   = "${aws_subnet.foo_az2.id}"
    associate_public_ip_address = false
    source_dest_check           = true
    monitoring                  = true
    iam_instance_profile        = "${aws_iam_instance_profile.foo.name}"
    count                       = "${var.foo.large_cluster_size}"
    user_data                   = "${template_file.userdata_foo_2.rendered}"

    ebs_block_device {
        device_name = "/dev/xvdb"
        volume_type = "gp2"
        volume_size = "${var.foo.disk_size}"
        delete_on_termination = "${var.foo.disk_termination}"
    }

}

A couple of other things I've noticed -

You see the ebs block device id above? "ebs_block_device.3905984573" - well the ID is the same for all instance resources in the output, that strikes me as a bit odd.

After terraform applying all my instances, I commented out all the ebs_block_device settings in all the instance definitions, and then tried a plan. Now terraform plan says there are no changes.

It looks like EBS block devices are severely not working in terraform 0.6.5

[hashibot]

_This comment was originally opened by @gtmtech as https://github.com/hashicorp/terraform/issues/3654#issuecomment-151465422. It was migrated here as part of the provider split. The original comment is below._

---

It seems very similar to https://github.com/hashicorp/terraform/issues/1260 . We are also building with AMIs with packer.

[hashibot]

_This comment was originally opened by @catsby as https://github.com/hashicorp/terraform/issues/3654#issuecomment-151493306. It was migrated here as part of the provider split. The original comment is below._

---

Hey @gtmtech – are there any external volume attachments?

[hashibot]

_This comment was originally opened by @gtmtech as https://github.com/hashicorp/terraform/issues/3654#issuecomment-151636067. It was migrated here as part of the provider split. The original comment is below._

---

Hi, in fact following the advice in #1260, i managed to fix it.

the output of packer was a 2 vol ami. terraform cant handle that it seems, when adding another volume

[hashibot]

_This comment was originally opened by @clkao as https://github.com/hashicorp/terraform/issues/3654#issuecomment-156096294. It was migrated here as part of the provider split. The original comment is below._

---

(with terraform 0.6.6)

This also happens if you created an ami with an io1 ebs on /dev/sdb, and wish terraform to override the ebs_block_device to gp2 volume_type. The resulting instance is created properly, however the state is not idempotent so each apply tries to relaunch the instance.

[hashibot]

_This comment was originally opened by @clkao as https://github.com/hashicorp/terraform/issues/3654#issuecomment-168592456. It was migrated here as part of the provider split. The original comment is below._

---

This is still happening with 0.6.8:

    ebs_block_device.#:                                "1" => "1"
    ebs_block_device.1247340589.delete_on_termination: "1" => "0"
    ebs_block_device.1247340589.device_name:           "/dev/sdb" => ""
    ebs_block_device.2576023345.delete_on_termination: "" => "1" (forces new resource)
    ebs_block_device.2576023345.device_name:           "" => "/dev/sdb" (forces new resource)
    ebs_block_device.2576023345.encrypted:             "" => "0" (forces new resource)
    ebs_block_device.2576023345.iops:                  "" => "<computed>" (forces new resource)
    ebs_block_device.2576023345.snapshot_id:           "" => "<computed>" (forces new resource)
    ebs_block_device.2576023345.volume_size:           "" => "80" (forces new resource)
    ebs_block_device.2576023345.volume_type:           "" => "gp2" (forces new resource)

The ebs_block_device config in question is:

  ebs_block_device {
    device_name = "/dev/sdb"
    volume_type = "gp2"
    delete_on_termination = 1
    encrypted = false
    volume_size = 80
  }

while the ami launched is with io1-type block device for /dev/sdb

[hashibot]

_This comment was originally opened by @Shruti29 as https://github.com/hashicorp/terraform/issues/3654#issuecomment-200270406. It was migrated here as part of the provider split. The original comment is below._

---

Running terraform 0.6.13 and still hitting this issue.
This happens for only some AMIs.

[hashibot]

_This comment was originally opened by @Shruti29 as https://github.com/hashicorp/terraform/issues/3654#issuecomment-212269380. It was migrated here as part of the provider split. The original comment is below._

---

I am trying to use terraform in production and failing to do so because of this issue. I understand this was fixed before in #1260. Can someone help me fix this issue?
@phinze

Using terraform version : v0.6.14

1. Not using any packer AMIs. We are importing the AMIs into AWS using vmimport.
2. The terraform config which is causing the issue

resource "aws_instance" "test" {
  count="${var.test_count}"
  ami = "${var.test_ami}"
  instance_type = "${var.test_instance_type}"
  subnet_id = "${var.test_subnet_id}"

  tenancy = "dedicated"
  disable_api_termination= false

  ebs_block_device {
   device_name = "/dev/sda1"
   delete_on_termination = true
   volume_type = "gp2"
 }
}

Terraform output everytime I run terraform plan

ebs_block_device.2613854568.delete_on_termination: "" => "1" (forces new resource)
Plan: 1 to add, 0 to change, 1 to destroy.

I get the same error with the below ebs_block_device config as well. Based on the suggested fix in #1260.

ebs_block_device {
   device_name = "/dev/sda1"
   delete_on_termination = true
   volume_type = "gp2"
   volume_size = 100
 }

Any help will be appreciated.

Thanks

[hashibot]

_This comment was originally opened by @b-dean as https://github.com/hashicorp/terraform/issues/3654#issuecomment-246437192. It was migrated here as part of the provider split. The original comment is below._

---

This is still a problem in 0.7.3

If you put the snapshot id from the ami on the ebs_block_device for the volume that has the same device name as the one in the ami, then it doesn't think it needs to change.

It's like terraform interprets us not specifying snapshot_id as if the block device does not have a snapshot applied, even though there is one in the ami.

[hashibot]

_This comment was originally opened by @b-dean as https://github.com/hashicorp/terraform/issues/3654#issuecomment-276109694. It was migrated here as part of the provider split. The original comment is below._

---

@mitchellh, sorry to mention you specifically on this bug but it's been sitting here for many versions (originally in 0.6.5 and still a bug in 0.8.5, just tried it).

I was wondering, are there plans to make it so values not specifically set on ebs_block_device do not cause the aws_instance to need to be destroyed and created? For example: snapshot_id which comes from the AMI. If I have block device mappings specified in my packer build and those non-root volumes end up being part of the AMI, in order to have the instance not get destroyed every time I either have to put the snapshot_id in the ebs_block_device (which makes data.aws_ami kind of useless since I can't easily get the snapshot_id for a particular block device mapping from that data).

Or I have to use the workaround mentioned in #1260, which basically makes the ebs_block_device settings only happen when the resource is created:

lifecycle {
    ignore_changes = ["ebs_block_device"]
}

That workaround is certainly better than nothing, but I'd rather have things like changes to disk types or volume sizes actually trigger the resource to be destroyed/created. I'm just not seeing a great solution here.

[alirodell]

Wanted to resurrect this discussion since the bug is still there as of version 0.10.2. Is the issue that is causing the forced replacement of instances known? Including the snapshot_id does work as a workaround but it adds complexity to the variables needed for builds.

terraform -version
Terraform v0.10.2

Thanks

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!