Terraform: Bug: creating instances with EIPs based on a count variable, applying, then increasing that count variable causes an error

[scottcunningham]

We have set up a repo at https://github.com/Ensighten/terraform-eip-bug which demonstrates the issue.

When trying to create instances with EIPs based on a count variable, the following error is produced after applying, increasing the count, and applying again:

Terraform version: Terraform v0.6.3

Error applying plan:

1 error(s) occurred:

* aws_eip.test-instance.0: diffs didn't match during apply. This is a bug with Terraform and should be reported.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

The instances were created and EIPs were successfully bound despite the error. On running further plans and applies, no
changes are detected.

Steps to reproduce:

1. Run terraform apply --var subnet=$SOME_SUBNET_ID
2. Modify main.tf to change the "count" variable for the "test" module to be 2

3. Run terraform plan --var subnet=$SOME_SUBNET_ID - observe that EIP instance attributes will be changed (from
   a concrete instance ID to an element function call into aws_instance.test-instance.*.id):

      ~ module.test.aws_eip.test-instance.0
        instance: "<INSTANCE ID>" => "${element(aws_instance.test-instance.*.id, count.index)}"
   

4. Run terraform apply --var subnet=$SOME_SUBNET_ID, see error message given above.

[catsby]

Reproduced with this config:

variable "count" {}

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "nat" {
  count                       = "${var.count}"
  ami                         = "ami-21f78e11"
  instance_type               = "t1.micro"
  associate_public_ip_address = true
  source_dest_check           = false

  tags {
    Name = "testing-vpc-subnet-things"
  }
}

resource "aws_eip" "nat" {
  instance = "${element(aws_instance.nat.*.id, count.index)}"
  count    = "${var.count}"
  vpc      = true
}

plan / apply with count = 2 works, changing that to 3 will result in diffs error

[Gary-Armstrong]

I forgot this happens, and so when someone asked me to add two more nodes, I increased count and then RED.

* aws_eip.wxmix_compute_eip.3: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.14
    Resource ID: aws_eip.wxmix_compute_eip.3
    Mismatch reason: attribute mismatch: instance
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-6e7695f4", New:"${element(aws_instance.compute.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

[Gary-Armstrong]

A second apply didn't go red, and I still have EIPs on the pre-existing instances plus EIPs on the new instances, so not a disaster at all. But I did panic a tiny bit.

[Dchamard]

I am having the same issue with terraform 0.7.5 or any version I tried.
I have 2 run terraform twice to be able to add nodes with eips.

Config:

resource "aws_instance" "cassandra_seed" {
count = "${var.cassandra_instance_seed_count}"
ami = "${lookup(var.ec2_base_amis, var.region)}"
instance_type = "${var.cassandra_instance_type}"
user_data = "${element(data.template_file.cassandra_seed.*.rendered, count.index)}"
key_name = "${var.key_name}"
iam_instance_profile = "${aws_iam_instance_profile.cassandra_profile.name}"
vpc_security_group_ids = ["${concat(split(",", var.base_security_group_ids),list(aws_security_group.cassandra.id))}"]
subnet_id = "${element(split(",", var.public_subnets), count.index)}"
monitoring = "true"
ephemeral_block_device {
device_name = "/dev/sdc"
virtual_name = "ephemeral0"
}
lifecycle { ignore_changes = [ "user_data" ] }
}

resource "aws_eip" "cassandra_seed" {
count = "${var.cassandra_instance_seed_count}"
instance = "${element(aws_instance.cassandra_seed.*.id, count.index)}"
vpc = true
}

error:

3 error(s) occurred:

• aws_eip.cassandra_seed.2: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.7.5
Resource ID: aws_eip.cassandra_seed.2
Mismatch reason: attribute mismatch: instance
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-00259514a68b4fa2a", New:"${element(aws_instance.cassandra_seed.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

• aws_eip.cassandra_seed.1: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.7.5
Resource ID: aws_eip.cassandra_seed.1
Mismatch reason: attribute mismatch: instance
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-05eee0c890c2f8baa", New:"${element(aws_instance.cassandra_seed.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

• aws_eip.cassandra_seed.0: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.7.5
Resource ID: aws_eip.cassandra_seed.0
Mismatch reason: attribute mismatch: instance
Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"instance":*terraform.ResourceAttrDiff{Old:"i-0bd933ef6b5d062c2", New:"${element(aws_instance.cassandra_seed.*.id, count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff(nil), Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

[mitchellh]

Finally have a fix queued up in #9618!!! This will be fixed in the next version of Terraform (after the merge, which hopefully is soon). This was not a fun bug to chase down. :)

[scottcunningham]

That is fantastic news, thanks a lot for investigating @mitchellh!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.