Terraform: Instance tags are not applied to new AWS instances in Gov & China

[joelittlejohn]

Terraform Version

0.9.5

Affected Resource(s)

• aws_instance

Terraform Configuration Files

resource "aws_instance" "elasticsearch" {
  count                   = "${var.nodes}"
  ami                     = "${var.ami}"
  disable_api_termination = false
  instance_type           = "${var.instance_type}"
  iam_instance_profile    = "${aws_iam_instance_profile.elasticsearch.name}"
  subnet_id               = "${element(var.subnet_ids, count.index)}"
  vpc_security_group_ids  = ["${var.security_group_id}", "${aws_security_group.elasticsearch.id}"]
  key_name                = "${var.key_pair}"

  root_block_device {
    volume_size           = "8"
    volume_type           = "gp2"
  }

  tags {
    Name = "elasticsearch-${var.name}-${count.index}"
    Application = "elasticsearch-${var.name}"
  }
}

Expected Behavior

Instance should have been created with 2 tags, as shown in the configuration.

Actual Behavior

The instance was created with no tags. terraform plan then showed that these three tags needed to be added, and a second terraform apply added the tags. So in short, I had to run terraform apply twice to end up with an instance that was tagged correctly.

Steps to Reproduce

1. Create a new aws_instance configuraiton that includes some tags
2. terraform apply -target module.mymodule - notice that the tags are not present
3. terraform plan -target module.mymodule - notice that the tag changes are shown
4. terraform apply -target module.mymodule - notice that the tags are now present

References

I suspect this may have something to do with #13173 and/or #14019.

[joelittlejohn]

This is a serious one I think.

Terraform completes its operations and runs successfully but leaves instances untagged. Any discovery mechanism that relies on tags will not work, but of course terraform may have destroyed a working configuration as part of its operations.

Any orchestration tools using terraform will assume that a single call to terraform apply that succeeds will have converged the system towards the new configuration correctly and these may also proceed to destroy older resources even though the new system is not tagged correctly.

I'm happy to provide more info if anyone is having trouble replicating this.

Thanks!

[stack72]

Hi @joelittlejohn

So, I have spent some time looking into this. From what I can see, we now use a different API call when we initially create instances with Terraform and create tags. This is part of the new TagSpecifications when running instances. We can see this is the case as when you apply terraform a second time, it creates the tags. Tag updates are controlled via a different part of the AWS API

Can you give me some idea of what the IAM user policy you are running looks like? I suspect that this is an IAM policy issue but want to confirm this is the case

Thanks

Paul

[joelittlejohn]

Yeah, it's a little surprising but my guess was that this was either a problem in the way the TagSpecification was built, or that this was an actual bug in the AWS Go SDK (at least the version of it that Terraform is using).

I'm running terraform as an admin, that is, I'm using the IAM managed policy 'AdministratorAccess' that gives full access to all services.

[stack72]

So it's not a bug IMO - we have a lot of tests running for this right now that pass each night. I will continue to look into it

Paul

[joelittlejohn]

One other thing I'm going to look into tonight:

So far I've only been seeing this in cn-north-1. Haven't actually tried this in a more typical region like eu-west-1.

[stack72]

@joelittlejohn yes, this will not work in china or gov - we have a note listed in the last changelog about this exact thing :(

[joelittlejohn]

I think this is reasonable for instance volume tagging - doing this during
creation is a new feature and the fact that this is not available in CN as
per the note in the changelog is fine. For instance tags though, I consider
this a serious regression, no?

If this cant be supported in CN and Gov but is still desirable elsewhere,
then I think Terraform must use the old strategy for CN and Gov. Without
using that approach, tagging configuration that worked in the past is now
broken.

On 17 May 2017 6:22 pm, "Paul Stack" [e-mail] wrote:

@joelittlejohn https://github.com/joelittlejohn yes, this will not work
in china or gov - we have a note listed in the last changelog about this
exact thing :(

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform/issues/14535#issuecomment-302162931,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAp5MIu-yI3bTXHXYadOgnYThvzjCzUZks5r6yy_gaJpZM4NcJ8Y
.

[stack72]

Hi @joelittlejohn

Using our Gov cloud account, I found the bug - patch incoming :)

Paul

[joelittlejohn]

@stack72 Wow, fantastic! Thanks for jumping on this so quickly.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.