Terraform: When using data template_file or template_cloudinit_config, forces new resource, even when nothing changed

[cdelorme]

Terraform Version

Tested with versions

• 0.9.11
• 0.9.9
• 0.9.5
• 0.9.4
• 0.9.3
• 0.9.2

Terraform Configuration Files

_This is just a summary of the related components:_

data "template_file" "bastion_userdata" {
    depends_on = ["aws_s3_bucket_object.authorized_keys"]
    template = "${file("templates/bastion.sh")}"

    vars {
        region = "${var.region}"
        bucket = "${aws_s3_bucket.bucket.id}"
        keys_version = "${aws_s3_bucket_object.authorized_keys.etag}"
    }
}

resource "aws_instance" "bastion_host" {
    ami = "${var.bastion_ami_id}"
    instance_type = "t2.micro"
    key_name = "${var.key_name}"
    vpc_security_group_ids = ["${aws_security_group.bastion_security_group.id}"]
    iam_instance_profile = "${aws_iam_instance_profile.bastion_profile.name}"
    associate_public_ip_address = true
    subnet_id = "${element(aws_subnet.public_subnets.*.id, 0)}"
    user_data = "${base64encode(data.template_file.bastion_userdata.rendered)}"

    tags {
        Name = "${var.environment}-${var.product_name}-bastion"
        CostCenter = "${var.cost_center}"
        Environment = "${var.environment}"
        Service = "${var.product_name}"
        POC = "${var.poc}"
    }
}

The repository lives in my companies privately hosted repository, otherwise I'd gladly share.

Expected Behavior

Making no changes to any files or variables, I run terraform plan and expect to see something akin to Plan: 0 to add, 0 to change, 0 to destroy.

Actual Behavior

Instead, running terraform plan a second time shows:

user_data:                        "60e9cb237503f410621a0e0876a7f97c1bc0b39a" => "607c1eb4ebdfe4181deea9851ae5a2199e278142" (forces new resource)

Sure enough, if I run terraform apply it will replace the aws_instance. To reiterate, no changes have been made to any files at all.

Steps to Reproduce

1. provision an aws_instance with user_data set via rendered template_file or template_cloudinit_config
2. try to plan or apply a second time

_It always shows user_data has changed, and will replace the host, even if nothing has changed at all._

References

Another issue opened on 0.6.x was closed and describes exactly the same problem I an experiencing:

• 4358 for 0.6.x (Closed)

_There are others I looked at while troubleshooting, but I didn't want to find them again._

Workaround

Currently the solution is to use EOF "here doc" syntax, which directly interpolates the values.

This makes it a bit harder to edit content, and I have not figured out how to combine base64encode with EOF syntax.

[apparentlymart]

Hi @cdelorme! Sorry things didn't work out as expected here.

I believe the problem here is the use of depends_on with that template_file data resource, which means that Terraform must always defer the rendering of the template to apply time. This interacts badly with user_data since it's a "forces new resource" attribute, and so Terraform has to pessimistically assume it needs to change that attribute because it doesn't know yet if the template rendering is going to change.

The solution here is to remove the depends_on from the template data resource. The keys_version = "${aws_s3_bucket_object.authorized_keys.etag}" part already creates a dependency on that specific attribute, which is enough for Terraform to understand that the template mustn't be rendered until after _that specific attribute_ is known -- which it will be, during plan -- whereas the depends_on makes the broader statement "don't render this template until the object has been applied".

I understand that this is not very intuitive behavior right now. The way depends_on works with data sources is tricky because of how data sources can be read either during plan _or_ apply, depending on how their dependencies are declared. In general we recommend against using depends_on with data resources unless it cannot be avoided.

[cdelorme]

@apparentlymart thank you for the swift reply.

When I execute without the depends_on the etag uses the previous value, ignoring changes to the authorized_keys file, as though the etag is computed after the template_file. As a result it does not see the changes to the file until apply is executed a second time.

Is there an alternative way of saying "If this file changes, then replace the aws_instance"?

Update: I just tried redundantly computing the md5 of the file instead of relying on the etag property and that appears to do the trick. Thank you for pointing me in the right direction.

I am happy to close this ticket with the answer provided.

[apparentlymart]

Ahh, I think you've hit an edge case here that we know about but Terraform doesn't currently handle well: attributes that get changed as a side-effect of an update but that Terraform can't predict the value of.

To be specific, without depends_on Terraform thinks it is able to render the template during the plan phase because it has all the data it needs. It doesn't know that updating the S3 object's data is going to, as a side-effect, update the etag. Thus the old value gets "baked in" to the diff, and gets applied as it was during plan even though the etag is subsequently updated during apply. (Here Terraform is trying to remain true to its promise that it will do what the plan said.)

Unfortunately even if Terraform _did_ handle this situation (which is one of the things #14887 is about), it would end up returning to your original problem, since Terraform would need to wait until apply time to determine the etag and thus cause there to in turn be an unknown user_data update, causing the "forces new resource" problem you originally mentioned.

So with all of this said, I think computing the hash locally is the right answer here, since it means Terraform can know the result much earlier and therefore there aren't cascading dependencies on things that can only be determined during apply. Always worth keeping in mind that Terraform's goal is always to define a set of steps to be done and then perform exactly those steps, which unfortunately sometimes requires giving Terraform some additional information that would otherwise have been dynamically computed along the way, so it can get a better picture of what the necessary steps are.

Thanks for following up here, and sorry this wasn't intuitive. It's a common problem that, due to some unfortunate layering, the <computed> user_data value gets hashed for display, obscuring what's going on here. There's another issue for that open somewhere, since it trips people up _a lot_, but I wasn't able to quickly find it to link it here.

I'm glad you found a path forward! I'm going to close this, since although there are some improvements to make here I believe they are all captured by existing issues. Thanks again!

[vikas027]

Thanks for the explanation @apparentlymart . This was hard to figure out and very unintuitive. I had encountered this on 0.10.2. So, I guess the golden rule is to not use depends_on on template_file data resource.

[toddbluhm]

Just spent the last hour or two on this issue of depends_on always forcing whatever is using the template_file data source to update itself each time, before I found this great explanation. Would be great if this got documented somewhere officially either on template_file or on the depends_on docs 😄 Thanks for the great explanation!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.