Terraform-provider-aws: EC2 instance recreation due to unexpected user_data change

[rmldsky]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.3
AWS Provider: 1.20.0

Affected Resource(s)

• aws_instance

Description

Not really sure what happened as my plan (remote) was up to date. I did not change any resource definition nor updated terraform or aws provider. Those resources were written months ago, plus one of those EC2 instances is stopped on AWS.

When I run terraform plan suddenly I see my plan wants to recreate some of my EC2 instances (not sure why those not others). From the plan I see the reason is change/update to user_data. Note I do not use or rely on user_data I ommit it completely in my resources.

user_data: "da39a3ee5e6b4b0d3255bfef95601890afd80709" => "" (forces new resource)

I need to use:

  lifecycle {
    ignore_changes = ["user_data"]
  }

in order to ignore this.

Not sure if AWS did something under the hood? Started returning something extra in the API response? No clue here.

Please advice.

[ctd]

I'm seeing the same thing, using:

Terraform v0.11.7
+ provider.aws v1.24.0

My aws_instance resource omits user_data entirely. I'm seeing the same user_data attribute change on a plan to an existing resource:

 user_data: "da39a3ee5e6b4b0d3255bfef95601890afd80709" => "" (forces new resource)

AWS reports the userData as "":

$ aws ec2 describe-instance-attribute --attribute userData --instance-id 'i-REDACTED'
{
    "InstanceId": "i-REDACTED",
    "UserData": {
        "Value": ""
    }
}

The issue can be worked around by either:

• creating a lifecycle rule to ignore user_data,
• or by explicitly setting user_data = "" in the resource definition.

da39a3ee5e6b4b0d3255bfef95601890afd80709 is equal to the SHA1 sum of "", so I suspect there's a bug in how the resource default user_data value is being handled in the provider.

[guikcd]

Maybe in https://github.com/terraform-providers/terraform-provider-aws/blob/544f540205c6171717d193fe79ca94698aff52d9/aws/resource_aws_instance.go#L117-L131 ?

[bflad]

As a hunch, this seems like it could be an EC2 service update that is rolling out.

Out of curiosity, is this only occurring in certain regions or spread out? Our daily acceptance testing in us-west-2 is not picking up this as an issue currently.

[bflad]

Also very curious to know if it really is only affecting older EC2 instances -- I'm unable to reproduce in us-east-1 with a brand new instance, for example. Reproduction ideas welcome!

If someone wants to try a potential fix in their environment, building the provider from this branch should alleviate the issue: https://github.com/terraform-providers/terraform-provider-aws/compare/b-aws_instance-user_data-empty-diff?expand=1 (if it does work, I'll submit it as a pull request and we can get it released)

[Seraf]

Hello,

On my side I get this error on eu-west-1 region

[tuomov-tieto]

eu-west-1 here as well. Seems to be inconsistent across instances, describe-instance-attribute reports the usual {} for some and { "Value": "" } for other instances.

[ChadScott]

This is hitting all of our instances in us-west-1. We're dead in the water at the moment. We're looking at adding user_data = "" to all instance definitions.

[tacchino]

Have been seeing this for the past week in us-east-1. Have it on old instances but also on instances that were created as recently as 2018-03-12

[ChadScott]

@bflad I have tested your branch in three of our affected environments and it resolves it completely.

[bflad]

Fantastic! I'll make sure #4991 gets in for release tomorrow.

[bflad]

A fix for ignoring the empty string SHA da39a3ee5e6b4b0d3255bfef95601890afd80709 has been merged and will be released with version 1.25.0 of the AWS provider shortly.

[bflad]

This has been released in version 1.25.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[Alberts00]

Hello @bflad ,
I was wondering is there any reason why #4991 is done only one way (hash -> empty string and not empty string -> hash)?

We have instances created before the API started returning the hash and we are using 1.25 AWS provider version and the affected instances are in us-west-2 region.
This leads to such change set which this PR does not resolve.
user_data: "" => "da39a3ee5e6b4b0d3255bfef95601890afd80709" (forces new resource)

[vmartin01]

I'm running into this issue with Provider 1.29

user_data: "3ebfe68e557aa6496668f4ddf212efccfb71b2e1" => "1c6c5ee445a93cae54625e7393685a024d46819d" (forces new resource)

If I run apply on MacOS and then I run apply or plan on RHEL 7.5, I get this problem. But If I run apply again on MacOS I get no problem. The userdata equality check across OS platforms is not portable.

[bflad]

The user_data: "" => "da39a3ee5e6b4b0d3255bfef95601890afd80709" (forces new resource) fix has been merged and will release with version 1.33.0 of the AWS provider, likely middle of next week.

For any other potential cases, please open new issues following the bug report issue template. Thanks.

[dolefirenko]

user_data change should stop instance, update user_data itself and start it again. But even in AWS provider 1.34 it wants to create new resource. BTW both options are dangerous on production.
So user_data should be ignored in lifecycle as well as AMI

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!