Terraform: aws: user_data diff on all instances, forcing new resource

[plainlystated]

Terraform v0.11.7
+ provider.aws v1.13.0

Expected Behavior

When running a terraform plan I should see no updates. I last applied terraform about 6 weeks ago, with the same version of terraform and the same terraform templates.

Actual Behavior

Every aws_instance I have says "new resource required" due to a user_data change. I don't have any user_data in my templates (and never have) so I'm guessing this is something terraform used to use but not any more. I don't use anything else to manage my EC2 resources, and I'm the only person on my team that touches this stuff.

Example output with IDs redacted:

-/+ module.workbench_shared.aws_instance.firebird_converter (new resource required)
      id:                                "i-123456" => <computed> (forces new resource)
      ami:                               "ami-8b9a63e0" => "ami-8b9a63e0"
      associate_public_ip_address:       "true" => <computed>
      availability_zone:                 "us-east-1c" => "us-east-1c"
      ebs_block_device.#:                "1" => <computed>
      ephemeral_block_device.#:          "0" => <computed>
      get_password_data:                 "false" => "false"
      instance_state:                    "running" => <computed>
      instance_type:                     "t2.micro" => "t2.micro"
      ipv6_address_count:                "" => <computed>
      ipv6_addresses.#:                  "0" => <computed>
      key_name:                          "patrick" => "patrick"
      network_interface.#:               "0" => <computed>
      network_interface_id:              "eni-12345" => <computed>
      password_data:                     "" => <computed>
      placement_group:                   "" => <computed>
      primary_network_interface_id:      "eni-12345" => <computed>
      private_dns:                       "ip-10-100-1-2.ec2.internal" => <computed>
      private_ip:                        "10.100.1.2" => <computed>
      public_dns:                        "" => <computed>
      public_ip:                         "1.2.3.4" => <computed>
      root_block_device.#:               "1" => <computed>
      security_groups.#:                 "0" => <computed>
      source_dest_check:                 "true" => "true"
      subnet_id:                         "subnet-12345" => "subnet-12345"
      tags.%:                            "1" => "1"
      tags.Name:                         "firebird_converter.workbench.lcp" => "firebird_converter.workbench.lcp"
      tenancy:                           "default" => <computed>
      user_data:                         "da39a3ee5e6b4b0d3255bfef95601890afd80709" => "" (forces new resource)
      volume_tags.%:                     "0" => <computed>
      vpc_security_group_ids.#:          "1" => "1"
      vpc_security_group_ids.12345: "sg-12345"=> "sg-12345"

All my ec2 instances have the same user_data. It is not distinct per instance. I tried to base64 decode it and it was not meaningful to me. I also tried to set user_data to the string, but that still resulted in a diff. I also tried to set user_data_base64 to this string, but that resulted in 2 diffs (one for user_data and one for user_data_base64).

[mildwonkey]

@plainlystated -

I am sorry you are experiencing this odd behavior. Terraform should not apply user_data to an instance without it being specified in the configuration files, which leads me to suspect that indeed someone or some process applied the user_data after the instance was created, outside of terraform.

Unfortunately, it is going to be difficult to troubleshoot this without the output from your original terraform apply.

One thing you might try is inspecting the user_data on your instance to see if you can get any hints as to who attached it, or when it was applied(https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-attribute.html). You could also check cloud trail to see when the user_data was applied to the instance, if you have it enabled in your AWS account, though I'm not positive that api call is logged (reference: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/using-cloudtrail.html).

As a workaround, you can add a lifecycle block to the aws_instance to ignore changes to the user_data:

  lifecycle {
    ignore_changes = ["user_data"]
  } 

While this does not answer the (important) question of how user_data was attached to the instance it will let you run terraform without replacing the instance due to the user_data change.

[BenjyOC]

Same issue with Terraform 0.9 :

-/+ aws_instance.cache-02
    ami:                               "ami-xxxxx" => "ami-xxxxx"
    associate_public_ip_address:       "true" => "<computed>"
    availability_zone:                 "eu-west-1b" => "<computed>"
    ebs_block_device.#:                "0" => "<computed>"
    ebs_optimized:                     "true" => "true"
    ephemeral_block_device.#:          "0" => "<computed>"
    instance_state:                    "running" => "<computed>"
    instance_type:                     "r4.large" => "r4.large"
    ipv6_address_count:                "0" => "<computed>"
    ipv6_addresses.#:                  "0" => "<computed>"
    key_name:                          "" => "<computed>"
    network_interface.#:               "0" => "<computed>"
    network_interface_id:              "eni-6459c41e" => "<computed>"
    placement_group:                   "" => "<computed>"
    primary_network_interface_id:      "eni-6459c41e" => "<computed>"
    private_dns:                       "ip-x-x-x-x.eu-west-1.compute.internal" => "<computed>"
    private_ip:                        "x.x.x.x" => "x.x.x.x"
    public_dns:                        "ec2-x-x-x-x.eu-west-1.compute.amazonaws.com" => "<computed>"
    public_ip:                         "x.x.x.x" => "<computed>"
    root_block_device.#:               "1" => "<computed>"
    security_groups.#:                 "0" => "<computed>"
    source_dest_check:                 "true" => "true"
    subnet_id:                         "subnet-2098e344" => "subnet-2098e344"
    tags.%:                            "2" => "2"
    tags.Name:                         "server.aws-eu-west-1-project" => "server.aws-eu-west-1-project"
    tags.Project:                      "project" => "project"
    tenancy:                           "default" => "<computed>"
    user_data:                         "da39a3ee5e6b4b0d3255bfef95601890afd80709" => "" (forces new resource)
    volume_tags.%:                     "0" => "<computed>"
    vpc_security_group_ids.#:          "1" => "1"
    vpc_security_group_ids.2097381630: "sg-f513ea93" => "sg-f513ea93"

[guikcd]

I think the right issue to track is here: https://github.com/terraform-providers/terraform-provider-aws/issues/4954

[plainlystated]

Very interesting that your user_data is the same as mine.

[mildwonkey]

Thanks @guikcd!

I will close this issue, since this bug is reported in terraform-provider-aws#4954. Please continue the conversation there.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.