Terraform-provider-aws: Unexpected diffs on aws_autoscaling_group since v2.63.0

Created on 22 May 2020 · 18Comments · Source: hashicorp/terraform-provider-aws

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.25
+ provider.aws v2.63.0

Affected Resource(s)

aws_autoscaling_group

Terraform Configuration Files

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

data "aws_vpc" "default" {
  default = true
}

data "aws_subnet_ids" "subs" {
  vpc_id = data.aws_vpc.default.id
}

resource "aws_launch_configuration" "as_conf" {
  name          = "web_config"
  image_id      = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
}

resource "aws_autoscaling_group" "bar" {
  name                 = "foobar3-terraform-test"
  max_size             = 0
  min_size             = 0
  desired_capacity     = 0
  launch_configuration = aws_launch_configuration.as_conf.name
  vpc_zone_identifier  = data.aws_subnet_ids.subs.ids

  tags = [
    {
      key                 = "foo"
      value               = "bar"
      propagate_at_launch = true
    },
    {
      key                 = "foo2"
      value               = "bar2"
      propagate_at_launch = true
    },
    {
      key                 = "2foo"
      value               = "2bar"
      propagate_at_launch = true
    },
  ]
}

Debug Output

Panic Output

Expected Behavior

No diff every run when the configuration has not changed.

Actual Behavior

Every plan produces the same diff:

  # aws_autoscaling_group.bar will be updated in-place
  ~ resource "aws_autoscaling_group" "bar" {
        arn                       = "arn:aws:autoscaling:us-east-1:111111111111:autoScalingGroup:5603abf2-a424-407b-b417-95bc90eb8c9f:autoScalingGroupName/foobar3-terraform-test"
        availability_zones        = [
            "us-east-1a",
            "us-east-1b",
            "us-east-1c",
            "us-east-1d",
            "us-east-1e",
            "us-east-1f",
        ]
        default_cooldown          = 300
        desired_capacity          = 0
        enabled_metrics           = []
        force_delete              = false
        health_check_grace_period = 300
        health_check_type         = "EC2"
        id                        = "foobar3-terraform-test"
        launch_configuration      = "web_config"
        load_balancers            = []
        max_instance_lifetime     = 0
        max_size                  = 0
        metrics_granularity       = "1Minute"
        min_size                  = 0
        name                      = "foobar3-terraform-test"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::111111111111:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = []
      ~ tags                      = [
          - {
              - "key"                 = "2foo"
              - "propagate_at_launch" = "true"
              - "value"               = "2bar"
            },
            {
                "key"                 = "foo"
                "propagate_at_launch" = "true"
                "value"               = "bar"
            },
            {
                "key"                 = "foo2"
                "propagate_at_launch" = "true"
                "value"               = "bar2"
            },
          + {
              + "key"                 = "2foo"
              + "propagate_at_launch" = "true"
              + "value"               = "2bar"
            },
        ]
        target_group_arns         = []
        termination_policies      = []
        vpc_zone_identifier       = [
            "subnet-a",
            "subnet-b",
            "subnet-c",
            "subnet-d",
            "subnet-e",
            "subnet-f",
        ]
        wait_for_capacity_timeout = "10m"
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Downgrading to 2.62.0 produces a diff on the first apply so this new order is being stored in the state file.

Steps to Reproduce

terraform apply

Important Factoids

Regression introduced in 2.63.0.

Does not happen for every state if you happen to have the tags in the "correct" order.

References

Probably introduced by this PR: https://github.com/terraform-providers/terraform-provider-aws/pull/13360

bug regression servicautoscaling

Source

dpiddockcmp

👍31 🚀6

Most helpful comment

The fix for the tags ordering issue has been merged and will release with version 2.64.0 of the Terraform AWS Provider, likely tomorrow.

bflad on 27 May 2020

❤3

All 18 comments

We are seeing this behaviour on ALL of our various ECS and EKS deployments as of this morning.
https://github.com/terraform-aws-modules/terraform-aws-eks/pull/889#issuecomment-632738786

dayglojesus on 22 May 2020

I should add, this behaviour affects Terraform 0.11.14 as well.

dayglojesus on 22 May 2020

Same issue in Terraform 0.11.14

esteban1983cl on 22 May 2020

See also:

https://discuss.hashicorp.com/t/tags-which-values-havent-changed-are-getting-updated/9100

ewbankkit on 23 May 2020

The Auto Scaling group's tags _are_ created in the order specified in the HCL code:

2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws: 2020/05/23 17:08:02 [DEBUG] AutoScaling Group create configuration: {
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   AutoScalingGroupName: "foobar3-terraform-test",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   HealthCheckGracePeriod: 300,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   LaunchConfigurationName: "web_config",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   MaxSize: 0,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   MinSize: 0,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   NewInstancesProtectedFromScaleIn: false,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   Tags: [{
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Key: "foo",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       PropagateAtLaunch: true,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceId: "foobar3-terraform-test",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceType: "auto-scaling-group",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Value: "bar"
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:     },{
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Key: "foo2",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       PropagateAtLaunch: true,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceId: "foobar3-terraform-test",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceType: "auto-scaling-group",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Value: "bar2"
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:     },{
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Key: "2foo",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       PropagateAtLaunch: true,
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceId: "foobar3-terraform-test",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       ResourceType: "auto-scaling-group",
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:       Value: "2bar"
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:     }],
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws:   VPCZoneIdentifier: ",subnet-0f9d4a44"
2020-05-23T17:08:02.096-0400 [DEBUG] plugin.terraform-provider-aws: }

but when they are read back, they are in a different order:

2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws: 2020/05/23 17:08:03 [DEBUG] [aws-sdk-go] <DescribeAutoScalingGroupsResponse xmlns="http://autoscaling.amazonaws.com/doc/2011-01-01/">
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:   <DescribeAutoScalingGroupsResult>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:     <AutoScalingGroups>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:       <member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <HealthCheckType>EC2</HealthCheckType>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <Instances/>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <TerminationPolicies>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           <member>Default</member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         </TerminationPolicies>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <DefaultCooldown>300</DefaultCooldown>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <EnabledMetrics/>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <AutoScalingGroupARN>arn:aws:autoscaling:us-west-2:123456789012:autoScalingGroup:bb771324-fd24-4beb-9a61-3ca596ea3e64:autoScalingGroupName/foobar3-terraform-test</AutoScalingGroupARN>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <AvailabilityZones>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           <member>us-west-2b</member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         </AvailabilityZones>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <MaxSize>0</MaxSize>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <TargetGroupARNs/>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <LaunchConfigurationName>web_config</LaunchConfigurationName>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <AutoScalingGroupName>foobar3-terraform-test</AutoScalingGroupName>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <HealthCheckGracePeriod>300</HealthCheckGracePeriod>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <NewInstancesProtectedFromScaleIn>false</NewInstancesProtectedFromScaleIn>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <CreatedTime>2020-05-23T21:08:04.361Z</CreatedTime>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <MinSize>0</MinSize>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <LoadBalancerNames/>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <Tags>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           <member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceId>foobar3-terraform-test</ResourceId>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <PropagateAtLaunch>true</PropagateAtLaunch>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Value>2bar</Value>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Key>2foo</Key>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceType>auto-scaling-group</ResourceType>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           </member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           <member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceId>foobar3-terraform-test</ResourceId>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <PropagateAtLaunch>true</PropagateAtLaunch>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Value>bar</Value>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Key>foo</Key>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceType>auto-scaling-group</ResourceType>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           </member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           <member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceId>foobar3-terraform-test</ResourceId>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <PropagateAtLaunch>true</PropagateAtLaunch>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Value>bar2</Value>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <Key>foo2</Key>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:             <ResourceType>auto-scaling-group</ResourceType>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:           </member>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         </Tags>
2020-05-23T17:08:03.135-0400 [DEBUG] plugin.terraform-provider-aws:         <ServiceLinkedRoleARN>arn:aws:iam::123456789012:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling</ServiceLinkedRoleARN>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:         <SuspendedProcesses/>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:         <DesiredCapacity>0</DesiredCapacity>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:         <VPCZoneIdentifier>subnet-0f9d4a44</VPCZoneIdentifier>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:       </member>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:     </AutoScalingGroups>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:   </DescribeAutoScalingGroupsResult>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:   <ResponseMetadata>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:     <RequestId>c1cc9d77-4e9d-4108-8a54-6cc45c9f492b</RequestId>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws:   </ResponseMetadata>
2020-05-23T17:08:03.136-0400 [DEBUG] plugin.terraform-provider-aws: </DescribeAutoScalingGroupsResponse>

The fix in #13360 means that these tags are now actually persisted in state.
Previously, e.g. in v2.62.0, there was an error on the d.Set("tags", ...) call

tags.0.propagate_at_launch: '' expected type 'string', got unconvertible type 'bool'

which prevented the tags from being written to state in the order returned in DescribeAutoScalingGroupsResponse.

Now that has been fixed, the tags ARE stored in state, hence the diff.

ewbankkit on 23 May 2020

Terraform v0.12.25
+ provider.aws v2.63.0

I could identify for aws_autoscaling_group the tags = should be sorted in alphabetical order using HCL to avoid the tags recreation.

dyegoe on 26 May 2020

The fix for the tags ordering issue has been merged and will release with version 2.64.0 of the Terraform AWS Provider, likely tomorrow.

bflad on 27 May 2020

❤3

This has been released in version 2.64.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

hashibot[bot] on 29 May 2020

The bug seems still present on v2.64.0

dud225 on 29 May 2020

👍1

Did you upgrade your AWS plugin? terraform init -upgrade

After re-order the tags, I ran terraforn plan using 2.64.0 and got no changes.

terraform {
  required_version = ">= 0.12.24"
  required_providers {
    aws      = ">= 2.64.0"
    template = ">= 2.1.2"
  }
}

But, if I force to use 2.63.0, I still got changes during the plan.

terraform {
  required_version = ">= 0.12.24"
  required_providers {
    aws      = "= 2.63.0"
    template = ">= 2.1.2"
  }
}

dyegoe on 29 May 2020

Hello
Yes I did upgrade.
AFAIU off https://github.com/terraform-providers/terraform-provider-aws/pull/13515 the order shouldn't matter so no I haven't reordered the tags.

dud225 on 1 Jun 2020

Sorry, I wrote wrong.
I tried to say that I changed the order of the tags. I didn't sort it in alphabetical order.
Maybe you can post here more details, but so far, it worked for me

dyegoe on 1 Jun 2020

We get the same unexpected diff again on 2.64.0 and Terraform 0.11.14, the bug seem to be present to me.

FarshadHadei on 3 Jun 2020

Yes I'm running TF 0.11.14 as well.

dud225 on 3 Jun 2020

when I use tag blocks in the aws_autoscaling_group resource, I get no diffs
If I use null data source, it gets me diff in every plan:

data "null_data_source" "tags" {
   count = "${length(keys(var.common_tags))}"
    
   inputs = {
    key                 = "${element(keys(var.common_tags), count.index)}"
    value               = "${element(values(var.common_tags), count.index)}"
    propagate_at_launch = true
    }
 }

resource "aws_autoscaling_group" "ecs_autoscaling_group" {
    tags = ["${data.null_data_source.tags.*.outputs}"]
}

FarshadHadei on 3 Jun 2020

👍1

Ah interesting, I'm also relying on null_data_source though the terraform-null-label module.

dud225 on 4 Jun 2020

@bflad any input on this issue will be appreciated ^^

FarshadHadei on 5 Jun 2020

👀1

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] on 27 Jun 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Terraform support for Alexa Smart Home Lambda trigger

joelittlejohn · 3Comments

Allow aws_acm_certificate to query the most recent certificate

hashibot · 3Comments

Provide releases more often

dvishniakov · 3Comments

On plan, redrive_policy is always marked as modified

hashibot · 3Comments

AWS ECS services are force updated on each apply

EmmN · 3Comments