Terraform-provider-aws: aws_autoscaling_group removes empty tag "value" attributes

[markchalloner]

When using tags with an empty value in the latest provider:

variable "tags" {
  default = [
    {
      key                 = "Foo"
      value               = ""
      propagate_at_launch = true
    },
    {
      key                 = "Baz"
      value               = ""
      propagate_at_launch = true
    },
  ]
}

output "foo" {
  value = var.tags[*]
}

resource "aws_autoscaling_group" "bar" {
  name                 = "foobar3-terraform-test"
  max_size             = 5
  min_size             = 2
  launch_configuration = "lt-xxxxxx"
  vpc_zone_identifier  = ["subnet-xxxxxx"]
  tags = var.tags[*]
}

the tag -> value attributes are removed from the plan and apply errors:

$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_autoscaling_group.bar will be created
  + resource "aws_autoscaling_group" "bar" {
      + arn                       = (known after apply)
      + availability_zones        = (known after apply)
      + default_cooldown          = (known after apply)
      + desired_capacity          = (known after apply)
      + force_delete              = false
      + health_check_grace_period = 300
      + health_check_type         = (known after apply)
      + id                        = (known after apply)
      + launch_configuration      = "lt-xxxxxx"
      + load_balancers            = (known after apply)
      + max_size                  = 5
      + metrics_granularity       = "1Minute"
      + min_size                  = 2
      + name                      = "foobar3-terraform-test"
      + protect_from_scale_in     = false
      + service_linked_role_arn   = (known after apply)
      + tags                      = [
          + {
              + "key"                 = "Foo"
              + "propagate_at_launch" = "true"
            },
          + {
              + "key"                 = "Baz"
              + "propagate_at_launch" = "true"
            },
        ]
      + target_group_arns         = (known after apply)
      + vpc_zone_identifier       = [
          + "subnet-xxxxxx",
        ]
      + wait_for_capacity_timeout = "10m"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_autoscaling_group.bar: Creating...

Error: foobar3-terraform-test: invalid tag attributes: value missing

  on main.tf line 21, in resource "aws_autoscaling_group" "bar":
  21: resource "aws_autoscaling_group" "bar" {

values are kept by the output as can be seen in the state file implying this is an issue with the provider not TF core:

{
  "version": 4,
  "terraform_version": "0.12.1",
  "serial": 6,
  "lineage": "2ffdc8d7-88e8-c8be-3b9a-1ea96074bd15",
  "outputs": {
    "foo": {
      "value": [
        {
          "key": "Foo",
          "propagate_at_launch": true,
          "value": ""
        },
        {
          "key": "Baz",
          "propagate_at_launch": true,
          "value": ""
        }
      ],
      "type": [
        "tuple",
        [
          [
            "object",
            {
              "key": "string",
              "propagate_at_launch": "bool",
              "value": "string"
            }
          ],
          [
            "object",
            {
              "key": "string",
              "propagate_at_launch": "bool",
              "value": "string"
            }
          ]
        ]
      ]
    }
  },
  "resources": []
}

[markchalloner]

This is version 2.15.

[thaiphv]

Hey guys, do you think this is a bug or not? If yes, how are you going to tackle it?
Thanks.

[jkim-codecademy]

Having the same issue on provider version 2.28.1 and on 2.29.0. Strangely enough this behavior is not consistent, I have multiple sets of ASGs but some of them will pick up on blank tag just fine

[ahaynssen]

Another case, ASGs deployed with empty tags prior to 0.12 pass the plan and apply on 2.29.0.

[seh]

We've found that this fails when _creating_ an ASG, but only after the ASG is actually created. That is, _terraform apply_ fails, but the ASG does exist afterward, with the empty tag value.

Once it exists, you can change the value of the tag—from empty to nonempty, and back again—without any complaint. It's only when first creating the ASG that this seems to fail.

[baxor]

Issue seems to only exist on object creation -- updates are fine.

That is, if you create with an empty tag, it will error and bail, even though the object is created. Amend the TF and run again, it will update the objects target tag from '' -> 'value' and proceed with the rest of the catalog. Adjust the value back to an empty string (or remove entirely) and it will happily update the object to an empty string.

[seh]

We had to work around this problem by replacing our intentionally empty tag values with a spurious value. We chose "ignored." However, we happen to be fortunate that the readers of those tags don't mind the strings being nonempty. There could be cases where the empty value is the exact value that the tag must bear.

[seh]

A few hours later, I realized that we _did_ have some IAM conditions that were sensitive to these empty tag values. Now we have to revise those conditions if we're going to use nonempty dummy values for our ASG tags to work around this bug.

[masseyc]

we've also encountered this bug on creating ASGs, and reapplying did not fix it, re had to remove the tag and apply wthout it before reapplying with it back in and still empty.

[crawforde]

I'm experiencing this problem even for creation of resources with non-empty tags. Terraform diff shows the following, which is correct:

+ tags  = [
          + {
              + "key" = "Name"
              + "propagate_at_launch" = "true"
              + "value" = "clustertest-02-generalCompute-node"
            },
          + {
              + "key" = "cluster-autoscaler.kubernetes.io/scale-down-disabled"
              + "propagate_at_launch" = "true"
              + "value" = "false"
            },
          + {
              + "key" = "k8s.io/cluster-autoscaler/clustertest-02"
              + "propagate_at_launch" = "true"
            },
          + {
              + "key" = "k8s.io/cluster-autoscaler/enabled"
              + "propagate_at_launch" = "true"
              + "value" = "true"
            },
          + {
              + "key" = "kubernetes.io/cluster/clustertest-02"
              + "propagate_at_launch" = "true"
              + "value" = "owned"
            },
          + {
              + "key" = "managed_by"
              + "propagate_at_launch" = "true"
              + "value" = "terraform"
            },
          + {
              + "key" = "scale-down-disabled-orig"
              + "propagate_at_launch" = "true"
              + "value" = "false"
            },
        ]

When I try to apply, I still get the error:

invalid tag attributes: value missing

I have Terraform v0.12.21 and AWS Provider v2.59.0.

[seh]

It's the tag with key "k8s.io/cluster-autoscaler/clustertest-02" that's causing the problem.

[crawforde]

Oh snap haha. Thank you. Looks like the shelter in place is getting to me 🙃. I'm generating these from a loop, so I missed that. Seems like the value that's getting passed in the loop resolves to "". Same problem that everyone else is having.

[headcr4sh]

I encounter the same problem.

This is especially annoying, since I am using EKS in conjunction with the Kubernetes cluster-autoscaler and want to make sure that my "compute" nodes are recognized as such. (according to docs / best practices, that means that my Kubernetes nodes should be labeled like that:

# This is the kubectl command line that can be used to mark a node with name ${node_name}
# to have the role ${node_role}. The value of this label is usually just an empty string.
kubectl label nodes/${node_name} node-role.kubernetes.io/${node_role}=

To make sure that the cluster-autoscaler knows that new nodes spawned from this autoscaling group will have the key/value combination, it is necessary to apply the correct AWS tags, though:

  {
      key                 = "k8s.io/cluster-autoscaler/node-template/label/node-role.kubernetes.io/${node_role}"
      value               = ""
      propagate_at_launch = true
    }

I can confirm that it is possible to change existing tag values to the empty string, but new ASGs cannot be created.

[odg0318]

Workaround

Use tag instead of tags. Even though empty value is removed in terraform plan, it works fine with empty tag value.

locals {
  tags = [
    {
      key                 = "foo"
      value               = ""
      propagate_at_launch = true
    },
    {
      key                 = "bar"
      value               = ""
      propagate_at_launch = true
    }
  ]
}

resource "aws_autoscaling_group" "this" {
  ...
  ...

  dynamic "tag" {
    for_each = local.tags
    content {
      key                 = tag.value["key"]
      value               = tag.value["value"]
      propagate_at_launch = tag.value["propagate_at_launch"]
    }
  }