Terraform: TF 0.14.0 ignores "ignore_changes" block

[KyMidd]

Summary

Terraform 0.14.0 ignores lifecycle block ignore_changes in modules. So far I have only tested aws_instance within a module to see this behavior, but expect it's ignored in other resources. You can help improve this bug report by testing other resource builds in modules and verifying if ignore_changes is respected.

Terraform Version

Terraform v0.14.0
+ provider registry.terraform.io/hashicorp/aws v3.19.0

Terraform Configuration Files

lifecycle {
    ignore_changes = [
      user_data,
      tags["SSM Status"],
      tags["OS Type"],
      tags["OS Name"],
      tags["OS Version"],
    ]
  }

I tested different syntaxes, like wrapping these attributes in different styles of quotes, or using ignore_changes = all without any change in Terraform behavior.

Expected Behavior

Expected Terraform to respect ignore_changes, as 0.13.X Terraform does.

Actual Behavior

Terraform doesn't respect ignore_changes. This leads to unexpected impacts when upgrading to 0.14.X.

      ~ tags                         = {
          ~ "OS Name"           = "Microsoft Windows Server 2016 Datacenter" -> ""
          ~ "OS Type"           = "Windows" -> ""
          ~ "OS Version"        = "10.0.14393" -> ""
          ~ "SSM Status"        = "Managed" -> ""
            # (12 unchanged elements hidden)
        }
      - user_data                    = "ae1a2d404aa5d578a1dde597a17b89cd4dadb283" -> null # forces replacement

Steps to Reproduce

1. Create ec2 resource in AWS using terraform (of any version) with a lifecycle ignore_change block.
2. Modify the item manually or using another tool.
3. Run a terraform plan using TF 0.14.0 version and note that Terraform wants to revert the attribute it should be ignoring

[dzirg44]

aws_autoscaling_group resource is also affected.

[mendhak]

Somewhat related - I'm seeing errors, instead of being ignored, with aws_route53_record and aws_service_discovery_public_dns_namespace in #27106

[favoretti]

Everything is affected. It also kicks in on plan stage, breaking some specific stuff.

[KyMidd]

Thanks Favoretti, I updated title to better reflect that all resources are affected. Since this is terraform core, I also believe this would affect all providers, but if anyone can test other providers that helps further inform this bug report.

[adrian291]

I can confirm it also affects Digital Ocean.

[jbardin]

Thanks for filing the issue.

I'd like to narrow down the scope here, to avoid a pileup of similar looking issues being attributed to the same cause.
From a preliminary investigation there appears to be 2 independent problems here:

• The aws provider is changing the value of user_data during plan. This isn't valid, but because of limitations in the legacy SDK we can only warn about the problem in the logs. After terraform ignores the change in the configuration, the provider still returns a new value. This will appear with a message along the lines of

[WARN]  Provider "registry.terraform.io/hashicorp/aws" produced an invalid plan for aws_instance.web, but we are tolerating it because it is using the legacy plugin SDK.
      - .user_data: planned value cty.StringVal("ae1a2d404aa5d578a1dde597a17b89cd4dadb283") does not match config value cty.StringVal("foo") nor prior value cty.StringVal("foo")

Since user_data forces replacement, it will show all tags being replaced as well in the rendered diff.

• While it is not causing the above diff, there does appear to be a bug with the use of individual map keys in ignore_changes which can cause some changes to be skipped.

[favoretti]

Dunno if I'm adding on top of the pile of unrelatedness - in azurerm provider, resource azurerm_eventhub_cluster in a module, has following section:

  lifecycle {
    ignore_changes = [
      sku_name
    ]
  }

nevertheless, during plan stage it triggers validation on that attribute and fails (the failure is core unrelated though). What I'm trying to understand is why provider validation is triggered during plan (I assume it's cause you pulled refresh forward in the chain) and why it doesn't ignore that attribute.

[nexxai]

Adding onto the chain here that ignore_changes is not ignoring any changes.

For example:

  lifecycle {
    ignore_changes = [
      tags["updated"],
      tags["deployed_by"],
      connection_string
    ]
  }

produces this output (using the azurerm provider, if it matters):

      ~ tags                           = {
          - "deployed_by"   = "Example person"
          - "updated"       = "2020-12-01T01:28:35Z"
        } -> (known after apply)
        # (12 unchanged attributes hidden)


      - connection_string {
          - name  = "Core" -> null
          - type  = "Custom" -> null
          - value = (sensitive value)
        }
      + connection_string {
          + name  = "Core"
          + type  = "Custom"
          + value = (sensitive value)
        }

[KyMidd]

@jbardin , tags are not an attribute which changes during a rebuild according to terraform. Look closely at this output, and you can see that only some of the tags are changing. The rest of the tags are not changing, and are therefore hidden by TF 0.14 default behavior.

      ~ tags                         = {
          ~ "OS Name"           = "Microsoft Windows Server 2016 Datacenter" -> ""
          ~ "OS Type"           = "Windows" -> ""
          ~ "OS Version"        = "10.0.14393" -> ""
          ~ "SSM Status"        = "Managed" -> ""
            # (12 unchanged elements hidden)   <-- This line shows all other tags are unchanged, even though resource plan will rebuild this resource. 
        }
      - user_data                    = "ae1a2d404aa5d578a1dde597a17b89cd4dadb283" -> null # forces replacement

This indicates that all ignore_changes items are ignored in TF 0.14.0. And since others have confirmed other providers are affected, this appears to be a core terraform bug which affects all resources on all providers.

[watsonjm]

This is affecting aws_instance resources that are not part of a module as well.

[RemcodM]

We're affected by this bug as well. We're seeing it across all of our resources managed by the AWS provider, which include:

• aws_instance
• aws_eks_node_group -> Terraform now wants to reset the desired_size of a node group, which is really breaking our workflow currently.

None of our resources are part of a module. Because of this bug, we have to remain using Terraform 0.13.5 for now.

[lscheidler]

We're also affected by this bug. Here is a simple example to reproduce this problem with an aws_security_group (aws provider):

provider "aws" {
  region = "eu-west-1"
}

data "aws_vpc" "default" {
  default = true
}

resource "aws_security_group" "terraform-0-14-test" {
  name = "terraform-0-14-test"
  description = "test terraform 0.14 ignore_changes"

  tags = {
    Name = "terraform-0.14-test"
    Created = timestamp()
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

[darren-johnson]

Hi all,

I've tested this with various azurerm resources and it consistently fails.

I've found that if ignore_changes is added for an argument that accepts null values its doesn't error, but if it is added to an argument that cannot accept null values it passes this as null and the error is produced. A good example here is the license_type in the azurerm_windows_virtual_machine resource.

[KyMidd]

I see this closed, fantastic, thanks @jbardin! @jbardin : Is 0.14.0 rebuilt with these changes or do I need to wait for a subrelease (0.14.0.1) or a new release (0.14.1) to validate these changes are working for my environment?

[denniswebb]

As @KyMidd has asked, is there an ETA for a new release as this is a fairly large breaking issue?

[apparentlymart]

We never change a release we already published, so v0.14.0 is what it is at this point, but indeed this fix was backported to be included in the next v0.14 release, which will be v0.14.1.

[patrickshuff]

This issue broke all of our automated CI/CD testing pipelines. This could have very easily caused significant damage in production. It is very concerning that such fundamental core functionality be so broken and released to the world. More concerning is that this was reported over 48 hours ago and there is yet to be a new release available in all the channels.

[favoretti]

In all honesty, if you auto-upgrade in production - you’re asking for
trouble :)

On Sat, 5 Dec 2020 at 05:09, Patrick Shuff [e-mail] wrote:

This issue broke all of our automated CI/CD testing pipelines. This could
have very easily caused significant damage in production. It is very
concerning that such fundamental core functionality be so broken and
released to the world. More concerning is that this was reported over 48
hours ago and there is yet to be a new release available in all the
channels.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform/issues/27097#issuecomment-739120054,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGERNQ3KRPKSWMLFVXZSXTSTGXBBANCNFSM4ULKNY4A
.

[iwt-pjost]

Seriously? It's not about auto upgrade. We expected the update to be broken as usual. Never had a major TF version jump without major manual intervention. And you know what? That's fine. But this time we upgraded the TF version and ended up with something completely unusable. This update regularly destroys our infrastructure on dev. Yes we were able to roll back to version 13, but to even think of shipping code which hasn't been tested for this behaviour is very unexpected. We expected bad handling of edge cases of various providers. Not complete disregard for terraform core functionalities. Really hope v14.1 will come soon

[pablo-ruth]

@iwt-pjost there was 1 alpha, 2 beta and 1 RC. Terraform is an open source project and you can contribute to its stability by testing those pre-versions and send bug reports. From what you say, you should have easily spotted this bug in your environment.

[danieldreier]

I've tested the fixes @jbardin made in https://github.com/hashicorp/terraform/pull/27141 and https://github.com/hashicorp/terraform/pull/27174 and believe they fix the issues people have reported here. We will be shipping Terraform 0.14.1 tomorrow with these bug fixes. Thank you to everyone to provided detailed reproduction cases for this issue!

[dmattia]

This is now released 😄

https://github.com/hashicorp/terraform/releases/tag/v0.14.1

[juliencoopengo]

great !

[KyMidd]

Confirmed fixed for my use-case. Thanks so much @jbardin and team!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.