Terraform: Sensitive values cause resources that did not have any value changed to be updated on terraform plan - version 14 RC

[romeomorcia]

Terraform Version

Terraform v0.14.0-rc1
+ provider registry.terraform.io/hashicorp/azuread v1.0.0
+ provider registry.terraform.io/hashicorp/azurerm v2.34.0

Terraform Configuration Files

resource "azurerm_resource_group" "example" {
  name     = "database-rg"
  location = "West US"
}

resource "azurerm_storage_account" "example" {
  name                     = "examplesa"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_sql_server" "example" {
  name                         = "mssqlserver"
  resource_group_name          = azurerm_resource_group.example.name
  location                     = azurerm_resource_group.example.location
  version                      = var.sqlserver-version
  administrator_login          = var.sqlserver-login
  administrator_login_password = data.azurerm_key_vault_secret.sql_password.value

  tags = {
    environment = "production"
  }
}

Expected Behavior

Nothing should've been shown to be changed after terraform plan

Actual Behavior

  # module.app.azurerm_sql_server.example will be updated in-place
  ~ resource "azurerm_sql_server" "example" {
      # Warning: this attribute value will be marked as sensitive and will
      # not display in UI output after applying this change
      ~ administrator_login_password = (sensitive value)
        id                           = "hidden-id-for-security"
        name                         = "mssqlserver"
        tags                         = {}
        # (7 unchanged attributes hidden)
    }

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

I'm using modules, did not test without modules but it happen with also azurerm_function_app when the app_settings contains a variable that is being set with a value of a resource (and not DATA like in the example above). In this case, because connection_string is a block, it will have this information instead:

        connection_string {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }

[alisdair]

Hi @romeomorcia, thanks for reporting this. Unfortunately I'm unable to reproduce the issue: the second terraform plan shows no changes. I have an attempted reproduction case here, which is using fundamentally the same test case you gave above.

Are you able to modify this reproduction to show the issue you're seeing?

[romeomorcia]

@alisdair could be the case that because you're not using a module, the issue cannot be reproduced?

[alisdair]

I don't see how that could be related. Can you modify the reproduction case to show me what you mean?

[romeomorcia]

@alisdair cannot reproduce the issue with your code + modules. Will try tomorrow to compare them with mine properly and see what is going on 👍

[alisdair]

Thanks! Some guesses/thoughts in case it helps with diagnosis:

• The diff you see _should_ only happen if the value from the data source changed somehow since the last state refresh. Since the diff UI suppresses it, you might want to look at the JSON plan output to check.

• If you can get a consistent reproduction, comparing the output with the sensitive experiment disabled would be an interesting step.

• It's also interesting to know if the Vault secret data source was refreshed as part of the plan.

Looking forward to more information! Thanks for looking into this further.

[romeomorcia]

@alisdair Looks like something was being changed in the backend but it wasn't being shown if the value that is going to be set is null!

After disabling the experiments = [provider_sensitive_attrs] and running terraform plan I got this:

PS: probably terraform think that this will change the value of the secret, instead is just changing the content_type

Could be this the case?

Also, shouldn't be those set to some default if not declared? Why are they set as null instead?

[alisdair]

I'm not sure about the specifics of the Azure APIs, but it's not surprising to me that some fields could be set to null if nothing is specified. If that's causing problems, I'd recommend reporting the issue on the AzureRM provider.

If you're changing Terraform-managed resources using another API or the Azure portal, it's not surprising that Terraform has odd behaviour like this when planning. I don't quite understand how this is related to the original issue, though.

On this specific sensitive issue, are you able to reliably reproduce the original problem? If so, can you check if the hidden sensitive value from the data source is actually changing by inspecting the JSON plan, or by disabling the experiment and retrying?

[romeomorcia]

@alisdair I'm not sure I understand the third paragraph.
The issue I've been experiencing shows exactly what is causing the resources to be changed.

If the experiment is enabled, even though something that is not sensitive is being modified (even if this something is not in the same block where the sensitive values are being shown) and replaced by null, the resource will see the change.

Let's take for example the function:
I did not change anything in the code (i have no os_type written in the function app) between the two terraform plan and because somehow the function app is setting by default os_type to "", every time when I do terraform plan, the function will be changing this value to null (because i did not declare it). This is shown when experiments = [provider_sensitive_attrs] is not set.
But when I set it in the code, we will have this change hidden but yet terraform will perform it (it will change it to null and after set it back to its default which is "").

Having this said, the issue probably is not on azurerm side BUT in the moment when terraform shows what is going to be changed; But because some sort of bug, if there is a resource that is having their values changed from something (or maybe _ONLY_ it's default) to null then the value is not being shown in the console even if the value that is being changed is not any type of secret or hidden value.

The same happens with the password for I've posted above; In fact this secret is being set as sql_db_connection_string for my sqlserver that i've raised the issue in the beginning.
Even though this connection string is not being touched, in terraform the resource is modified which could trigger a possible flag "something changed in the secret, therefore i have to update also the connection string of the sqlserver".

THIS would explain also why i'm only seeing the changes on my azurerm_sql_database when I have the experiments = [provider_sensitive_attrs] not set and not when I have them set. In fact, my SQL DB does not have any sensitive value set (which is why nothing is being shown as changed even though the changes are happening)

[alisdair]

I'm a bit lost now, sorry. Your latest message seems to be for an entirely separate set of resources from the original report, and we're now talking about several other issues that seem to be different from the first one.

For us to investigate what's going on here, we need to be able to reproduce the issue. Let's focus and return to the original report: your azurerm_sql_server instance had an unexpected change to the administrator_login_password attribute when running a plan, and it showed (sensitive value).

1. Can you show me a configuration which does this reliably? Starting with my attempted reproduction configuration might help.
2. If you can't find a reduced configuration, but can reproduce locally, please show the full output of two runs of terraform plan: one with the experiment enabled, and one with it disabled. (Of course please censor the password value!)

Thanks in advance! Hope we can figure out what's happening here.

[sharebear]

I apologise for deviating to the request to stick to the resources in the original report but the I got surprise upgraded to 0.14 tonight (my bad, lazy github actions configuration) and I'm seeing the same issue (also with the azurerm provider).

  # module.<redacted>.azurerm_function_app.updates will be updated in-place
  ~ resource "azurerm_function_app" "updates" {
      ~ app_settings                   = {
          # Warning: this attribute value will be marked as sensitive and will
          # not display in UI output after applying this change
          ~ "APPINSIGHTS_INSTRUMENTATIONKEY" = (sensitive)
            # (6 unchanged elements hidden)
        }
        id                             = "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Web/sites/<redacted>"
        name                           = "<redacted>"
      # Warning: this attribute value will be marked as sensitive and will
      # not display in UI output after applying this change
      ~ storage_account_access_key     = (sensitive value)
        tags                           = {
            "environment" = "dev"
        }
        # (18 unchanged attributes hidden)




        # (4 unchanged blocks hidden)
    }

Interestingly I couldn't get a reduced test case to fail. Not obvious what the difference is yet.... I need to go to bed now but can try and follow up tomorrow.

[sharebear]

Interesting observation here... a little more context from my setup. I have two modules base and applicationX. Module base creates an appinsights instance and a storage account and has outputs for the access key and instrumentation key noted above. These outputs are marked as sensitive in my config.
The applicaitonX module then has variables to receive these values. These variables do not have the sensitive flag set.

If I do edit the variables to add the sensitive flag I get the following error from terraform plan

Error: .app_settings["APPINSIGHTS_INSTRUMENTATIONKEY"]: value has marks, so it cannot be serialized

Assuming this is relevant information I should be able to craft a reproduction quite easily.

[sharebear]

@alisdair I think I was able to craft a reproduction here https://github.com/sharebear/terraform-bug-reproductions/tree/master/26959

[douglaswth]

Interesting observation here... a little more context from my setup. I have two modules base and applicationX. Module base creates an appinsights instance and a storage account and has outputs for the access key and instrumentation key noted above. These outputs are marked as sensitive in my config.
The applicaitonX module then has variables to receive these values. These variables do _not_ have the sensitive flag set.

If I do edit the variables to add the sensitive flag I get the following error from terraform plan

Error: .app_settings["APPINSIGHTS_INSTRUMENTATIONKEY"]: value has marks, so it cannot be serialized

Assuming this is relevant information I should be able to craft a reproduction quite easily.

The behavior you described in this comment seems like what I just reported in #27095.

[alisdair]

@sharebear Thanks for your reproduction case! For clarity, my summary of what's happening here is:

• Resource has an attribute which is bound to the value of a sensitive output
• This causes a plan where the diff displays as a change of a value from non-sensitive to sensitive
• Applying this plan does not store this new sensitive value in state, so it's a permanent diff

That's something that we should be able to fix. I'm going to try to find an even more minimal reproduction case, but this has already been very helpful, so thanks again.

As noted by @douglaswth, the crash you're seeing does seem to be a duplicate of #27095, which is also on our list of bugs to fix for 0.14.1.

[alisdair]

Reduced reproduction case:

main.tf:

module "secrets" {
  source = "./secrets"
}

resource "random_pet" "pet" {
  prefix = module.secrets.prefix
}

secrets/main.tf:

output "prefix" {
  value = "obsequious"
  sensitive = true
}

Reproduction steps:

• terraform-0.13.5 init
• terraform-0.13.5 apply -auto-approve
• terraform-0.14.0 init
• terraform-0.14.0 plan (this has a plan to note the change to a now-sensitive attribute value)
• terraform-0.14.0 apply -auto-approve (this seems to change the resource but doesn't)
• terraform-0.14.0 plan (same diff as before as the sensitive path has not been written to state)

[alisdair]

This issue will be fixed in the upcoming Terraform 0.14.1. Thanks again for reporting and helping us to reproduce it!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.