Terraform-provider-azurerm: Having to run apply twice to get correct version when updating keyvault value.

[sonic1981]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/azurerm v2.29.0

Affected Resource(s)

• azurerm_key_vault_secret
• azurerm_key_vault
• azurerm_function_app

Terraform Configuration Files

resource "azurerm_key_vault" "kv" {
  name                                = "..."
  location                            = azurerm_resource_group.rg.location
  resource_group_name                 = azurerm_resource_group.rg.name
  enabled_for_disk_encryption         = true
  tenant_id                           = data.azurerm_client_config.current.tenant_id
  soft_delete_enabled                 = true
  purge_protection_enabled            = false

  sku_name                            = "standard"

  tags                                = var.tags
}

resource "azurerm_key_vault_secret" "UnoSoftwareID" {
  key_vault_id                      = azurerm_key_vault.kv.id
  name                              = "UnoSoftwareID"
  value                             = "<sensitive>"
}

resource "azurerm_function_app" "takeapi" {
  ...
 app_settings = {
      "UnoSoftwareID"                               = "@Microsoft.KeyVault(SecretUri=${ azurerm_key_vault.keyvault.vault_uri }secrets/UnoSoftwareID/${ azurerm_key_vault_secret.UnoSoftwareID.version })"
  }

Expected Behavior

Updating the key vault secret value should update the connection string in the azurerm_function_app. Specifically the version number. Without a second run.

Actual Behavior

New key vault value is added but the new version is not updated. Running a second time does upate the version reference but I'd expect this to work first time. This has meant that we've had issues with the old key vault secret in app is used.

Steps to Reproduce

1) First request applies all changes correctly resulting in

+ resource "azurerm_key_vault_secret" "UnoSoftwareID" {
      + id           = (known after apply)
      + key_vault_id = "<sensitive>"
      + name         = "UnoSoftwareID"
      + value        = (sensitive value)
      + version      = (known after apply)
    }

  # module.process.azurerm_function_app.process will be updated in-place
  ~ resource "azurerm_function_app" "process" {
        app_service_plan_id            = "<sensitive>"
      ~ app_settings                   = {

        } -> (known after apply)

2) If you then change the value of the keyvault secret and run apply again:

  ~ resource "azurerm_key_vault_secret" "UnoSoftwareID" {
        id           = "<sensitive>/secrets/UnoSoftwareID/e4e3d1582ac04157b87713a7928d99c1"
        key_vault_id = "<sensitive>/providers/Microsoft.KeyVault/vaults/<sensitive>"
        name         = "UnoSoftwareID"
        tags         = {}
      ~ value        = (sensitive value)
        version      = "e4e3d1582ac04157b87713a7928d99c1"
    }

No changes are made to the azurerm_function_app app_settings despite the reference using azurerm_key_vault_secret.UnoSoftwareID.version. The version value here is the old version value, not the new one of the new keyvault secret.

3) Running this again then results in:

  ~ resource "azurerm_function_app" "process" {
      ~ app_settings                   = {
          ~ "UnoSoftwareID"                      = "@Microsoft.KeyVault(SecretUri=<sensitive>/secrets/UnoSoftwareID/e4e3d1582ac04157b87713a7928d99c1)" -> "@Microsoft.KeyVault(SecretUri=<sensitive>secrets/UnoSoftwareID/69e1ab8bfc394e239430387ae4208da5)"

This then appears to want to update to the new value. e4e3d1582ac04157b87713a7928d99c1 -> 69e1ab8bfc394e239430387ae4208da5. Once this has happened everything seems to be correct but I've had to run terraform apply, twice.

[sonic1981]

This also seems to be intermittently producing this error:

When expanding the plan for module.process.azurerm_function_app.process to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
.app_settings["MongoConnectionString"]: was
cty.StringVal("@Microsoft.KeyVault(SecretUri=sensitive/secrets/MongoConnectionString/13cb1a8acee04f208065a4a4efc7e8ad)"),
but now
cty.StringVal("@Microsoft.KeyVault(SecretUri=sensitive/secrets/MongoConnectionString/0bdae9738bf44ab092e8a59fec5e1709)").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Re-running seems to manage to get past it

[magne]

This is most definitely a bug. Even tainting a keyvault secret (which will replace it) does not update any usage of the secret. As far as I understand, any update of a secret value will produce a new version of the secret, and since the terraform secret includes the version in the ID it should be a destroy/create operation, not an update-in-place operation. A secret (and possibly other keyvault items) should probably be some kind of versioned resource, as I don't think you currently can disable a previous version, for example.

Also, using the

app_settings = {
  "SECRET" = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.kv.vault_uri}secrets/${azurerm_key_vault_secret.secret.name}/${azurerm_key_vault_secret.secret.version})"
}

construct never updates my function_app settings after a secret value update (azurerm v. 2.29).

[mboutet]

I'm not sure if it's related or not, but it seems the problem began when I enabled soft-delete on my keyvault.

[sonic1981]

@mboutet Well disabling soft delete on key vaults is now deprecated so disabling soft delete will not be an option soon.

[mboutet]

~@sonic1981, it seems that with the latest release, the bug is no longer present, can you confirm?~

Edit: I spoke too soon, the bug is still present.

[vladnega]

I've upgraded azurerm to the latest version (v2.34.0), but I'm still getting the same behaviour and error.

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/azurerm v2.34.0

...

azurerm_key_vault_secret.phub_sp_pass: Modifying... [id=https://xxx.vault.azure.net/secrets/phub-sp-pass/26a772e83a8e4cfca5bad6f3c65425f0]
azurerm_key_vault_secret.phub_sp_pass: Modifications complete after 2s [id=https://xxx.vault.azure.net/secrets/phub-sp-pass/edcb10ef1bb544fd8c781f65b16a62a4]

Error: Provider produced inconsistent final plan

When expanding the plan for azurerm_function_app.phub to include new values
learned so far during apply, provider
"registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
.app_settings["AzureAd:ClientSecret"]: was
cty.StringVal("@Microsoft.KeyVault(SecretUri=https://xxx.vault.azure.net/secrets/phub-sp-pass/26a772e83a8e4cfca5bad6f3c65425f0)"),
but now
cty.StringVal("@Microsoft.KeyVault(SecretUri=https://xxx.vault.azure.net/secrets/phub-sp-pass/edcb10ef1bb544fd8c781f65b16a62a4)").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

[ttjackott]

Same issue - this is causing production issues as our key vault secrets aren't updating to the newest values. We have to manually delete the secrets and re-apply in order to get them to refresh.

[ttjackott]

@tombuildsstuff @WodansSon can the priority and tags be changed on this? Key vault secret deployments are broken