Terraform-provider-azurerm: azurerm_key_vault_secret.id is not updated if new version is created

[StefanSchoof]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Affected Resource(s)

• azurerm_key_vault_secret

Terraform Configuration Files

provider "azurerm" {
    version = "~> 1.23.0"
}

variable "objectid" {
    type = "string"
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "tf-secret-test"
  location = "westeurope"
}

resource "azurerm_storage_account" "this" {
    name                      = "tfsecrettest"
    resource_group_name       = "${azurerm_resource_group.this.name}"
    location                  = "${azurerm_resource_group.this.location}"
    account_tier              = "Standard"
    account_replication_type  = "LRS"
    account_kind              = "StorageV2"
}

resource "azurerm_key_vault" "this" {
    name                      = "tf-secret-test"
    resource_group_name       = "${azurerm_resource_group.this.name}"
    location                  = "${azurerm_resource_group.this.location}"
    tenant_id                 = "${data.azurerm_client_config.current.tenant_id}"

    sku {
        name = "standard"
    }
}

resource "azurerm_key_vault_access_policy" "service" {
    key_vault_id = "${azurerm_key_vault.this.id}"

    tenant_id = "${data.azurerm_client_config.current.tenant_id}"
    object_id = "${var.objectid}"

    key_permissions = [
    ]

    secret_permissions = [
      "get",
      "set",
      "list",
      "delete",
    ]
  }

  resource "azurerm_key_vault_access_policy" "gui" {
   key_vault_id = "${azurerm_key_vault.this.id}"

    tenant_id = "${azurerm_function_app.gui.identity.0.tenant_id}"
    object_id = "${azurerm_function_app.gui.identity.0.principal_id}"

    key_permissions = [
    ]

    secret_permissions = [
      "get",
    ]
  }

resource "azurerm_key_vault_secret" "IotHubConnection" {
    name         = "IotHubConnection"
    value        = "secret"
    key_vault_id = "${azurerm_key_vault.this.id}"

    depends_on = ["azurerm_key_vault_access_policy.service"]
}

resource "azurerm_app_service_plan" "this" {
    name                = "tf-secret-test"
    resource_group_name = "${azurerm_resource_group.this.name}"
    location            = "${azurerm_resource_group.this.location}"
    kind                = "functionapp"

    sku {
        tier = "Dynamic"
        size = "Y1"
    }
}

resource "azurerm_function_app" "gui" {
  name                      = "tf-secret-test"
  resource_group_name       = "${azurerm_resource_group.this.name}"
  location                  = "${azurerm_resource_group.this.location}"
  app_service_plan_id       = "${azurerm_app_service_plan.this.id}"
  storage_connection_string = "${azurerm_storage_account.this.primary_connection_string}"
  identity {
    type = "SystemAssigned"
  }

  app_settings {
    FUNCTIONS_WORKER_RUNTIME       = "dotnet"
    IotHubConnection               = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault_secret.IotHubConnection.id})"
    WEBSITE_RUN_FROM_PACKAGE       = "1"
  }
  version = "~2"
}

Debug Output

Panic Output

Expected Behavior

Changing the secret eg. through adding a "2" in the secret does change the id of secret. So the id of the current secret version is written to the environment var of the function. This done in a single terraform apply.

Actual Behavior

Changing the secret eg. through adding a "2" in the secret does not change the id of secret. So the environment var of the function is not written. Running terraform again will change the environment var.

Steps to Reproduce

1. terraform apply -var 'objectid=<objectid of user>
2. change secret in azurerm_key_vault_secret.IotHubConnection to secret2
3. terraform apply -var 'objectid=<objectid of user>
4. terraform apply -var 'objectid=<objectid of user>

Important Factoids

The uri of a key vault sercet changes for every version. (see
https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references#reference-syntax)

References

• #0000

[daveobike]

I just ran into this same issue attempting to do what @StefanSchoof described, but my azurerm_key_vault_secret id happens to be output from a separate module and I am referencing the output for use in a function_app app_settings block. But the behavior is the same:

1. Initial terraform apply creates a sets a azurerm_key_vault_secret, sets the secret value, properly evaluates the resultant secret id, and passes the secret id to a separate module app_setting block for the function app. All is well and as expected.
2. Run terraform apply with an updated secret value. Terraform properly sets the updated secret value (technically a new secret value version in azure) and displays the updated value id console. But, the dependent module variable does not change (assuming a dependency or state error is missing the updated id value?)and therefore the app_settings block is not updated with the new id value.
3. Re-run terraform apply, with no explicit changes, and terraform now picks up the discrepancy in the dependent module variable, tied to the output value of the secret id.

The following alternate path creates an explicit terraform error:
Repeat Step 1 above
Repeat Step 2 above

1. Change the secret value to a different value from Step 1 and run terraform apply
2. The following error is produced:

Error: Provider produced inconsistent final plan

When expanding the plan for module.api.azurerm_function_app.api-dev to include
new values learned so far during apply, provider "azurerm" produced an invalid
new value for .app_settings["xxxxxxAuthKey"]: was
cty.StringVal("@Microsoft.KeyVault(SecretUri=https://xxxxxx-dev.vault.azure.net/secrets/xxxxxx-key/139fcc1e99e44dc888ab52ba8b4b7e39)"),
but now
cty.StringVal("@Microsoft.KeyVault(SecretUri=https://xxxxxx-dev.vault.azure.net/secrets/xxxxxx-key/7d479680095f48c58d3b49c1047ff847)").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

The bug will repeat from there on out because the secret id will always be out of sync (and behind) secret value changes. The bug can be cleared by running terraform apply with out updating secret value from Step 3, effectively getting the out of sync id value back in current state.

[jagregory]

I'm seeing the same thing, except I'm explicitly using the version of the azurerm_key_vault_secret. version also doesn't appear to cause any cascading updates when a new version is created. It's only on a subsequent run of apply does Terraform release the version has changed and updates my app_settings.

[bradcavanagh-ffs]

I'm seeing the same thing, except I'm explicitly using the version of the azurerm_key_vault_secret. version also doesn't appear to cause any cascading updates when a new version is created. It's only on a subsequent run of apply does Terraform release the version has changed and updates my app_settings.

@jagregory I was running into the same thing, but it looks like if I explicitly taint that resource by running terraform taint module.keyvault.azurerm_key_vault_secret.secret then the subsequent apply appears to work. It's a bit of a hammer because the secret gets deleted and created again, but it seems like this is a workaround.

[syedhassaanahmed]

If you're always interested in latest version of the Key Vault secret, you can use this workaround - basically omit the version from SecretUri. e.g. https://kv-we-retrieve-kv-secret.vault.azure.net/secrets/MySecretValue/

[sebader]

@syedhassaanahmed depending on where you use the secret, this workaround will not help. For instance when using KeyVault-referenced AppSettings in AppService, you would manually need to restart the AppService after you updated a secret for it to be picked up. Otherwise the AppService will not fetch the "latest".

[cosminstirbu]

Any news on this item or other workarounds that don't require a restart to the App Service?

[sanderginn]

I'm also interested if there will be a fix for this. Only knowing about the version ID after a modification breaks our CI pipeline.

[cosminstirbu]

Our workaround was to run terraform with the target option and update the secrets first and only after run it for all resources (without the target option, as you normally do).

This might not be ideal when running it from your local machine, but it definitely helps when you run it part of your Azure Devops pipeline for instance.

[sanderginn]

Thanks, @cosminstirbu - that is a neat and easy solution which provided the workaround we needed!