Terraform-provider-azurerm: Deleting keyvault+certificate+policy deletes policy first, then fails to delete certificate

[pixelicous]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

terraform 0.11.11
azurerm 1.23.0

Affected Resource(s)

• azurerm_key_vault_access_policy
• azurerm_key_vault_certificate
• azurerm_key_vault

Terraform Configuration Files

data "azurerm_client_config" "current" {}

resource "azurerm_key_vault_certificate" "test" {
  count                 = "${var.vm_instances}"
  name         = "test"
  key_vault_id = "${azurerm_key_vault.test.id}"

  certificate_policy {
    issuer_parameters {
      name = "Self"
    }

    key_properties {
      exportable = true
      key_size   = 2048
      key_type   = "RSA"
      reuse_key  = true
    }

    lifetime_action {
      action {
        action_type = "AutoRenew"
      }

      trigger {
        days_before_expiry = 30
      }
    }

    secret_properties {
      content_type = "application/x-pkcs12"
    }

    x509_certificate_properties {
      # Server Authentication = 1.3.6.1.5.5.7.3.1
      # Client Authentication = 1.3.6.1.5.5.7.3.2
      extended_key_usage = ["1.3.6.1.5.5.7.3.1"]

      key_usage = [
        "cRLSign",
        "dataEncipherment",
        "digitalSignature",
        "keyAgreement",
        "keyCertSign",
        "keyEncipherment",
      ]

      subject_alternative_names {
        dns_names = ["test"]
      }

      subject            = "CN=test"
      validity_in_months = 12
    }
  }
}

#Keyvault
resource "azurerm_key_vault" "test" {
  name                = "name-kv"
  location            = "${terraform.workspace}"
  resource_group_name = "${azurerm_resource_group.test.name}"

  tenant_id = "${data.azurerm_client_config.current.tenant_id}"

  sku {
    name = "standard"
  }

  enabled_for_deployment          = true
  enabled_for_template_deployment = true

  tags = "${var.tags}"
}

#Policy

resource "azurerm_key_vault_access_policy" "spn_terraform" {
  key_vault_id = "${azurerm_key_vault.test.id}" 

  tenant_id = "${data.azurerm_client_config.current.tenant_id}"
  object_id = "${data.azurerm_client_config.current.service_principal_object_id}"

  certificate_permissions = [
    "create",
    "delete",
    "deleteissuers",
    "get",
    "getissuers",
    "ifrt",
    "list",
    "listissuers",
    "managecontacts",
    "manageissuers",
    "setissuers",
    "update",
  ]

  key_permissions = [
    "backup",
    "create",
    "decrypt",
    "delete",
    "encrypt",
    "get",
    "import",
    "list",
    "purge",
    "recover",
    "restore",
    "sign",
    "unwrapKey",
    "update",
    "verify",
    "wrapKey",
  ]

  secret_permissions = [
    "backup",
    "delete",
    "get",
    "list",
    "purge",
    "recover",
    "restore",
    "set",
  ]
}

Debug Output

Error: Error applying plan:

1 error(s) occurred:

• azurerm_key_vault_certificate.test (destroy): 1 error(s) occurred:

• azurerm_key_vault_certificate.test: Error deleting Certificate "test" from Key Vault: keyvault.BaseClient#DeleteCertificate: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Access denied" InnerError={"code":"AccessDenied"}

Expected Behavior

Keyvault related entities should be removed before policy

Actual Behavior

Policy got removed before, denying the option to remove the certificate, everything was destroyed in the resource group besides the keyvault and the certificate

Steps to Reproduce

terraform apply

Important Factoids

In my HCL files i also had VM resource, nic etc, but these were removed successfully. Terraform didn't finish and exit with error. Consecutive executions resulted in the same behavior.

[pixelicous]

Did anyone stumble upon this particular problem? Seems to me I might be doing something wrong as this is basic and it would be weird if I am the first to write on it

[infogulch]

I'm encountering this same issue. When deleting a key vault with secrets in it, terraform is deleting its own access before trying to delete the keys/secrets/certs inside. Silly terraform, don't do that!

Using azure provider version 1.24, and terraform 0.11.13.

[pixelicous]

@tombuildsstuff @katbyte Hey! Any chance you could tag this for a check up?? Thanks in advance! 🙏

[pixelicous]

@infogulch Did you overcome this problem?? It is a real pain when destroying resource to go to the portal and delete those resources yourself.

[infogulch]

No I haven't found a solution yet. I was going to try manually specifying dependencies with depends_on next if it remained a problem, but my current environments are not churning very quickly right now so I haven't had the impetus to try it yet.

[pixelicous]

Thanks! Hopefully somebody will reply on this matter soon.. i reproduced it on a couple of machines so it is either a terraform problem or provider problem..

[tombuildsstuff]

@pixelicous in this instance there's no dependencies between the azurerm_key_vault_certificate resource and the azurerm_key_vault_access_policy resource - such that it's possible for either to be destroyed in any order.

As @infogulch has mentioned I'd investigate using a depends_on to ensure the Access Policies are created prior to any resources within this; or specifying them inline within the azurerm_key_vault resource such that they're created with the Key Vault itself - which should solve this; would you be able to take a look and see if that works for you?

Since this is a question about Terraform Configuration rather than a bug in Terraform I'm going to close this issue for the moment - but please let us know if that doesn't work for you and we'll take another look.

Thanks!

[infogulch]

I very much disagree with closing this as solved. The issue is that running terraform apply with 3 simple resources followed by terraform destroy which fails because terraform randomly chose the wrong order to destroy items.

There is a deletion dependency here -- you must delete all keys/secrets/certs in the keyvault before removing any of the access policies. (Or terraform could just delete the whole keyvault which manages all of them so they automatically disappear, but that might be a separate discussion.) The fact that terraform-provider-azurerm just doesn't know about this (very real) dependency that's built into the Azure platform doesn't make it a configuration error.

For example, when terraform destroys a vm it destroys it's components in the correct order. It doesn't try to destroy the nic before the vm is destroyed because that's an invalid state of the platform and it's literally the terraform provider's raison d'etre to create and destroy things in the order that the platform allows and expects.

[pixelicous]

@infogulch @tombuildsstuff
Why close this though? This is an issue and a bug, there are many resources terraform knows of their dependency, and this dependency is "very real" as @infogulch put it.
In the same way that terraform knows to build keyvault>policy>secret/certificate without any dependency set (because you cannot create a secret without a policy or keyvault) it should know to delete it the same way the other way around.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!