Terraform-provider-azurerm: Error deleting a VM linked to a backend address pool association

[mikhailshilkov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.5
provider.azurerm v1.33.0

Affected Resource(s)

• azurerm_network_interface_backend_address_pool_association
• azurerm_lb_backend_address_pool
• azurerm_virtual_machine

Terraform Configuration Files

provider "azurerm" {
  version = "~> 1.33"
}

variable "prefix" {
  default = "tfvmex"
}

resource "azurerm_resource_group" "test" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_virtual_network" "test" {
  name                = "example-network"
  address_space       = ["10.0.0.0/16"]
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
}

resource "azurerm_subnet" "test" {
  name                 = "internal"
  resource_group_name  = "${azurerm_resource_group.test.name}"
  virtual_network_name = "${azurerm_virtual_network.test.name}"
  address_prefix       = "10.0.2.0/24"
}

resource "azurerm_public_ip" "test" {
  name                = "example-pip"
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
  allocation_method   = "Static"
}

resource "azurerm_lb" "test" {
  name                = "example-lb"
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"

  frontend_ip_configuration {
    name                 = "primary"
    public_ip_address_id = "${azurerm_public_ip.test.id}"
  }
}

resource "azurerm_lb_backend_address_pool" "test" {
  resource_group_name = "${azurerm_resource_group.test.name}"
  loadbalancer_id     = "${azurerm_lb.test.id}"
  name                = "acctestpool"
}

resource "azurerm_network_interface" "test" {
  name                = "example-nic"
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"

  ip_configuration {
    name                          = "testconfiguration1"
    subnet_id                     = "${azurerm_subnet.test.id}"
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_network_interface_backend_address_pool_association" "test" {
  network_interface_id    = "${azurerm_network_interface.test.id}"
  ip_configuration_name   = "testconfiguration1"
  backend_address_pool_id = "${azurerm_lb_backend_address_pool.test.id}"
}

resource "azurerm_virtual_machine" "main" {
  name                  = "${var.prefix}-vm"
  location              = "${azurerm_resource_group.test.location}"
  resource_group_name   = "${azurerm_resource_group.test.name}"
  network_interface_ids = ["${azurerm_network_interface.test.id}"]
  vm_size               = "Standard_DS1_v2"

  delete_os_disk_on_termination = true
  delete_data_disks_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
  storage_os_disk {
    name              = "myosdisk1"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }
  os_profile {
    computer_name  = "hostname"
    admin_username = "testadmin"
    admin_password = "Password1234!"
  }
  os_profile_linux_config {
    disable_password_authentication = false
  }
  tags = {
    environment = "staging"
  }
}

Expected Behavior

The resources can be successfully created and terraform apply and deleted with terraform destroy.

Actual Behavior

terraform destroy fails with

Error waiting for removal of Backend Address Pool Association for NIC "example-nic" 
(Resource Group "example-resources"): Code="OperationNotAllowed" 
Message="Operation 'startTenantUpdate' is not allowed on VM 'tfvmex-vm' since 
the VM is marked for deletion. You can only retry the Delete operation (or wait for an 
ongoing one to complete)." Details=[]

Since there's no dependency between the VM and the Backend Address Pool resources, the pool is deleted before the VM, which causes to VM deletion errror. VM deletion should handle this situation gracefully. Alternatively, the resource model should be adjusted to prevent the backend pool association to be deleted before the VM.

Steps to Reproduce

1. terraform apply
2. terraform destroy

[ArcturusZhang]

I am trying to reproduce this issue in go sdk using this code snippet, but I did not reproduce this issue.

When I change the order of destruction of each resource, it may fail, but with different error message.
When I am using the tf configurations you posted, sometimes I does not get any errors and everything is destroying normally, sometimes I get same error as you did.

I suspect this is related with the order terraform execute the destruction of each resource.

[mybayern1974]

Given this is not-stable repro and related with the terraform resource deletion order, @tombuildsstuff to possibly comment on whether there could be any fundamental reason

[tombuildsstuff]

hi @mikhailshilkov

Thanks for opening this issue.

Taking a look into this this appears to work as intended on v1.34:

$ tf init

Initializing the backend...

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
✔ ~/code/src/tmp/4330
08:59 $ tfyolo
azurerm_resource_group.test: Creating...
azurerm_resource_group.test: Creation complete after 2s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources]
azurerm_virtual_network.test: Creating...
azurerm_public_ip.test: Creating...
azurerm_public_ip.test: Creation complete after 3s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/publicIPAddresses/example-pip]
azurerm_lb.test: Creating...
azurerm_lb.test: Creation complete after 2s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb]
azurerm_lb_backend_address_pool.test: Creating...
azurerm_lb_backend_address_pool.test: Creation complete after 1s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_virtual_network.test: Still creating... [10s elapsed]
azurerm_virtual_network.test: Creation complete after 11s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network]
azurerm_subnet.test: Creating...
azurerm_subnet.test: Still creating... [10s elapsed]
azurerm_subnet.test: Creation complete after 11s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network/subnets/internal]
azurerm_network_interface.test: Creating...
azurerm_network_interface.test: Creation complete after 1s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic]
azurerm_network_interface_backend_address_pool_association.test: Creating...
azurerm_virtual_machine.main: Creating...
azurerm_network_interface_backend_address_pool_association.test: Creation complete after 1s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic/ipConfigurations/testconfiguration1|/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_virtual_machine.main: Still creating... [10s elapsed]
azurerm_virtual_machine.main: Still creating... [20s elapsed]
azurerm_virtual_machine.main: Still creating... [30s elapsed]
azurerm_virtual_machine.main: Still creating... [40s elapsed]
azurerm_virtual_machine.main: Still creating... [50s elapsed]
azurerm_virtual_machine.main: Still creating... [1m0s elapsed]
azurerm_virtual_machine.main: Still creating... [1m10s elapsed]
azurerm_virtual_machine.main: Still creating... [1m20s elapsed]
azurerm_virtual_machine.main: Still creating... [1m30s elapsed]
azurerm_virtual_machine.main: Creation complete after 1m34s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachines/tfvmex-vm]

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
✔ ~/code/src/tmp/4330
09:01 $ envchain azurerm terraform destroy --auto-approve
azurerm_resource_group.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources]
azurerm_virtual_network.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network]
azurerm_public_ip.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/publicIPAddresses/example-pip]
azurerm_lb.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb]
azurerm_subnet.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network/subnets/internal]
azurerm_lb_backend_address_pool.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_network_interface.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic]
azurerm_network_interface_backend_address_pool_association.test: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic/ipConfigurations/testconfiguration1|/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_virtual_machine.main: Refreshing state... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachines/tfvmex-vm]
azurerm_network_interface_backend_address_pool_association.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic/ipConfigurations/testconfiguration1|/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_virtual_machine.main: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachines/tfvmex-vm]
azurerm_network_interface_backend_address_pool_association.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...ple-lb/backendAddressPools/acctestpool, 10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 10s elapsed]
azurerm_network_interface_backend_address_pool_association.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...ple-lb/backendAddressPools/acctestpool, 20s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 20s elapsed]
azurerm_network_interface_backend_address_pool_association.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...ple-lb/backendAddressPools/acctestpool, 30s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 30s elapsed]
azurerm_network_interface_backend_address_pool_association.test: Destruction complete after 31s
azurerm_lb_backend_address_pool.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb/backendAddressPools/acctestpool]
azurerm_lb_backend_address_pool.test: Destruction complete after 2s
azurerm_lb.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/loadBalancers/example-lb]
azurerm_lb.test: Destruction complete after 0s
azurerm_public_ip.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/publicIPAddresses/example-pip]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 40s elapsed]
azurerm_public_ip.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-....Network/publicIPAddresses/example-pip, 10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 50s elapsed]
azurerm_public_ip.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-....Network/publicIPAddresses/example-pip, 20s elapsed]
azurerm_public_ip.test: Destruction complete after 21s
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m0s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m20s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m30s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m40s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 1m50s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m0s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m20s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m30s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m40s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 2m50s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m0s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m20s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m30s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m40s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 3m50s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 4m0s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 4m10s elapsed]
azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...soft.Compute/virtualMachines/tfvmex-vm, 4m20s elapsed]
azurerm_virtual_machine.main: Destruction complete after 4m21s
azurerm_network_interface.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/networkInterfaces/example-nic]
azurerm_network_interface.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-....Network/networkInterfaces/example-nic, 10s elapsed]
azurerm_network_interface.test: Destruction complete after 12s
azurerm_subnet.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network/subnets/internal]
azurerm_subnet.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...works/example-network/subnets/internal, 10s elapsed]
azurerm_subnet.test: Destruction complete after 11s
azurerm_virtual_network.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network]
azurerm_virtual_network.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...etwork/virtualNetworks/example-network, 10s elapsed]
azurerm_virtual_network.test: Destruction complete after 11s
azurerm_resource_group.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...f2c02/resourceGroups/example-resources, 10s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...f2c02/resourceGroups/example-resources, 20s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...f2c02/resourceGroups/example-resources, 30s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-...f2c02/resourceGroups/example-resources, 40s elapsed]
azurerm_resource_group.test: Destruction complete after 47s

Destroy complete! Resources: 9 destroyed

Would you be able to take a look and see if upgrading fixes this issue for you? Since this appears to be fixed in 1.34 I'm going to close this issue for the moment but please let us know if this is still an issue after upgrading and we'll take another look.

Thanks!

[mikhailshilkov]

@tombuildsstuff I upgraded to 1.34, tried again, and the destruction still fails for me with the same error:

azurerm_virtual_machine.main: Still destroying... [id=/subscriptions/...soft.Compute/virtualMachines/tfvmex-vm, 2m30s elapsed]
azurerm_virtual_machine.main: Destruction complete after 2m31s

Error: Error waiting for removal of Backend Address Pool Association for NIC "example-nic" 
(Resource Group "example-resources"): Code="OperationNotAllowed" Message="Operation 
'startTenantUpdate' is not allowed on VM 'tfvmex-vm' since the VM is marked for deletion. 
You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]

Ran apply-destroy 3 times, got 3 errors. A subsequent destroy after the failure succeeds.

I'm on Terraform 0.12.8 on Windows, running the copy-pasted script from the above (updated to ~> 1.34).

[tombuildsstuff]

@mikhailshilkov out of interest is there any organizational-wide policies applied to the Tenant/Subscription, e.g. Security Center? Whilst we could add locks within the VM resources deletion function on the NIC ID, the startTenantUpdate sounds like this is something mutating the resources outside of Terraform?

[mikhailshilkov]

@tombuildsstuff It's my personal subscription that I use for test purposes, with no policies etc.

[dbilleci-lightstream]

Could someone explain why this is closed? We're seeing more dependency issues suddenly appearing in this same vein. I'll work on some reproduction code if needed, but this is clearly not a solved issue.

[mybayern1974]

@dbilleci-lightstream @mikhailshilkov , in addition to identify the root cause that could be related with the deletion dependency, may i suggest you have a try on the work around of using the depends_on feature in terraform? It explicitly declares the dependency among resources, though you need to understand ahead of time the correct dependency among VM, LB and Backend_Pool. We'll meanwhile try this workaround on our side and will update here for any of our progress.

[mikhailshilkov]

The depends_on workaround works, that's what I use from the day I discovered the problem. However, it shouldn't be needed and this issue should be reopened, in my opinion.

[sorenhansendk]

@mikhailshilkov Which dependency have you added? Can you please paste your TF-file?

[mikhailshilkov]

@sorenhansendk In the TF-file from the original message, you should add a depends_on from resource "azurerm_virtual_machine" "main" to azurerm_lb_backend_address_pool.test

[sorenhansendk]

@mikhailshilkov Ok, I will try that 👍 Thank you for the quick response!

[sorenhansendk]

@mikhailshilkov Hmm - the problem still exists for me. It this not what you mean?

I have removed lot of configuration, that is what the ... means 😎

resource "azurerm_virtual_machine" "vm" {
   ...
   depends_on = [azurerm_lb_backend_address_pool.nodes]
}

resource "azurerm_lb_backend_address_pool" "nodes" {
   ...
}

Error: Error waiting for removal of Backend Address Pool Association for NIC "XXX" (Resource Group "XXX"): Code="OperationNotAllowed" Message="Operation 'startTenantUpdate' is not allowed on VM 'XXX' since the VM is marked for deletion. You can only retry the Delete operation (or wait for an ongoing one to complete)." Details=[]

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

[tombuildsstuff]

@sorenhansendk depending on the configuration being used you may need to add an explicit dependency on the azurerm_network_interface_backend_address_pool_association resource.

Similar to the azurerm_lb_backend_address_pool resource - Azure allows adding a VM to a LB's Backend Address Pool asynchronously during creation but during deletion the ordering matters unfortunately.