Terraform-provider-azurerm: Kubernetes cluster: cannot index effective_outbound_ips array during plan

[emeka]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.26

• provider.azurerm v2.13.0

Affected Resource(s)

• azurerm_kubernetes_cluster

Terraform Configuration Files

provider "azurerm" {
  version = "=2.13.0"
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "issue2-example"
  location = "westeurope"
}

# Networking
resource "azurerm_virtual_network" "example" {
  name                = "issue2-example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  address_space       = ["192.168.0.0/16"]
}

resource "azurerm_subnet" "example" {
  name                 = "issue2-example"
  resource_group_name  = azurerm_resource_group.example.name
  address_prefixes     = ["192.168.1.0/24"]
  virtual_network_name = azurerm_virtual_network.example.name

  service_endpoints = ["Microsoft.Sql"]
}

# Kubernetes example
resource "azurerm_kubernetes_cluster" "example" {
  name                = "issue2-example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "issue2-example"

  default_node_pool {
    name           = "issue2example"
    node_count     = 2
    vm_size        = "Standard_B2s"
    vnet_subnet_id = azurerm_subnet.example.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "kubenet"
    load_balancer_sku = "Standard"
  }
}

data "azurerm_public_ip" "example" {
  name                = azurerm_kubernetes_cluster.example.network_profile.0.load_balancer_profile.0.effective_outbound_ips.0
  resource_group_name = azurerm_resource_group.example.name
}

output "cluster_egress_ip" {
  value = data.azurerm_public_ip.example.ip_address
}

Debug Output

https://gist.github.com/emeka/d3a986280019637d8ee481defd6335fc

Panic Output

N/A

Expected Behavior

Expecting the first and only public_ip name to be returned in the output

Actual Behavior

effective_outbound_ips is not recognized as an array during plan and effective_outbound_ips.0 returns an error. As a result, we cannot get the data for the corresponding azurerm_public_ip resource.

Steps to Reproduce

1. terraform init
2. terraform plan

Important Factoids

If your remove the index .0 and run terraform plan again, then you get the following error as expected:

Error: Incorrect attribute value type

  on main.tf line 53, in data "azurerm_public_ip" "example":
  53:   name                = azurerm_kubernetes_cluster.example.network_profile.0.load_balancer_profile.0.effective_outbound_ips

Inappropriate value for attribute "name": string required.

References

N/A

• #0000

[pb6]

Experiencing this too, would be nice if this worked. Would be useful to use that IP with firewall rules later.

[c4m4]

I did one step forward, but the problem now is the resource group

data "azurerm_public_ip" "example" {
  name                = tolist(azurerm_kubernetes_cluster.k8s.network_profile.0.load_balancer_profile.0.effective_outbound_ips)[0]
  resource_group_name = azurerm_kubernetes_cluster.k8s.node_resource_group
}

[c4m4]

I had to workaround this using this:

+resource "azurerm_public_ip" "k8s-outbound-ip" {
+  name                = "${var.name}-outbound-ip"
+  resource_group_name = var.resource_group_name
+  location            = var.location
+  allocation_method   = "Static"
+  sku                 = "standard"
+
+  tags = {
+    environment = local.environment
+  }
+}
+
 resource "azurerm_kubernetes_cluster" "k8s" {
   name                = var.name
   location            = var.location
@@ -17,6 +29,9 @@ resource "azurerm_kubernetes_cluster" "k8s" {
   network_profile {
     network_plugin    = "kubenet"
     load_balancer_sku = "standard"
+    load_balancer_profile {
+      outbound_ip_address_ids = [azurerm_public_ip.k8s-outbound-ip.id]
+    }
   }

[trotman23]

The effective_outbound_ips attribute is an array of azure resource ids.
(e.g. /subscriptions/<subscription_id>/resourceGroups/MC_resource_group/providers/Microsoft.Network/publicIPAddresses/resource_name )

The azurerm_public_ip data source needs the public IP name and resource_group as arguments.

I was able to get this to work (without creating an azurerm_public_ip resource myself), but IMO there should be an easier way to do it.

version:

~ terraform -version
Terraform v0.12.26
+ provider.azurerm v2.13.0

terraform:

provider "azurerm" {
  version = "=2.13.0"
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "centralus"
}

# Networking
resource "azurerm_virtual_network" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  address_space       = ["192.168.0.0/16"]
}

resource "azurerm_subnet" "example" {
  name                 = "example"
  resource_group_name  = azurerm_resource_group.example.name
  address_prefixes     = ["192.168.1.0/24"]
  virtual_network_name = azurerm_virtual_network.example.name

  service_endpoints = ["Microsoft.Sql"]
}

# Kubernetes example
resource "azurerm_kubernetes_cluster" "example" {
  name                = "example"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "example"

  default_node_pool {
    name           = "example"
    node_count     = 2
    vm_size        = "Standard_B2s"
    vnet_subnet_id = azurerm_subnet.example.id
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin    = "kubenet"
    load_balancer_sku = "Standard"
  }
}

data "azurerm_public_ip" "example" {
  name                = reverse(split("/", tolist(azurerm_kubernetes_cluster.example.network_profile.0.load_balancer_profile.0.effective_outbound_ips)[0]))[0]
  resource_group_name = azurerm_kubernetes_cluster.example.node_resource_group
}

output "cluster_egress_ip" {
  value = data.azurerm_public_ip.example.ip_address
}

plan output

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.azurerm_public_ip.example will be read during apply
  # (config refers to values not yet known)
 <= data "azurerm_public_ip" "example"  {
      + allocation_method       = (known after apply)
      + domain_name_label       = (known after apply)
      + fqdn                    = (known after apply)
      + id                      = (known after apply)
      + idle_timeout_in_minutes = (known after apply)
      + ip_address              = (known after apply)
      + ip_version              = (known after apply)
      + location                = (known after apply)
      + name                    = (known after apply)
      + resource_group_name     = (known after apply)
      + reverse_fqdn            = (known after apply)
      + sku                     = (known after apply)
      + zones                   = (known after apply)

      + timeouts {
          + read = (known after apply)
        }
    }

  # azurerm_kubernetes_cluster.example will be created
  + resource "azurerm_kubernetes_cluster" "example" {
      + dns_prefix              = "example"
      + fqdn                    = (known after apply)
      + id                      = (known after apply)
      + kube_admin_config       = (known after apply)
      + kube_admin_config_raw   = (sensitive value)
      + kube_config             = (known after apply)
      + kube_config_raw         = (sensitive value)
      + kubelet_identity        = (known after apply)
      + kubernetes_version      = (known after apply)
      + location                = "centralus"
      + name                    = "example"
      + node_resource_group     = (known after apply)
      + private_cluster_enabled = (known after apply)
      + private_fqdn            = (known after apply)
      + private_link_enabled    = (known after apply)
      + resource_group_name     = "example"

      + addon_profile {
          + aci_connector_linux {
              + enabled     = (known after apply)
              + subnet_name = (known after apply)
            }

          + azure_policy {
              + enabled = (known after apply)
            }

          + http_application_routing {
              + enabled                            = (known after apply)
              + http_application_routing_zone_name = (known after apply)
            }

          + kube_dashboard {
              + enabled = (known after apply)
            }

          + oms_agent {
              + enabled                    = (known after apply)
              + log_analytics_workspace_id = (known after apply)
              + oms_agent_identity         = (known after apply)
            }
        }

      + default_node_pool {
          + max_pods        = (known after apply)
          + name            = "issue2example"
          + node_count      = 2
          + os_disk_size_gb = (known after apply)
          + type            = "VirtualMachineScaleSets"
          + vm_size         = "Standard_B2s"
          + vnet_subnet_id  = (known after apply)
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = "SystemAssigned"
        }

      + network_profile {
          + dns_service_ip     = (known after apply)
          + docker_bridge_cidr = (known after apply)
          + load_balancer_sku  = "Standard"
          + network_plugin     = "kubenet"
          + network_policy     = (known after apply)
          + outbound_type      = "loadBalancer"
          + pod_cidr           = (known after apply)
          + service_cidr       = (known after apply)

          + load_balancer_profile {
              + effective_outbound_ips    = (known after apply)
              + managed_outbound_ip_count = (known after apply)
              + outbound_ip_address_ids   = (known after apply)
              + outbound_ip_prefix_ids    = (known after apply)
            }
        }

      + role_based_access_control {
          + enabled = (known after apply)

          + azure_active_directory {
              + client_app_id     = (known after apply)
              + server_app_id     = (known after apply)
              + server_app_secret = (sensitive value)
              + tenant_id         = (known after apply)
            }
        }

      + windows_profile {
          + admin_password = (sensitive value)
          + admin_username = (known after apply)
        }
    }

  # azurerm_resource_group.example will be created
  + resource "azurerm_resource_group" "example" {
      + id       = (known after apply)
      + location = "centralus"
      + name     = "example"
    }

  # azurerm_subnet.example will be created
  + resource "azurerm_subnet" "example" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "192.168.1.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "example"
      + resource_group_name                            = "example"
      + service_endpoints                              = [
          + "Microsoft.Sql",
        ]
      + virtual_network_name                           = "example"
    }

  # azurerm_virtual_network.example will be created
  + resource "azurerm_virtual_network" "example" {
      + address_space       = [
          + "192.168.0.0/16",
        ]
      + guid                = (known after apply)
      + id                  = (known after apply)
      + location            = "centralus"
      + name                = "example"
      + resource_group_name = "example"
      + subnet              = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Apply output (redacted):

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

cluster_egress_ip = <redacted>

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!