Terraform-provider-azurerm: [azurerm_kubernetes_cluster] load balancer profile outbound ips validation errors

[tscully49]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.16

• provider.azurerm v1.42.0

Affected Resource(s)

• azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "aks_cluster" {
  name       = "uat-ops-cluster"
  location   = var.location
  dns_prefix = "pkpuatops"

  resource_group_name = var.resource_group_name
  node_resource_group = "${var.resource_group_name}-aks-nodes"

  linux_profile {
    admin_username = "***"
    ssh_key {
      key_data = "***"
    }
  }

  kubernetes_version = "1.13.12"

  addon_profile {
    http_application_routing {
      enabled = false
    }
  }

  default_node_pool {
    name            = "appagentpool"
    node_count      = 1
    vm_size         = "Standard_DS2"
    os_disk_size_gb = 40
    vnet_subnet_id  = azurerm_subnet.kubesubnet.id
  }

  service_principal {
    client_id     = "***"
    client_secret = "***"
  }

  network_profile {
    network_plugin     = "azure"
    dns_service_ip     = "15.0.0.10"
    docker_bridge_cidr = "172.17.0.1/16"
    service_cidr       = "15.0.0.0/16"
    load_balancer_sku  = "Standard"

    load_balancer_profile {
      outbound_ip_address_ids = ["/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/publicIP/testIpAddress"]
    }
  }

  tags = var.tags
}

Debug Output

Snippet of debug logs:

2020-02-11T11:32:13.021-0600 [DEBUG] plugin.terraform-provider-azurerm_v1.42.0_x4: {
2020-02-11T11:32:13.021-0600 [DEBUG] plugin.terraform-provider-azurerm_v1.42.0_x4:   "code": "InvalidLoadBalancerProfileOutboundIPs",
2020-02-11T11:32:13.021-0600 [DEBUG] plugin.terraform-provider-azurerm_v1.42.0_x4:   "message": "Error in validating load balancer profile outbound IPs",
2020-02-11T11:32:13.021-0600 [DEBUG] plugin.terraform-provider-azurerm_v1.42.0_x4:   "target": "networkProfile.loadBalancerProfile.outboundIPs"
2020-02-11T11:32:13.021-0600 [DEBUG] plugin.terraform-provider-azurerm_v1.42.0_x4:  }
2020/02/11 11:32:13 [DEBUG] azurerm_kubernetes_cluster.aks_cluster: apply errored, but we're indicating that via the Error pointer rather than returning it: Error creating Managed Kubernetes Cluster "uat-ops-cluster" (Resource Group "pkp-uat-ops"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfileOutboundIPs" Message="Error in validating load balancer profile outbound IPs" Target="networkProfile.loadBalancerProfile.outboundIPs"
2020/02/11 11:32:13 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/02/11 11:32:13 [TRACE] EvalMaybeTainted: azurerm_kubernetes_cluster.aks_cluster encountered an error during creation, so it is now marked as tainted
2020/02/11 11:32:13 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: Error creating Managed Kubernetes Cluster "uat-ops-cluster" (Resource Group "pkp-uat-ops"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfileOutboundIPs" Message="Error in validating load balancer profile outbound IPs" Target="networkProfile.loadBalancerProfile.outboundIPs"
2020/02/11 11:32:13 [ERROR] <root>: eval: *terraform.EvalSequence, err: Error creating Managed Kubernetes Cluster "uat-ops-cluster" (Resource Group "pkp-uat-ops"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfileOutboundIPs" Message="Error in validating load balancer profile outbound IPs" Target="networkProfile.loadBalancerProfile.outboundIPs"
2020-02-11T11:32:13.340-0600 [DEBUG] plugin: plugin process exited: path=/Development/roadrunner/operations/terraform/environments/tmp-uat-ops/.terraform/plugins/darwin_amd64/terraform-provider-azurerm_v1.42.0_x4 pid=666
2020-02-11T11:32:13.340-0600 [DEBUG] plugin: plugin exited

Panic Output

Expected Behavior

The expected behavior is the cluster to be created with the outbound IP addresses without error

Actual Behavior

Error: Error creating Managed Kubernetes Cluster "uat-ops-cluster" (Resource Group "pkp-uat-ops"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfileOutboundIPs" Message="Error in validating load balancer profile outbound IPs" Target="networkProfile.loadBalancerProfile.outboundIPs"

  on main.tf line 37, in resource "azurerm_kubernetes_cluster" "aks_cluster":
  37: resource "azurerm_kubernetes_cluster" "aks_cluster" {

Steps to Reproduce

1. terraform apply with the above resource

Important Factoids

References

Using config from https://www.terraform.io/docs/providers/azurerm/r/kubernetes_cluster.html

[mariojacobo]

Most likely what's happening is that your public IP has a "basic" SKU. I was getting the same error until I deployed with a "Standard" SKU.

[tscully49]

Thanks for the response! I double-checked and am seeing that the public IP prefix and the public IP address I created from the prefix both have the "standard" SKU. For what it's worth I receive the same error when I tried to use the public IP prefix option for the cluster as well with a prefix containing the "standard" SKU as well.

[tscully49]

I think I found my problem. I wasn't using the correct id for the public IP address. I was able to make it work gathering the IP address id via the az CLI. I also found my problem with the prefix was I already had allocated IP addresses in the prefix, which the outbound load balancer for AKS does not like. I will go ahead and close the issue. Thanks!

[hasanertenli]

resource "azurerm_public_ip" "public_ip" {
  name                = "${var.cluster_name}publicIP"
  resource_group_name = var.rg_name
  location            = var.location
  allocation_method   = "Static"
}

network_profile {
    network_plugin = var.network_profile_plugin
    # Required for availability zones
    load_balancer_sku = var.load_balancer_sku

    load_balancer_profile {
      outbound_ip_address_ids = [ "${azurerm_public_ip.public_ip.id}" ]
    }

Hi. I created new public IP and use this public IP's id for kubernetes cluster. I'm getting the same error. Can you tell me where i made a mistake? Thanks!

[mariojacobo]

@hasanertenli - I believe it's the same thing, the IP needs to be Standard sku. Add the "sku" field to the public IP resource definition:

resource "azurerm_public_ip" "public_ip" {
  name                = "${var.cluster_name}publicIP"
  resource_group_name = var.rg_name
  location            = var.location
  allocation_method   = "Static"
  sku                           = "Standard"
}

network_profile {
    network_plugin = var.network_profile_plugin
    # Required for availability zones
    load_balancer_sku = var.load_balancer_sku

    load_balancer_profile {
      outbound_ip_address_ids = [ "${azurerm_public_ip.public_ip.id}" ]
    }

Try this sample (replace service principal ID and Secret accordingly): https://gist.github.com/mariojacobo/01708a83ce6db3056ebc6439a100cab2

[hasanertenli]

@mariojacobo it worked. Thank you.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!