Terraform-provider-kubernetes: Kubernetes Ingress on Azure Kubernetes Service Doesn't Wait for Public IP

[MattMencel]

Terraform Version

Terraform v0.12.9

• provider.azurerm v1.34.0
• provider.helm v0.10.2
• provider.http v1.1.1
• provider.kubernetes v1.9.0
• provider.null v2.1.2
• provider.random v2.2.0
• provider.tls v2.1.0

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_ingress

Terraform Configuration Files

resource "kubernetes_ingress" "vault_ingress" {
  metadata {
    name = "vault-ui"
    annotations = {
      "kubernetes.io/ingress.class" = "addon-http-application-routing"
    }
  }

  spec {
    rule {
      host = "vault-ui.${azurerm_kubernetes_cluster.aks_cluster.addon_profile[0].http_application_routing[0].http_application_routing_zone_name}"
      http {
        path {
          backend {
            service_name = "vault-ui"
            service_port = 8200
          }

          path = "/"
        }
      }
    }

    tls {
      secret_name = "vault-ui-tls"
    }
  }
}

resource "azurerm_postgresql_firewall_rule" "allow_vault" {
  name                = "vault"
  resource_group_name = "${azurerm_resource_group.vaultdemo.name}"
  server_name         = "${azurerm_postgresql_server.psql_server.name}"
  start_ip_address    = "${kubernetes_ingress.vault_ingress.load_balancer_ingress[0].ip}"
  end_ip_address      = "${kubernetes_ingress.vault_ingress.load_balancer_ingress[0].ip}"
}

Debug Output

https://gist.github.com/MattMencel/0948154a1835b8fcf4dcf5045add2ee8

Expected Behavior

I expected the Ingress public IP address to be available to use in a postgres firewall rule.

If I run kubectl get ingress -w during the terraform apply, the public IP takes up to a minute to become available and appear in the ingress.

Actual Behavior

Terraform fails immediately after deploying the Ingress, when it tries to create the firewall rule.

Error: Invalid index

on DB.tf line 41, in resource "azurerm_postgresql_firewall_rule" "allow_vault":
41: start_ip_address = "${kubernetes_ingress.vault_ingress.load_balancer_ingress[0].ip}"
|----------------
| kubernetes_ingress.vault_ingress.load_balancer_ingress is empty list of object

The given key does not identify an element in this collection value.

If I wait a minute or two and re-run terraform apply it works fine because the public IP appears on the Ingress by then.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

With Terraform

1. Deploy an AKS resource with http_application_routing enabled.
2. Deploy a Service and kubernetes_ingress... Azure should provision the public IP automatically and attach it to the Ingress.
3. Deploy a resource or output that references the public IP from the kubernetes_ingress resource.

[brettcurtis]

Same issue, I'm just wanting to use it for an A record in Google Cloud DNS.

[MMBRitter]

Same issue for me while tying to use the ip attribute of a GKE kubernetes_ingress resource to set an A record with a google_dns_record_set resource. Apply is not handling the asynchronous call and returning success before the external IP is propagated, causing failure until the attribute is present. Trying to find a workaround but would love to see this fixed to keep my deployment process clean and straightforward.

[jmonson]

I have a similar issue in AWS using "kubernetes.io/ingress.class" = "alb". I have a aws_route53_record being created after the ingress that requires .load_balancer_ingress[0].hostname.

[cregev]

Any workarounds ?

[MMBRitter]

@cregev For my specific use case I have switched to using a reserved static IP for my DNS record using a google_compute_global_address resource which is not considered created until the address attribute is available. This is not viable for me in production as we have a limited number of static IP addresses available to reserved. If this issue doesn't get any attention in the next few weeks I'll start exploring the possibility of a null resource using a local-exec provisioner to run a script that checks for the presence of the ip address on the ingress using google kubernetes engine api calls but its kind of hacky and I would prefer not to. There are many terraform resources that successfully handle asynchronous api's like this and the method for doing so is explained in this provider developer guide. https://www.terraform.io/docs/extend/resources/retries-and-customizable-timeouts.html

The fix is straightforward but requires this issue to get attention. I'm sure the developers of this provider are very busy and have to prioritize which issues they pick up so it might be a while before we see this being worked on unless there is more demand for it to be resolved.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!