Terraform-provider-helm: Unable to set up nginx Ingress with advanced networking in AKS

Created on 2 Nov 2018 · 7Comments · Source: hashicorp/terraform-provider-helm

We are trying to use the advanced networking option with nginx ingress controller but getting the following error mentioned below. I am new to terraform and AKS :). i am able to set up properly when i just use the basic network. There was a issue reported #2167 and #2160 to remove the NSG which might block the ingress. I have done that also but still get this below error. This has been a blocker for us. Please help us to resolve ASAP

The error is

1 error(s) occurred:
* helm_release.nginx_ingress: 1 error(s) occurred:
* helm_release.nginx_ingress: rpc error: code = Unknown desc = release nginx-ingress failed: timed out waiting for the condition

Following is the code to create the AKS

# Set default name prefix
variable "name_prefix" {
  default = "k8s-cluster"
}

# Set default location
variable "location" {
  default = "westeurope"
}

# Create Resource Group
resource "azurerm_resource_group" "aksrg" {
  name     = "${var.name_prefix}-rg"
  location = "${var.location}"
}

# Create Azure AD Application for Service Principal
resource "azurerm_azuread_application" "aksad" {
  name = "${var.name_prefix}-sp"
}

# Create Service Principal
resource "azurerm_azuread_service_principal" "akssp" {
  application_id = "${azurerm_azuread_application.aksad.application_id}"
}

# Generate random string to be used for Service Principal Password
resource "random_string" "password" {
  length  = 32
  special = true
}

# Create Service Principal password
resource "azurerm_azuread_service_principal_password" "akspwd" {
  end_date             = "2299-12-30T23:00:00Z"                        # Forever
  service_principal_id = "${azurerm_azuread_service_principal.akssp.id}"
  value                = "${random_string.password.result}"
}



resource "azurerm_virtual_network" "vnet" {
  name                = "${var.name_prefix}-vnet"
  location            = "${azurerm_resource_group.aksrg.location}"
  resource_group_name = "${azurerm_resource_group.aksrg.name}"
  address_space       = ["10.1.0.0/16"]
}

resource "azurerm_subnet" "subnet" {
  name                      = "${var.name_prefix}-subnet"
  resource_group_name       = "${azurerm_resource_group.aksrg.name}"
  address_prefix            = "10.1.0.0/24"
  virtual_network_name      = "${azurerm_virtual_network.vnet.name}"
}


# Create managed Kubernetes cluster (AKS)
resource "azurerm_kubernetes_cluster" "aks" {
  name                = "${var.name_prefix}-aks"
  location            = "${azurerm_resource_group.aksrg.location}"
  resource_group_name = "${azurerm_resource_group.aksrg.name}"
  dns_prefix          = "${var.name_prefix}"
  kubernetes_version  = "1.11.3"

  agent_pool_profile {
    name            = "linuxpool"
    count           = 1
    vm_size         = "Standard_DS2_v2"
    os_type         = "Linux"
    os_disk_size_gb = 30
    # Required for advanced networking
      vnet_subnet_id = "${azurerm_subnet.subnet.id}"

  }

  service_principal {
    client_id     = "${azurerm_azuread_application.aksad.application_id}"
    client_secret = "${azurerm_azuread_service_principal_password.akspwd.value}"
  }

  network_profile {
      network_plugin     = "azure"
  }



  tags {
    Environment = "Production"
  }
}

# Initialize Helm (and install Tiller)
provider "helm" {
  install_tiller = true

  kubernetes {
    host                   = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
  }
}

# Create Static Public IP Address to be used by Nginx Ingress
resource "azurerm_public_ip" "nginx_ingress" {
  name                         = "nginx-ingress-pip"
  location                     = "${azurerm_kubernetes_cluster.aks.location}"
  resource_group_name          = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
  public_ip_address_allocation = "static"
  domain_name_label            = "${var.name_prefix}"
}

# Add Kubernetes Stable Helm charts repo
resource "helm_repository" "stable" {
  name = "stable"
  url  = "https://kubernetes-charts.storage.googleapis.com"
}

# Install Nginx Ingress using Helm Chart
resource "helm_release" "nginx_ingress" {
  name       = "nginx-ingress"
  repository = "${helm_repository.stable.metadata.0.name}"
  chart      = "nginx-ingress"

  set {
    name  = "rbac.create"
    value = "false"
  }

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }

  set {
    name  = "controller.service.loadBalancerIP"
    value = "${azurerm_public_ip.nginx_ingress.ip_address}"
  }
}

output "subnet_id" {
  value = "${azurerm_kubernetes_cluster.aks.agent_pool_profile.0.vnet_subnet_id}"
}

output "network_plugin" {
  value = "${azurerm_kubernetes_cluster.aks.network_profile.0.network_plugin}"
}

output "service_cidr" {
  value = "${azurerm_kubernetes_cluster.aks.network_profile.0.service_cidr}"
}

output "dns_service_ip" {
  value = "${azurerm_kubernetes_cluster.aks.network_profile.0.dns_service_ip}"
}

output "docker_bridge_cidr" {
  value = "${azurerm_kubernetes_cluster.aks.network_profile.0.docker_bridge_cidr}"
}

output "pod_cidr" {
  value = "${azurerm_kubernetes_cluster.aks.network_profile.0.pod_cidr}"
}

Source

girishgouda

Most helpful comment

I have no experience with AKS here but you seem to hit a timeout, from experience with other providers I think it could be waiting on something to be created by Azure. Could you try setting wait = false in resource "helm_release" "nginx_ingress"? This will make Helm create all resources and not wait for them to be fully up and running.

meyskens on 9 Nov 2018

👍3 🚀1 🎉1

All 7 comments

We are trying to set up production grade AKS set up. Let us also know if there any changes to be done for the scripts

girishgouda on 2 Nov 2018

hi @girishgouda

Thanks for opening this issue :)

From what I can see above this appears to be an error message coming from the Helm Provider (and looking at the AKS Configuration this appears to be correct) - as such I'm going to transfer this issue over to that repository.

Thanks!