I use Terraform to create AKS clusters with advanced networking. The Terraform code is similar to the example provided on the official documentation. As part of my Terraform, I also provision a static IP address which I plan to as the static IP address for my Ingress.
I can successfully create the cluster and connect to it with kubectl
.
The cluster isn't configured with the Azure Ingress. Instead, I install the nginx Ingress. I use helm to template the resources and submit them to the API:
helm template nginx-ingress \
--set controller.service.loadBalancerIP="<insert IP here>" \
--set rbac.create=true --name ingress | kubectl apply -f -
When the Ingress is provisioned, Azure creates a load balancer which is attached to the node pool. Please note that the load balancer is created by AKS and Terraform doesn't that the resource exists.
I can curl the static IP address and verify that the ingress is working. All fair and square.
When I decide to dispose of my cluster with terraform destroy
, I experience errors.
Terraform can't delete the static IP address because there's a resource (which isn't aware of) holding a reference to it. It's the Load balancer created by AKS!
I have to manually delete the LB and then I can finally destroy the cluster.
Ideally, I don't want resources such as a load balancer to be created outside of a normal Terraform lifecycle. The infrastructure should be brought up and destroyed with a single command: terraform apply
and terraform destroy
.
I thought I could customise things a little and create a load balancer in Terraform connected to the agent pool. The plan was:
With the above setup, I can deploy AKS and an LB as well as provisioning static IP addresses and Application Gateways linked to it. The infrastructure is managed in a single location: Terraform.
When I deploy the ingress, I have to pay attention that the port exposed by the nodes in the agent pool is the same port used by the Load Balancers.
Unfortunately to provision the LB with Terraform I need to link the NIC which is created by AKS. The resource is not exposed as an attribute by AKS in Terraform so there's no way to connect to the agent pool.
Back to square one.
Any thoughts on how one could have the infrastructure managed by Terraform in AKS?
Hi, maybe I don't understand the context of this problem, but wouldn't this example Terraform configuration solve your problem?
# Set default name prefix
variable "name_prefix" {
default = "k8s-cluster"
}
# Set default location
variable "location" {
default = "westeurope"
}
# Create Resource Group
resource "azurerm_resource_group" "aks" {
name = "${var.name_prefix}-rg"
location = "${var.location}"
}
# Create Azure AD Application for Service Principal
resource "azurerm_azuread_application" "aks" {
name = "${var.name_prefix}-sp"
}
# Create Service Principal
resource "azurerm_azuread_service_principal" "aks" {
application_id = "${azurerm_azuread_application.aks.application_id}"
}
# Generate random string to be used for Service Principal Password
resource "random_string" "password" {
length = 32
special = true
}
# Create Service Principal password
resource "azurerm_azuread_service_principal_password" "aks" {
end_date = "2299-12-30T23:00:00Z" # Forever
service_principal_id = "${azurerm_azuread_service_principal.aks.id}"
value = "${random_string.password.result}"
}
# Create managed Kubernetes cluster (AKS)
resource "azurerm_kubernetes_cluster" "aks" {
name = "${var.name_prefix}-aks"
location = "${azurerm_resource_group.aks.location}"
resource_group_name = "${azurerm_resource_group.aks.name}"
dns_prefix = "${var.name_prefix}"
kubernetes_version = "1.11.3"
agent_pool_profile {
name = "linuxpool"
count = 1
vm_size = "Standard_DS2_v2"
os_type = "Linux"
os_disk_size_gb = 30
}
service_principal {
client_id = "${azurerm_azuread_application.aks.application_id}"
client_secret = "${azurerm_azuread_service_principal_password.aks.value}"
}
}
# Initialize Helm (and install Tiller)
provider "helm" {
install_tiller = true
kubernetes {
host = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
client_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
client_key = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
}
}
# Create Static Public IP Address to be used by Nginx Ingress
resource "azurerm_public_ip" "nginx_ingress" {
name = "nginx-ingress-pip"
location = "${azurerm_kubernetes_cluster.aks.location}"
resource_group_name = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
public_ip_address_allocation = "static"
domain_name_label = "${var.name_prefix}"
}
# Add Kubernetes Stable Helm charts repo
resource "helm_repository" "stable" {
name = "stable"
url = "https://kubernetes-charts.storage.googleapis.com"
}
# Install Nginx Ingress using Helm Chart
resource "helm_release" "nginx_ingress" {
name = "nginx-ingress"
repository = "${helm_repository.stable.metadata.0.name}"
chart = "nginx-ingress"
set {
name = "rbac.create"
value = "false"
}
set {
name = "controller.service.externalTrafficPolicy"
value = "Local"
}
set {
name = "controller.service.loadBalancerIP"
value = "${azurerm_public_ip.nginx_ingress.ip_address}"
}
}
@joakimhellum-in clever.
I didn't think about running the helm provider within the same terraform script. The only issue with that would be the tiller, which I'd rather not install.
Anyway, nice workaround ๐
I'm facing a very similar issue to Daniel, however Joakim's workaround doesn't seem to fix things.
I've provisioned a new public IP address as as resource. When installing nginx-ingress
with helm, I'm setting controller.service.loadBalancerIP
to the new IP address as mentioned above.
However the ingress seems to ignore this setting; instead the ingress service is assigned a different public IP address (which is the one automatically created by Azure).
This is an issue because, as mentioned earlier, I now have state which exists that was not provisioned by Terraform making it difficult to destroy. Additionally, it becomes tricky to create for example firewall rules for the ingress IP address using terraform. How do I retrieve the IP address that Azure creates for the ingress?
Any advice would be very much appreciated.
I don't want to/can't use the Tiller, so I'm still looking for options too.
The current workarounds are not ideal:
terraform apply
a second time.The other option is to use ACS-engine. ACS-engine is the engine behind AKS. It creates the ARM templates necessary to provision a Kubernetes cluster (and AKS). You can customise a lot of things, but as far as I understand you can't provision node pools in advanced. So even with the extra flexibility of ACS-engine, we can't use terraform to create a load balancer on top of the existing VMSS/VMAS.
I opened an issue on ACS-engine too, but without much luck.
As far as I understand the limitation is not in Terraform or the ARM template, but in the Azure API that doesn't support any of the above.
Only MS can save us ๐
@danielepolencic I believe #1837 might help you with the destroy issue, we had a similar problem recently.
Thanks @andrey-moor .
I just tested it and I think this is the best workaround I've seen. Thanks for that!
The last piece of the puzzle is to connect an Application Gateway to an ILB. At the moment the workaround works because the LB and the IP address are public. However, I don't want to expose the IP address or the load balancer to the internet. I'd like the traffic to be routed through the AG.
Unfortunately, the workaround seems to work only for public IP addresses ๐จ
Are you exposing your LB directly to the internet?
Yeah in our case the load balancer frontend static IP address was public. I haven't tried this with ILB, but I think you can try to deploy private static IP address to another (non-managed) resource group to the same subnet where you have AKS. Have you tried that?
I don't think you can select a private IP address in the same way that you can for a public IP address.
I'm facing a very similar issue to Daniel, however Joakim's workaround doesn't seem to fix things.
I've provisioned a new public IP address as as resource. When installing
nginx-ingress
with helm, I'm settingcontroller.service.loadBalancerIP
to the new IP address as mentioned above.However the ingress seems to ignore this setting; instead the ingress service is assigned a different public IP address (which is the one automatically created by Azure).
This is an issue because, as mentioned earlier, I now have state which exists that was not provisioned by Terraform making it difficult to destroy. Additionally, it becomes tricky to create for example firewall rules for the ingress IP address using terraform. How do I retrieve the IP address that Azure creates for the ingress?
Any advice would be very much appreciated.
I had the same issue and just managed to resolve it. The trick is in formatting of values in helm. Ensure that service section where the IP specified is nested in controller section, not on the same level with it.
๐
Thanks for opening this issue - apologies for the delayed response here.
From Terraform's side unfortunately since the AKS API doesn't expose this functionality this isn't something that we're able to support at this time. I'd suggest instead opening an issue on the AKS Repository where a member of the AKS Team should be able to comment around exposing this functionality - however since this isn't something we're able to support at this time I'm going to close this issue for the moment.
Thanks!
I'm going to lock this issue because it has been closed for _30 days_ โณ. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error ๐ค ๐ , please reach out to my human friends ๐ [email protected]. Thanks!
Most helpful comment
Hi, maybe I don't understand the context of this problem, but wouldn't this example Terraform configuration solve your problem?