Terraform-provider-kubernetes: Apply annotations / labels to existing resources

[jon-walton]

Terraform Version

Terraform v0.12.15

• provider.kubernetes v1.9.0

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_namespace
• most other kubernetes resources

Terraform Configuration Files

resource "kubernetes_namespace" "kube_system_labels" {
  metadata {
    name        = "kube-system"
    annotations = {
      "linkerd.io/inject": "disabled"
    }
    labels      = {
      "config.linkerd.io/admission-webhooks": "disabled"
    }
  }
}

Debug Output

Error: namespaces "kube-system" already exists

  on ..\modules\aks\kubernetes.tf line 35, in resource "kubernetes_namespace" "kube_system_labels":
  35: resource "kubernetes_namespace" "kube_system_labels" {

Expected Behavior

I would like to apply annotations / labels to an existing namespace (kube-system). However I cannot use the kubernetes_namespace resource because the namespace already exists.

While I could import the namespace, or use kubectl to apply the annotation / label, that feels fragile, I also prefer not to call out to other executables within my terraform config

I would like to propose 2 new resources

• kubernetes_annotation
• kubernetes_label

which can be applied to any kubernetes resource

Actual Behavior

Error due to the namespace already existing (which is actually also expected)

[pdecat]

Somewhat related to #238.

[atikhono]

Hi, I'm really looking forward to this feature. It is particularly useful when you need to configure Azure's OMS agent for scraping Prometheus metrics from kube-system Pods ini AKS: set prometheus.io/scrape: "true" annotation on existing coredns Deployment.

[mrpaws]

Does using the namespace data source to lookup the kubernetes namespace work? If known in advance it could work?

[bryankaraffa]

👍 on this... I tried getting around using kubernetes_all_namespaces datasources and contains() to check if namespace exists, and only create if not [with the labels provided]... but it only mitigates/prevents running into that namespace doesn't exist error if the namespace exists already during plan. It doesn't allow adding labels to an existing namespace like kube-system, or if two modules attempt to create the same namespace I presume it would fail as well:

data "kubernetes_all_namespaces" "allns" {}

resource "kubernetes_namespace" "release_namespace" {
  # Only create namespace if namespace from var.release_namespace does not exist
  count = contains(data.kubernetes_all_namespaces.allns, var.release_namespace) ? 0 : 1
  metadata {
    name   = var.release_namespace
    labels = var.release_namespace_labels
  }
}

Agree an additional resource like kubernetes_labels or kubernetes_annotations would be helpful. AWS provider has a concept where certain resources have a *_tag resource as well to manage tags

[den-is]

The same thing is required for configuring CNI Add-on for AWS EKS using IAM Role for Service Account (IRSA).
You need to annotate the aws-node service account with the newly created dedicated IAM Role ARN

[theFong]

Wanted to add additional uses for this feature. I was looking to add annotation to services after running into issues deploying ory/hydra via helm behind a GCE Load Balancer/ingress and hoped there was a terraform resource for doing so. Current work around uses kubectl but definitely fragile/stop gap.

[aareet]

We'll investigate this when investigating https://github.com/hashicorp/terraform-provider-kubernetes/issues/723

[gingerhelp]

For anybody looking for a way to accomplish this while awaiting the proposed enhancement, here is what has worked well for me:

resource "null_resource" "default-namespace" {
  provisioner "local-exec" {
  command = <<EOT
set -e
mkdir -p ~/.kube/
mkdir -p $HOME/bin
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.19.7/bin/linux/amd64/kubectl
chmod +x kubectl
cp ./kubectl $HOME/bin/
export PATH=$PATH:$HOME/bin
az aks get-credentials --name ${var.env_name}-k8s --overwrite-existing --resource-group ${azurerm_resource_group.pyp.name}
kubectl label namespaces default role=platform
kubectl annotate namespaces default test=123
EOT
  }
}