Terraform-provider-kubernetes: Variables not correctly interpreted when configuring labels on a namespace

[rolandkool]

When working with the kubernetes namespace resource, I ran into an issue where it seems that variables that are fetched from remote state are not interpreted and seem to be treated as literals.
Only the labels seem to be affected, the name property does not have an issue.

Terraform Version

This is with terraform version 0.10-rc1

Affected Resource(s)

The labels within the kubernetes namespace

Terraform Configuration Files

data "terraform_remote_state" "globalvars" {
  backend = "local"
  config {
    path = "path/to/terraform.tfstate"
  }
}
resource "kubernetes_namespace" "k8s_namespace" {
  metadata {
    labels {
      application = "${data.terraform_remote_state.globalvars.application_name}"
    }
    name = "${data.terraform_remote_state.globalvars.application_name}"
  }
}

The remote state is filled through:

output "application_name" { value = "myapp" }

Debug Output

module.kubernetes-namespace.kubernetes_namespace.k8s_namespace: metadata.0.labels ("${data.terraform_remote_state.globalvars.application_name}") a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

Expected Behavior

Label should be set to application = myapp

Actual Behavior

Get an error stating I try to create an invalid label (see debug output)

Steps to Reproduce

First run code that outputs variable in a state file
Then create a tf file that includes the kubernetes namespace as a module

Important Factoids

If you replace the variable with a local one (${var.application_name}) then there is no issue.
If you set the label to a hardcoded value, you'll see it works with interpolation for name.
I also tried to wrap the variable in a function (substr/length/chomp) but the error remains.
Could it be that it doesn't recognize the remote state variable as a valid variable?

References

The globalvars trick as described in https://github.com/hashicorp/terraform/issues/5480 is used here.

[rolandkool]

One other thing that I noticed, the output of terraform plan show the following self_link:
metadata.0.self_link: "/api/v1/namespacesmyapp" => "<computed>"
Notice the / missing between namespaces and myapp.
Not sure if this is related to the bug described above.

[rolandkool]

Just tested another scenario where wrapped the module in another module so I could pass the remote_state vars to the module managing the kubernetes namespace. In that scenario, the module would be able to use just local vars like var.application_name
However, this seems to result in the same error message, which I did not expect.

[rolandkool]

More testing revealed the same occurs with for example the resource_quota resource. And not just labels but also with setting the quotas themselves in the spec section.
Which leaves me to believe it may have something to do with the deepness of the structure where the variable occur.
labels and limits.memory are both nested. For labels, the variables are nested 2 layers deep (from metadata where the labels are part of). For the limits.memory the nesting is also 2 layers (spec -> hard)
If variable is nested one layer (for instance name below metadata) then it works.

Again, only if I use terraform_remote_state variables somewhere.

[radeksimko]

Hi @rolandkool
I did poke around and I believe this is a core bug, which is mostly discussed in https://github.com/hashicorp/terraform/issues/5480 as you mentioned.

I will keep it open just because you provided a great repro case we can use to verify the bug was fixed, but the work has to be done in hashicorp/terraform, not in this repo.

Thanks.

[olvesh]

@radeksimko: I believe this is not an upstream bug, but at bug in the terraform-provider-kubernetes.
The hashicorp/terraform#5480 issue is only an enhancement discussion about propagating variables, or being able to provide global variables.

This issue is triggered when a module output is given as input to the configmap resource:

provider "kubernetes" {}

resource "local_file" "tmpfile" {
  filename = "/tmp/resource.txt"
  content = "foo-bar"
}

data "local_file" "datafile" {
  filename = "/tmp/resource.txt"
  depends_on = ["local_file.tmpfile"]
}

resource "local_file" "this_works" {
  filename = "/tmp/resource2.txt"
  content = "${data.local_file.datafile.content}"
}

resource "kubernetes_config_map" "this_fails" {
  "metadata" {
    name = "test"
    labels {
      fromfile = "${data.local_file.datafile.content}"
    }
  }
}

It looks like variables that should have the computed state, as the actual value cannot be known before the apply phase.

What triggered this bug for me was the attempt to label a configmap with a value from the output of google_pubsub_subscription resource.

I hope you can remove the upstream-terraform label and re-investigate whether this is caused by a local bug/design flaw?

[Phylu]

I encounter this issue, when I want to use the UID of a config map as a label for a deployment. The ultimate goal is, that the pod is restarted when the configmap changes.

Here is the configuration:

resource "kubernetes_config_map" "config_map" {
  metadata {
    name = "test"
  }

data {
    test = "test"
  }
}

resource "kubernetes_deployment" {
  metadata {
    labels {
      configmap = "${kubernetes_config_map.config_map.metadata.0.uid}"
     }
  }
}

This produces the error:

Error: kubernetes_deployment.test: metadata.0.labels ("${kubernetes_config_map.config_map.metadata.0.uid}") a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

This does not happen if the used value for the label is static. This also does not happen if the very same value is used e.g. as the name within the matadata structure.

[mkyc]

Because configmaps changes do not refresh deployments, it's best idea to change label. In case of terraform it would be easiest to have md5 from ConfigMap but due to this error its impossible.

[Phylu]

We are using environment variables as workaround now:

```
resource "kubernetes_config_map" "config_map" {
metadata {
name = "test"
}

data {
test = "test"
}
}

resource "kubernetes_deployment" {
container {
env {
name = "configmap-hash"
value = "${base64encode(jsonencode(kubernetes_config_map.config_map.data))}"
}
env_from {
config_map_ref {
name = "${kubernetes_config_map.config_map.metadata.0.name}"
}
}
}
}

[mkyc]

yes, exactly the same workaround here:

env {
  name  = "xxx-configmap-hash"
  value = "${md5(sha256(jsonencode(kubernetes_config_map.xxx.data)))}"
}

But it's ugly.

[JSchwerberg]

Having this same issue with a kubernetes_deployment - in my case, I'm trying to set a label with the current commit hash that is deployed, retrieved by running git rev-parse HEAD in a Terraform null_resource. This is a blocker for me doing blue-green deployments, as I want to use this label as a selector in conjunction with lifecycle.create_before_destroy to do blue-green deployments with Terraform.

[pablo-ruth]

@pdecat @alexsomesan it's still blocking for us too. I just tested if this problem is resolved with Terraform 0.12 but sadly not. In #357 you said that ValidateFunc on map attributes is in fact available and working, but on my test it doesn't seem to work.

I created a minimal terraform provider to test if interpolation is working with ValidateFunc: https://github.com/pablo-ruth/terraform-provider-example

There is only one resource type "example_server" with two arguments: "address" of type string and "tags" of type map. Both have a ValidateFunc defined which expose values of each key as seen from inside.

With this simple main.tf, I test interpolation of attributes and variables:

resource "example_server" "my-server" {
  address = "127.0.0.1"
}

resource "example_server" "my-server2" {
  address = example_server.my-server.address

  tags = {
    env = "development"
    name = example_server.my-server.address
    var = var.foo
  }
}

variable "foo" {
  default = "bar"
}

And the plan output:

Terraform will perform the following actions:

  # example_server.my-server will be created
  + resource "example_server" "my-server" {
      + address = "127.0.0.1"
      + id      = (known after apply)
    }

  # example_server.my-server2 will be created
  + resource "example_server" "my-server2" {
      + address = "127.0.0.1"
      + id      = (known after apply)
      + tags    = {
          + "env"  = "development"
          + "name" = "127.0.0.1"
          + "var"  = "bar"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Warning: simple -> value: 127.0.0.1

  on main.tf line 1, in resource "example_server" "my-server":
   1: resource "example_server" "my-server" {



Warning: complex -> key: env - value: development

  on main.tf line 5, in resource "example_server" "my-server2":
   5: resource "example_server" "my-server2" {



Warning: complex -> key: name - value: 74D93920-ED26-11E3-AC10-0800200C9A66

  on main.tf line 5, in resource "example_server" "my-server2":
   5: resource "example_server" "my-server2" {



Warning: complex -> key: var - value: 74D93920-ED26-11E3-AC10-0800200C9A66

  on main.tf line 5, in resource "example_server" "my-server2":
   5: resource "example_server" "my-server2" {

So it seems that ValidateFunc doesn't support interpolation on maps... or I am missing something...

[pablo-ruth]

ping @pdecat

[protomonk]

This is preventing us from using kubernetes provider. How could it be that the issue is not resolved since 2017?

[pablo-ruth]

@protomonk this problem should be resolved with the latest version of Terraform and of this provider (see #491 )

[hashibot]

This issue has been open 180 days with no activity. If this issue is reproducible with the latest version of the provider and with Terraform 0.12, please comment. Otherwise this issue will be closed in 30 days.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!