Terraform-provider-kubernetes: kubernetes_secret data source is always showing diffs at plan

[chihaiaalex]

I'm trying to read a service account token using a kubernetes_secret data source.

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.14.2
Kubernetes provider version: 2.0.2
Kubernetes version: v1.19.2

Affected Resource(s)

kubernetes_secret

Terraform Configuration Files

resource "kubernetes_service_account" "token_reviewer" {
  metadata {
    name = var.token_reviewer.service_account.name
    namespace = var.token_reviewer.service_account.namespace
  }
}

data "kubernetes_secret" "token_reviewer_secret" {
  metadata {
    name = kubernetes_service_account.token_reviewer.default_secret_name
    namespace = var.token_reviewer.service_account.namespace
  }
}

resource "vault_auth_backend" "vault_kubernetes_auth" {
  type = "kubernetes"
  path = var.path
}

resource "vault_kubernetes_auth_backend_config" "vault_kubernetes_auth_config" {
  backend = vault_auth_backend.vault_kubernetes_auth.path
  kubernetes_host = var.kubernetes.host
  kubernetes_ca_cert = var.kubernetes.ca_cert
  token_reviewer_jwt = data.kubernetes_secret.token_reviewer_secret.data.token
}

Steps to Reproduce

terraform apply

terraform plan

Expected Behavior

The code works, the service account is created and I'm able to get the token from the secret.

The problem is that each time I run a terraform plan, I get a diff saying that the data source will be read during apply and all related resources that are referencing that datasource will be updated in-place.

Actual Behavior

``` # module.vault_kubernetes_backend.data.kubernetes_secret.token_reviewer_secret will be read during apply
# (config refers to values not yet known)
<= data "kubernetes_secret" "token_reviewer_secret" {
~ data = (sensitive value)
~ id = "kubernetes-external-secrets/kubernetes-external-secrets-token-reviewer-token-bd4wf" -> (known after apply)
~ type = "kubernetes.io/service-account-token" -> (known after apply)

  ~ metadata {
      - annotations      = {} -> null
      ~ generation       = 0 -> (known after apply)
      - labels           = {} -> null
        name             = "kubernetes-external-secrets-token-reviewer-token-bd4wf"
      ~ resource_version = "16516532" -> (known after apply)
      + self_link        = (known after apply)
      ~ uid              = "1650e5c1-254a-4f59-8e49-719b940b29f3" -> (known after apply)
        # (1 unchanged attribute hidden)
    }
}

# module.vault_kubernetes_backend.vault_kubernetes_auth_backend_config.vault_kubernetes_auth_config will be updated in-place
~ resource "vault_kubernetes_auth_backend_config" "vault_kubernetes_auth_config" {
id = "auth/k8s/play01-dev-clj/config"
~ token_reviewer_jwt = (sensitive value)
# (6 unchanged attributes hidden)
}
```

Other comments

I've read about Data Source Lifecycle and I even tried hardcoding the secret name that I reference in the kubernetes_secret datasource (so that it doesn't depend on a resource that may change), but the behaviour is the same.

Is there any solution/workaround for this issue?

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

Hi, I'm not able to reproduce this using a simple Kind cluster. It works as expected in my setup.
Can you please share more details about the cluster type you are doing this operation on? Is it a hosted cloud-provider cluster? All the detail you can share helps.

[alexsomesan]

Also, where is the variable token_reviewer being set? Is this from user input or from another resource / module?

[chihaiaalex]

It's an on prem cluster, deployed with Rancher.
The resources above are defined inside a module.
This is the definition of the token_reviewer variable, along with the default values which are not overwritten by the calling module.
Storage backend for Terraform is Postgres.
I'm not sure what other details can help..

variable "token_reviewer" {
  type = object({
    service_account = object({
      name = string
      namespace = string
    })

    cluster_role_binding = object({
      name = string
      role_ref_name = string
    })
  })

  default = {
    service_account = {
      name = "kubernetes-external-secrets-token-reviewer"
      namespace = "kubernetes-external-secrets"
    }

    cluster_role_binding = {
      name = "kubernetes-external-secrets-token-reviewer-auth"
      role_ref_name = "system:auth-delegator"
    }
  }
}

[AKishchak]

I have the same issue with volume and volume_mount settings of k8s deployment
Settings are successfully applied to the cluster (I can verify this from kubectl), but terraform shows changes after each apply

```
spec {
container {
name = "metrics-server"
image = "k8s.gcr.io/metrics-server/metrics-server:v0.3.7"
image_pull_policy = "Always"
args = [
"/metrics-server",
"--v=4",
"--kubelet-insecure-tls",
"--kubelet-preferred-address-types=InternalIP",
"--secure-port=4443",
"--logtostderr"
]

      port {
        name           = "main-port"
        container_port = 4443
        protocol       = "TCP"
      }

      volume_mount {
        name       = "tmp-dir"
        mount_path = "/tmp"
      }
      volume_mount {
        mount_path = "/var/run/secrets/kubernetes.io/serviceaccount"
        name       = kubernetes_service_account.metrics_server_account.default_secret_name
        read_only  = true
      }
    }
    volume {
      name = "tmp-dir"
      empty_dir {}
    }
    volume {
      name = kubernetes_service_account.metrics_server_account.default_secret_name
      secret {
        secret_name = kubernetes_service_account.metrics_server_account.default_secret_name
      }
    }
    service_account_name = kubernetes_service_account.metrics_server_account.metadata[0].name
  }

 Terraform plan: 

# module.cluster.kubernetes_deployment.metrics_server will be updated in-place
~ resource "kubernetes_deployment" "metrics_server" {
id = "kube-system/metrics-server"
# (1 unchanged attribute hidden)

  ~ spec {
        # (5 unchanged attributes hidden)



      ~ template {

          ~ spec {
                # (12 unchanged attributes hidden)

              ~ container {
                    name                       = "metrics-server"
                    # (9 unchanged attributes hidden)



                  + volume_mount {
                      + mount_path        = "/var/run/secrets/kubernetes.io/serviceaccount"
                      + mount_propagation = "None"
                      + name              = "metrics-server-token-mqsf9"
                      + read_only         = true
                    }
                    # (3 unchanged blocks hidden)
                }

              + volume {
                  + name = "metrics-server-token-mqsf9"

                  + secret {
                      + default_mode = "0644"
                      + secret_name  = "metrics-server-token-mqsf9"
                    }
                }
                # (1 unchanged block hidden)
            }
            # (1 unchanged block hidden)
        }
        # (2 unchanged blocks hidden)
    }
    # (1 unchanged block hidden)
}

```

[alexsomesan]

Followup question: did you by chance update the provider version in between the first apply and the second one? In other words, was the state generated by an earlier provider version? And if so, which version specifically?

[chihaiaalex]

@alexsomesan there was no update of the provider between the first and second apply.

[dak1n1]

It looks like we need to remove self_link from metadata on the provider side.

[andreivnst]

@dak1n1 +1 - could you remove the self_link from metadata attribute of kubernetes_secret datasource? Or set it up as an optional value ?
I see that the self_link attribute is setup in the database as an empty string value (for what @chihaiaalex exposed):

"metadata": [
{
"annotations": {},
"generation": 0,
"labels": {},
"name": "kubernetes-external-secrets-token-reviewer-token-fzphq",
"namespace": "kubernetes-external-secrets",
"resource_version": "15070018",
"self_link": "",
"uid": "b5e1d645-51a5-4fa0-a75d-831b9b75e8c9"
}
Furthermore,
I tried to directly specify it as an empty string (like the above from database) but it is not taken into consideration, the terraform plan still considers as a new value:
~ metadata {
~ generation = 0 -> (known after apply)
name = "kubernetes-external-secrets-token-reviewer-token-hlsj7"
~ resource_version = "15260191" -> (known after apply)
+ self_link = (known after apply)
~ uid = "51c4a0ee-f5bc-43c2-ad0c-3d47af86acca" -> (known after apply)
# (3 unchanged attributes hidden)
}

I found that the metadata field for kubernetes_secret datasource object is build here:
https://github.com/hashicorp/terraform-provider-kubernetes/blob/8835e933de82e210e4924d5cf76a2fe4968f2627/kubernetes/schema_metadata.go#L9

Thank you

[dak1n1]

@andreivnst Thanks for pointing me right to it! That was a quick fix. It'll be in version 2.3.1. (This was some leftover code that shouldn't have been in there anyway, since we removed self-link in a previous release).