Terraform-provider-kubernetes: automountServiceAccountToken default disagrees with k8s

[bukzor]

Affected Resource(s)

• kubernetes_service_account

Terraform Configuration Files

resource "kubernetes_service_account" "controller" {
  metadata {
    name      = "gcp-filestore-csi-controller-sa"
    namespace = "gcp-filestore-csi-driver"
  }
}

Debug Output

Expected Behavior

I expected the result of terraform to be the same as the result of the equivalent kubectl apply myyaml. For example:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: garbage
  namespace: "gcp-filestore-csi-driver"
EOF
serviceaccount/garbage created

$ kubectl -n gcp-filestore-csi-driver get serviceaccounts -o yaml
  kind: ServiceAccount
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"garbage","namespace":"gcp-filestore-csi-driver"}}
    creationTimestamp: "2018-12-24T01:24:33Z"
    name: garbage
    namespace: gcp-filestore-csi-driver
    resourceVersion: "1012602"
    selfLink: /api/v1/namespaces/gcp-filestore-csi-driver/serviceaccounts/garbage
    uid: ab22d4e1-071a-11e9-90c1-42010a2fffe6
  secrets:
  - name: garbage-token-w9kx5

Actual Behavior

Note that automountServiceAccountToken is set to false, unlike the expected behavior.

$ kubectl -n gcp-filestore-csi-driver get serviceaccounts -o yaml
- apiVersion: v1
  automountServiceAccountToken: false
  kind: ServiceAccount
  metadata:
    creationTimestamp: "2018-12-23T21:05:41Z"
    name: gcp-filestore-csi-controller-sa
    namespace: gcp-filestore-csi-driver
    resourceVersion: "962672"
    selfLink: /api/v1/namespaces/gcp-filestore-csi-driver/serviceaccounts/gcp-filestore-csi-controller-sa
    uid: 817b6081-06f6-11e9-a434-42010a2fffe4
  secrets:
  - name: gcp-filestore-csi-controller-sa-token-n4msk

References

The Reference doesn't explicitly mention the default value of this field, but behaviorally, the default is true, as implied by the "can opt out" language in the user guide. Currently terraform-provider-kubernetes sets the default to "false" which disagrees with the default behavior in a subtle but completely-breaking way. As a newcomer, I didn't know anything about this file, much less that it could be turned off, so I assumed that the mysterious "open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" error I was encountering was caused by my own fault, and it took me ~4 hours to find the true cause.

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#podspec-v1-core
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#use-the-default-service-account-to-access-the-api-server

[bukzor]

There's something worse wrong than the default, because changing the terraform config for this field has no effect on the actual value. I've toggled it from true to false and back in the tf config, and tf says it updates it, but nothing actually happens to the kubernetes service account's config.

$ tf apply --refresh=false --target="module.filestore-csi-driver.kubernetes_service_account.node"
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.filestore-csi-driver.kubernetes_service_account.node
      automount_service_account_token: "false" => "true"


Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

filestore-csi-driver.kubernetes_service_account.node: Modifying... (ID: gcp-filestore-csi-driver/gcp-filestore-csi-node-sa)
  automount_service_account_token: "false" => "true"
module.filestore-csi-driver.kubernetes_service_account.node: Modifications complete after 0s (ID: gcp-filestore-csi-driver/gcp-filestore-csi-node-sa)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

$ kubectl -n gcp-filestore-csi-driver get serviceaccounts/gcp-filestore-csi-node-sa -o yaml
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
  creationTimestamp: "2018-12-24T00:16:12Z"
  name: gcp-filestore-csi-node-sa
  namespace: gcp-filestore-csi-driver
  resourceVersion: "999344"
  selfLink: /api/v1/namespaces/gcp-filestore-csi-driver/serviceaccounts/gcp-filestore-csi-node-sa
  uid: 1e978618-0711-11e9-90c1-42010a2fffe6
secrets:
- name: gcp-filestore-csi-node-sa-token-dh2vf

$ tf state show module.filestore-csi-driver.kubernetes_service_account.node
id                              = gcp-filestore-csi-driver/gcp-filestore-csi-node-sa
automount_service_account_token = true
default_secret_name             = gcp-filestore-csi-node-sa-token-dh2vf
image_pull_secret.#             = 0
metadata.#                      = 1
metadata.0.annotations.%        = 0
metadata.0.generate_name        =
metadata.0.generation           = 0
metadata.0.labels.%             = 0
metadata.0.name                 = gcp-filestore-csi-node-sa
metadata.0.namespace            = gcp-filestore-csi-driver
metadata.0.resource_version     = 999344
metadata.0.self_link            = /api/v1/namespaces/gcp-filestore-csi-driver/serviceaccounts/gcp-filestore-csi-node-sa
metadata.0.uid                  = 1e978618-0711-11e9-90c1-42010a2fffe6
secret.#                        = 0

[bukzor]

tf taint improved the situation; the service account has automountServiceAccountToken: true, now.

[kzap]

This was defaulted to false in https://github.com/terraform-providers/terraform-provider-kubernetes/issues/38 but IMHO thats a wrong decision.

It was only recently added to docs but not yet released in the docs: https://github.com/terraform-providers/terraform-provider-kubernetes/pull/253/files

Whats worse is that for some specs like Pod its just set to false
https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/resource_kubernetes_pod.go#L56
despite it being part of the API: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#pod-v1-core

This should be added to the configuration spec and not fixed to false
Looks like we have an open PR here https://github.com/terraform-providers/terraform-provider-kubernetes/pull/261

If this provider has specific opinions just so that it can work, it is essentially crippled. The lengths I had to go to find out this was the problem in the provider... :)

[jhoblitt]

@kzap I've added a commit to #261 that removes the default for automountServiceAccountToken but that PR is essentially stalled on needing the ability to identify (and ignore) the auto-magic mounts.

[dploeger]

This is breaking my ingress controller setup. Is it possible to manually mount the service account token currently or any workaround to this?

[j00p34]

@dploeger set automountServiceAccountToken: true in your deployment config. (kubectl edit deploy/deployment_name)

[dploeger]

@j00p34 I thought, that the Pod setting to false would override the deployment setting like @kzap mentioned

[j00p34]

@dploeger No I believe the default of false results in the automountServiceAccountToken being set on a pod when you deploy the pod with the kubernetes provider. A single pod that is. If you set it for the deployment the pod is created by the deployment, not by the provider. So the deployment sets the value

[dploeger]

@j00p34 I saw, that in the pods automountServiceAccountToken was set to false after I have changed it in the deployment. But I'm currently unsure, wether I had recreated the pods from the deployment. Will check it out and keep you posted.

[dh-harald]

It's definetely true in k8s api, I can confirm that... This default false value completely brakes kube-state-metrics deployment without automountServiceAccountToken=True

[dploeger]

To come back to this, yes. kubectl edit (I used patch) enables the automount in the deployment again.

However, this is obviously not the way I want to do that with Terraform. 😄

Any progress on setting this on the deployment using terraform?

[pcj]

Agree here. I get that the maintainers want to not expose this option to keep it "strict" (and force you mount the service account directly) but this makes things very painful.

[Natalique]

Is this planned for some release?
Very much needed.

[botzill]

Hi.

Any updates on this issue? Why not have a way to configure this as we need?

[pdecat]

There's an active PR: https://github.com/terraform-providers/terraform-provider-kubernetes/pull/261

[rossigee]

FTR, this appears to be related to #38.

[JnMik]

In terraform it seems even if you set automount_service_account_token = true on _kubernetes_service_account_, it seems the _kubernetes_deployment_ resource block overwrite it with "false" by default.

The issue is that _kubernetes_deployment_ resource doesn't permit to change this config.
https://www.terraform.io/docs/providers/kubernetes/r/deployment.html
IMO, this is one of the real issue here ?

I had to create my deployment config in a yaml file and call it with a local-exec provisioner to make things work

resource "null_resource" "deployment" {

  depends_on = ["kubernetes_service_account.alb-ingress-controller"]
  provisioner "local-exec" {
    command = "kubectl apply -f deployment.yaml"
  }

  provisioner "local-exec" {
    when = "destroy"
    command = "kubectl delete -f deployment.yaml"
  }
}

[tdmalone]

@JnMik See the workaround at https://github.com/terraform-providers/terraform-provider-kubernetes/issues/38#issuecomment-318581203

A fix is likely to be merged soon for Deployments: https://github.com/terraform-providers/terraform-provider-kubernetes/pull/261#issuecomment-502400193

[hashibot]

This issue has been open 180 days with no activity. If this issue is reproducible with the latest version of the provider and with Terraform 0.12, please comment. Otherwise this issue will be closed in 30 days.

[rsalmond]

Yo @hashibot the fixes to problem with disabling token automount have been merged but the fact that k8s tf provider inverts the default behaviour could be better documented: #814

[aareet]

814 was merged, I believe this can now be closed.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!