Terraform-provider-kubernetes: Data resource for Service Account fails to find default token

[servo1x]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.13.3
Kubernetes provider version: 1.13.2
Kubernetes version: 1.16.11

Affected Resource(s)

data.kubernetes_service_account

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
data "kubernetes_service_account" "test_sa" {
  metadata {
    name      = "test"
    namespace = "test"
  }
}

Debug Output

2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 326
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "ServiceAccount",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/serviceaccounts/test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195689",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-06-05T05:51:48Z"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "secrets": [
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "name": "test-token-ncwqf"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  ]
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.702-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Request Details:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ REQUEST ]---------------------------------------
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: GET /api/v1/namespaces/test/secrets/test-token-ncwqf HTTP/1.1
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Host: k8s
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: User-Agent: HashiCorp/1.0 Terraform/0.13.3
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept: application/json, */*
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Authorization: Bearer ...
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept-Encoding: gzip
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 3113
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "Secret",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/secrets/test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "baef6c8c-e549-4962-9e3c-eb0a9de64e6c",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195687",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-10-04T00:57:08Z",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "annotations": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.name": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "data": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "ca.crt": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "token": "..."
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "type": "kubernetes.io/service-account-token"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.856-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Skipping test-token-ncwqf as it wasn't created at the same time as the service account
2021/01/04 05:34:13 [ERROR] eval: *terraform.evalReadDataRefresh, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one
2021/01/04 05:34:13 [ERROR] eval: *terraform.EvalSequence, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create a Service Account
2. Rotate Service Account Default Token
3. terraform apply apply fails because of differing creation timestamps

Expected Behavior

A valid service token should be found regardless of creation timestamp drift.

Actual Behavior

When a service account default token gets rotated, the new secret has a different timestamp and Terraform is unable to find the default token.

References

• GH-848

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[amsgeodis]

@servo1x Have you found any workaround for this issue?

[amsgeodis]

The problem is with the 3 seconds check performed in the below line.

https://github.com/hashicorp/terraform-provider-kubernetes/blob/e6ae58f9b75369c84965a7a13e67f0450e9aa65c/kubernetes/resource_kubernetes_service_account.go#L189

Can we have it increased to a bigger limit, may be 60 seconds? If we have mutation webhooks, the secret creation can take longer.

[wojtek-oledzki]

@wjam, @alexsomesan why do we have that time check in the first place (#377)?

[wjam]

To be able to identify which token was the default one
https://github.com/hashicorp/terraform-provider-kubernetes/blob/master/kubernetes/resource_kubernetes_service_account.go#L160-L167

[wojtek-oledzki]

Why not use the secrets list from the sa object itself?
My token got rotated and is month older then the sa.

[wjam]

Then that secret would not be the default one as defined by the documentation.

Name of the default secret, containing service account token, created & managed by the service
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account#default_secret_name

[wjam]

Note that the PR you linked to was just for adding support to _import_ a service account and nothing more

[wojtek-oledzki]

But why do we use findDefaultServiceAccount at all when we have getServiceAccountDefaultSecret that returns, what it looks like to be the sa secret (default token)

[wjam]

Because import has to deal with the fact that there may be many secrets associated with the service account and it needs to discover which the 'default' one was - which the create operation has already defined as the secret that was created alongside the service account.

I would argue that the default_secret_name attribute should be removed and the secret list also be computed, but that would be a breaking change and I've not contributed to this provider in years.

[amsgeodis]

Note that the PR you linked to was just for adding support to _import_ a service account and nothing more

@wjam Doesn't it impact datasource?

[wjam]

No, PR #377 is about adding support for importing service accounts.

[amsgeodis]

Thanks.

We have found the following snippet failing.
data "kubernetes_service_account" "vault_injector" { metadata { name = var.k8s_vault_sa namespace = var.k8s_vault_namespace } }
.. with the error log Unable to find any service accounts tokens which could have been the default one. And the code/method modified is the only place we have found that message. https://github.com/hashicorp/terraform-provider-kubernetes/blob/cbe392a099bf3a6f7e9981e58cb53be0cdee97c6/kubernetes/resource_kubernetes_service_account.go#L199

[wjam]

Okay. But remember, the PR I raised didn't touch this - review the PR and you can see that it doesn't touch the data source. Adding the service account data source was PR #731.

[amsgeodis]

Got it. I had another PR raised 20 days ago to _mitigate_ it. https://github.com/hashicorp/terraform-provider-kubernetes/pull/1165