Terraform-provider-kubernetes: Kubernetes Provider 1.11.3 "Unauthorized"

[privomark]

Terraform Version and Provider Version

Terraform v0.12.26

• provider.archive v1.3.0
• provider.aws v2.70.0
• provider.helm v1.2.3
• provider.kubernetes v1.11.3
• provider.null v2.1.2

Affected Resource(s)

• kubernetes_cluster_role

Terraform Configuration Files

provider "kubernetes" {
  version                = "1.11.3"
  host                   = aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  #config_context         = aws_eks_cluster.cluster.arn
  load_config_file       = false
}

resource "kubernetes_cluster_role" "admin_role" {
  metadata {
    name = "k8s-admin-role"
    labels = {
      name = "AdminRole"
    }
  }

  rule {
    api_groups = ["", "apps", "batch", "extensions"]
    resources = [
      "pods", "cronjobs", "deployments", "devents", "ingresses",
      "jobs", "pods/attach", "pods/exec", "pods/log", "pods/portfoward",
      "secrets", "services", "nodes"
    ]
    verbs = [
      "get", "list", "watch", "delete", "describe", "patch",
    "update", "create"]
  }

  depends_on = [aws_eks_node_group.cluster_node_group]
}

Expected Behavior

What should have happened?
Apply complete!

Terraform should create my cluster, node group and cluster role successfully.

Actual Behavior

Error: Unauthorized

  on ../../modules/eks/rbac.tf line 35, in resource "kubernetes_cluster_role" "admin_role":
  35: resource "kubernetes_cluster_role" "admin_role" {

Works properly 2nd time running terraform apply

Steps to Reproduce

1. terraform apply --> Fails
2. terraform apply --> Works 2nd Time (no changes between commands)

Important Factoids

• Running k8s provider v1.9 works on initial run, but when running terraform destroy I get an error against the kubernetes_cluster_role saying system:anonymous does not have access to destroy.
  
  
  
  • If I change the provider to load_config_file = false and use the config_context, then it works. But this method does not work on terraform apply. I have to manually switch each time I want to do a destroy.
  
  

References

• #679

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[holms]

This problem appeared today, although I've haven't done terraform init for like a week already. Also downgrading up to 3 versions back didn't help it either, so I guess maybe it's a problem with Digital ocean? Although my kubectl works well locally.

[Vlaaaaaaad]

EKS tokens need to be refreshed every 15 minutes. If a Terraform apply runs and takes longer than 15 minutes, we get this error. A re-plan and re-apply works without issue.

What's weird is that we sometimes also get this error on a simple change that takes less than 15 minutes, like changing a value in a ConfigMap. Re-plan and re-apply also work here, so it's not a credential/ AWS IAM issue.

[djakielskiadesso]

It would be nice if you could make the error message (Unauthorized) clearer. At the moment we can only guess that this is a Kubernetes problem. There is no more context surround this error message. No message like stacktrace or on resource ...

Just:

My Versions:
Terraform 1.12.28
Kubernets Provider 1.12
Kubernetes API 1.17

Proposal: Could not connect to kubernetes api (https://api.k8s.de), Authorization failed. Please check your credentials.

[holms]

In my case it was actually unauthorised (into k8s cluster). Problem is that I'm using creds from another state, I've split infra and post-infra projects, and grabbing k8s creds from output which is masked. Seems to me token been expired, when I've did terraform apply in the first project, second one started to work. Interestingly I've never had this issue before. Seems to me digital ocean decreased token lifetime maybe.

[assafcoh]

we also encounter this "Error: Unauthorized" after updating our k8s cluster to new kubernetes version because this takes more than 15 minutes.
It would be great if the Kubernetes Provider could automatically refresh an expired token and continue without errors.

[Carles-Figuerola]

Having this same problem when EKS upgrades take longer than 15 minutes. I think that using a kubeconfig file instead of supplying the token to the provider this way token = data.aws_eks_cluster_auth.cluster.token fixes it as for every kubectl call, a new token is requested. However, I have not confirmed this 100%.

One other option I thought is creating a second provider for post-upgrade usage by aliasing, but I believe that all providers are initialized at the same time so the tokens would expire at the same time: https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-configurations

A solution that doesn't expect the kubeconfig to be set up before the provider runs and that auto-renews the token would be greatly appreciated

[yardensachs]

It seems to be an issue that I would need to apply some change first, to be able to run something like terraform import.
I had to add some resource with missing information so it would fetch the token but fail the apply.
Any better solutions?

[Monkman08]

It seems to be an issue that I would need to apply some change first, to be able to run something like terraform import.
I had to add some resource with missing information so it would fetch the token but fail the apply.
Any better solutions?

This is the exact thing i am seeing with the latest provider. Trying to import without an apply first (so far its only affecting clusterrolebindings and clusterroles)

once i apply something else first, im then able to complete the import.

[sosimon]

Very weird. I just experienced the same thing - Error: unauthorized - when trying to terraform import a resource from an EKS cluster.

I ran a terraform apply (0 diffs! i.e. Apply complete! Resources: 0 added, 0 changed, 0 destroyed.), and then tried the import again and it worked.

[adrianmiron]

I am having the same issues on EKS with 1.13.3 ( did not have this issue with 1.10.x) when deleting configmaps/secrets as part of a big plan which includes a lot of non EKS stuff( i assume the token is requested at the start but by the time it gets to using it it's expired; if doing targeted deletes it works fine). I can also confirm the issue is inconsistent, doing the same plan a few times ( create/delete) it appears about 50% if the times.

Would be great if a fix would be included in future versions, cheers.

[ghassencherni]

Still reproduced with 1.14

[iarlyy]

Experiencing the same with the following stack:

• EKS K8s v1.19
• Terraform v0.14.5
• Kubernetes Provider v2.0.2 (should we open another issue?)

provider "kubernetes" {
  host                   = data.aws_eks_cluster.k8s_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.k8s_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.k8s_cluster.token
}

Everything works locally, however when the same is tried in a pod, the "Unauthorized" message comes up.

Extra note: worker nodes on my stack are EC2 based.

[iarlyy]

This worked for me (using the same provider config above):

Added the role that is being used by the pod to aws-auth configmap.

    - rolearn: arn:aws:iam::**account_id**:role/terraform-runner-role
      username: admin:{{SessionName}}
      groups:
        - system:masters

[fenethtool]

Still reproduced with eks 1.18 and terraform 1.14.

As a temporary fix, I was able to update the token by regenerating the statefile using the terraform refresh command, which avoided this problem.

I hope that you will implement a process to automatically update the token when creating and building kubernetes-related resources.

[rmasclef]

Still reproduced with 1.18.

As a temporary fix, I was able to update the token by regenerating the statefile using the terraform refresh command, which avoided this problem.

I hope that you will implement a process to automatically update the token when creating and building kubernetes-related resources.

I can add the fact that Terraform-cloud users don't have the possibility to to this workaround unless they declare a backend in tf code in order to be able to perf tf actions using cli instead of tfCloud interface.

[Cajga]

Hi Guys,

Seems the recommended way to use the kubernetes provider for EKS is to use exec instead of token due to the short living token for AWS more info in the official documentation:

Some cloud providers have short-lived authentication tokens that can expire relatively quickly. To ensure the Kubernetes provider is receiving valid credentials, an exec-based plugin can be used to fetch a new token before initializing the provider.

At the same time as we are running the config in CI/CD (gitlab) we did not want to have a full blown aws cli (version 2 requires proper glibc which is of course not available in alpine images and it is also few 100MB) in our terraform image so, I developed a very simple go code that can be built into a single static binary (and is 13MB) and can be used by the exec call of the kubernetes provider. You can find it here: terraform-kubernetes-provider-exec-plugin-eks

UPDATE: moved my proj to gitlab so, updated the link

[ipleten]

I get such problem while trying to import kuberenets_secret:

kubernetes_secret.newrelic-license-key: Importing from ID "newrelic/newrelic-license-key"...
kubernetes_secret.newrelic-license-key: Import prepared!
  Prepared kubernetes_secret for import
kubernetes_secret.newrelic-license-key: Refreshing state... [id=newrelic/newrelic-license-key]

Error: Unauthorized

Interesting:
I've replaced:
```
token = data.aws_eks_cluster_auth.cluster.token

with 

exec {
  api_version = "client.authentication.k8s.io/v1alpha1"
  args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.cluster.name]
  command     = "aws"
}

```
and now import works.

[stealthHat]

Still reproduced with eks 1.20 and terraform 1.15. but the workarround that @ipleten posted works for me

[jsun6383]

@ipleten that is perfect. Given the token generated by aws-iam-authenticator have a short life. It would make sense creating a new token on every run would be the better solution.