Terraform-provider-kubernetes: API version mismatch using azure/kubelogin exec plugin

[noahbirrer-8451]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 0.13.6
Kubernetes provider version: 2.0.2
Kubernetes version: 1.18.14
Kubelogin version: 0.0.8

Affected Resource(s)

NA

Terraform Configuration Files

provider "kubernetes" {
  version = "=2.0.2"
  host        = data.azurerm_kubernetes_cluster.kubernetes_cluster.kube_config[0].host
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    args = ["convert-kubeconfig", "-l", "spn"]
    command = "kubelogin"
    env = {
      KUBECONFIG = var.kubeconfig
      AAD_SERVICE_PRINCIPAL_CLIENT_ID = var.cluster_config_sp_client_id
      AAD_SERVICE_PRINCIPAL_CLIENT_SECRET = var.cluster_config_sp_secret
    }
  }
}

resource "kubernetes_namespace" "namespace" {
  metadata {
    name = "test"
  }
}

Steps to Reproduce

1. Install kubelogin
2. Configure Kubernetes provider block as such
3. Run Terraform Init, Plan, and Apply

Expected Behavior

Terraform should have successfully authenticated to the cluster using kubelogin, the kubeconfig supplied, and the service principal. A Terraform Apply should have been able to run.

Actual Behavior

Running the exec plugin during Terraform Apply failed:

Error: Post "https://<redacted>/api/v1/namespaces": getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1beta1, plugin returned version client.authentication.k8s.io/__internal

Important Factoids

I am attempting to non-interactively authenticate to an AAD Managed AKS cluster using a Service Principal and a secret via Azure's own Kubelogin tool.
Using the same kubelogin binary locally outside of terraform (just from CLI) with the same environment variables works just fine.
It is worth noting that kubelogin runs successfully from the exec block of the kubernetes provider (successfully converts kubeconfig)

References

Related to this issue: https://github.com/Azure/AKS/issues/1763

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

@noahbirrer-8451 If you set api_version in the provider exec block to client.authentication.k8s.io/__internal does it work ? Seems to me that's what the error message is suggesting.

[noahbirrer-8451]

Hi @alexsomesan, When I set the api_version to client.authentication.k8s.io/__internal, I receive this error:
Error: Failed to configure client: exec plugin: invalid apiVersion "client.authentication.k8s.io/__internal

[dak1n1]

hmm, I'm hitting this issue with GKE too. I was trying to update the GKE example, but this issue is blocking that progress. EKS is still working though.

I'm suspecting a bug in one of the Kubernetes libraries. I was able to reproduce it using kubectl too.

$ kube --kubeconfig=./kubeconfig get pods
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-acc-5642.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-acc-5642.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-acc-5642.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-acc-5642.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-acc-5642.
Unable to connect to the server: getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1alpha1, plugin returned version client.authentication.k8s.io/__internal

Here are the versions I'm using:

Kubernetes version: 1.18.17-gke.100
Kubectl version: v1.21.0
gcloud version: Google Cloud SDK 340.0.0

My GKE reproducer is here:
https://github.com/hashicorp/terraform-provider-kubernetes/pull/1271/commits/a40f780f6a20e9f74031c48a074bac8b47009c83

I'll try to make a reproducer that just uses the bare minimum kubernetes libraries (probably client-go) and see if I can open an issue upstream.