Terraform-provider-kubernetes: Cannot connect to Azure AKS cluster using Kubernetes Provider

[hashibot[bot]]

_This issue was originally opened by @starlord-dixon as hashicorp/terraform#18468. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Terraform Version

terraform version
Terraform v0.11.7
+ provider.azurerm v1.6.0
+ provider.kubernetes v1.1.0
...

Debug Output

azurerm_resource_group.resource_group: Creating...
  location:         "" => "eastus"
  name:             "" => "devops_resource_group"
  tags.%:           "" => "1"
  tags.environment: "" => "Development"
azurerm_resource_group.resource_group: Creation complete after 2s (ID: /subscriptions/2365ee15-ff8e-4dab-9592-...4/resourceGroups/devops_resource_group)
azurerm_kubernetes_cluster.cluster: Creating...
  agent_pool_profile.#:                       "" => "1"
  agent_pool_profile.0.count:                 "" => "1"
  agent_pool_profile.0.dns_prefix:            "" => "<computed>"
  agent_pool_profile.0.fqdn:                  "" => "<computed>"
  agent_pool_profile.0.name:                  "" => "default"
  agent_pool_profile.0.os_disk_size_gb:       "" => "30"
  agent_pool_profile.0.os_type:               "" => "Linux"
  agent_pool_profile.0.vm_size:               "" => "Standard_DS1_v2"
  dns_prefix:                                 "" => "devopsdev"
  fqdn:                                       "" => "<computed>"
  kube_config.#:                              "" => "<computed>"
  kube_config_raw:                            "<sensitive>" => "<sensitive>"
  kubernetes_version:                         "" => "<computed>"
  linux_profile.#:                            "" => "1"
  linux_profile.0.admin_username:             "" => "devops"
  linux_profile.0.ssh_key.#:                  "" => "1"
  linux_profile.0.ssh_key.0.key_data:         "" => "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlDcNhkDzIsqtz/RSyFmyNO2AHFIETepZaYwGdiGRabGUzjNDqPOeRZZOfP6PBm7lJAAWTVsZ34yrfeTRhciYr8klKk0+0o8L0WhSF14NYD/kr4ALqXnE6DAa16cs2/rUx69qhNO1auJ+ogLiKrZMy6iYqre8Es0TBy+BLQf9uA8SIxAUnsrOokYOx/nnzEvexNVZhikgCiessAgb0hfqs6XBhPtytyEElUueO8jRcGLsLyD5saENyL dixon_almeida@GOL008527"
  location:                                   "" => "eastus"
  name:                                       "" => "cluster_devops"
  resource_group_name:                        "" => "devops_resource_group"
  service_principal.#:                        "" => "1"
  service_principal.3132445594.client_id:     "" => "88f377f9-3d65-40c1-ba9c-4e0f7"
  service_principal.3132445594.client_secret: "<sensitive>" => "<sensitive>"
  tags.%:                                     "" => "1"
  tags.Environment:                           "" => "Development"
azurerm_kubernetes_cluster.cluster: Still creating... (10s elapsed)
azurerm_kubernetes_cluster.cluster: Still creating... (20s elapsed)
azurerm_kubernetes_cluster.cluster: Still creating... (30s elapsed)
azurerm_kubernetes_cluster.cluster: Still creating... (40s elapsed)
azurerm_kubernetes_cluster.cluster: Still creating... (50s elapsed)
.
.
azurerm_kubernetes_cluster.cluster: Still creating... (15m20s elapsed)
azurerm_kubernetes_cluster.cluster: Creation complete after 15m21s (ID: /subscriptions/2365ee15-ff8e-4dab-9592-...Service/managedClusters/cluster_devops)

Error: Error applying plan:

1 error(s) occurred:

* provider.kubernetes: Failed to configure: username/password or bearer token may be set, but not both

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Expected Behavior

Create AKS in Azure
Cannot Connect to Kubernetes provider

Actual Behavior

Create AKS in Azure
Connect to AKS using Kubernetes provider
create a namsapce

Steps to Reproduce

provider "azurerm" {
  subscription_id = "${var.subscription_id}"
  client_id       = "${var.client_id}"
  client_secret   = "${var.client_secret}"
  tenant_id       = "${var.tenant_id}"
}

resource "azurerm_resource_group" "resource_group" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"

  tags {
    environment = "${var.Environment}"
  }
}

resource "azurerm_kubernetes_cluster" "cluster" {
  name                = "${var.cluster_name}"
 resource_group_name = "${azurerm_resource_group.resource_group.name}"
  location            = "${azurerm_resource_group.resource_group.location}"
  dns_prefix          = "${var.dns_prefix}"

  linux_profile {
    admin_username = "${var.admin_username}"

    ssh_key {
      key_data = "${var.key_data}"
    }
  }

  agent_pool_profile {
    name            = "${var.name}"
    count           = "${var.count}"
    vm_size         = "${var.vm_size}"
    os_type         = "${var.os_type}"
    os_disk_size_gb = "${var.os_disk_size_gb}"
  }

  service_principal {
    client_id     = "${var.client_id}"
    client_secret = "${var.client_secret}"
  }

  tags {
    Environment = "${var.Environment}"
  }
}

provider "kubernetes" {
  host               = "${azurerm_kubernetes_cluster.cluster.kube_config.0.host}"
  username           = "${azurerm_kubernetes_cluster.cluster.kube_config.0.username}"
  password           = "${azurerm_kubernetes_cluster.cluster.kube_config.0.password}"
  client_certificate = "${base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_certificate)}"
  client_key         = "${base64decode(azurerm_kubernetes_cluster.cluster.kube_config.0.client_key)}"
}

resource "kubernetes_namespace" "namespace" {
  metadata {
    annotations {
      name = "tenant"
    }

    labels {
      mylabel = "development"
    }

    name = "terraform-tenant-namespace"
  }
}

1. terraform apply -var-file config.tfvars

[vzshi]

A workaround is to just get rid of the username and password in provider "kubernetes":

provider "kubernetes" {
    host                   = "${azurerm_kubernetes_cluster.k8s.kube_config.0.host}"
#    username               = "${azurerm_kubernetes_cluster.k8s.kube_config.0.username}"
#    password               = "${azurerm_kubernetes_cluster.k8s.kube_config.0.password}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_config.0.cluster_ca_certificate)}"
}

[vzshi]

Also looks like the documentation for the Kubernetes provider isn't up to date: https://www.terraform.io/docs/providers/kubernetes/index.html#statically-defined-credentials, only shows TLS certificate credentials method.

[RobCannon]

I have made the changes suggested above, but I am unable to authenticate to cluster. I am using RBAC with the cluster. I get these errors:

• kubernetes_service_account.tiller: 1 error(s) occurred:
• kubernetes_service_account.tiller: Unauthorized

I am guessing the credentials that are generated do not have admin access to the cluster, perhaps?

EDIT: Found the issue, you should use the kube_admin_config attribute when using RBAC

provider "kubernetes" {
    host                   = "${azurerm_kubernetes_cluster.k8s.kube_admin_config.0.host}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_admin_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_admin_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.k8s.kube_admin_config.0.cluster_ca_certificate)}"
}

[paultyng]

This seems like the upstream progressive apply issue: https://github.com/hashicorp/terraform/issues/4149

You cannot currently (reliably) chain together a provider's config with the output of a resource.