Terraform-provider-kubernetes: Unable to create service_account on OpenShift 3.7 due to 2 default secrets

Created on 2 Dec 2017 · 1Comment · Source: hashicorp/terraform-provider-kubernetes

Terraform Version

0.11.0

Affected Resource(s)

kubernetes_service_account
kubernetes_namespace

Terraform Configuration Files

terraform {
  required_version = ">= 0.11.0"
}

variable "k8s_endpoint" {
  description = "k8s_endpoint"
}

variable "k8s_master_auth_client_certificate" {
  description = "k8s_master_auth_client_certificate"
}

variable "k8s_master_auth_client_key" {
  description = "k8s_master_auth_client_key"
}

variable "k8s_master_auth_cluster_ca_certificate" {
  description = "k8s_master_auth_cluster_ca_certificate"
}

provider "kubernetes" {
  host = "${var.k8s_endpoint}"
  client_certificate = "${base64decode(var.k8s_master_auth_client_certificate)}"
  client_key = "${base64decode(var.k8s_master_auth_client_key)}"
  cluster_ca_certificate = "${base64decode(var.k8s_master_auth_cluster_ca_certificate)}"
  load_config_file = false
}

resource "kubernetes_namespace" "dogs" {
  metadata {
    name = "dogs"
  }
}

resource "kubernetes_service_account" "dogs" {
  metadata {
    name = "dogs"
    namespace = "${kubernetes_namespace.dogs.metadata.0.name}"
  }
}

Expected Behavior

The Kubernetes Provider should have created both a namespace and a service account in that namespace on OpenShift without errors.

Note that the namespace and service account actually are created despite the error and show up in terraform state list.

Actual Behavior

Rogers-MacBook-Pro:test-service-account roger$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + kubernetes_namespace.dogs
      id:                                               <computed>
      metadata.#:                                       "1"
      metadata.0.annotations.%:                         "3"
      metadata.0.annotations.openshift.io/description:  "Dogs"
      metadata.0.annotations.openshift.io/display-name: "Dogs"
      metadata.0.annotations.openshift.io/requester:    "admin"
      metadata.0.generation:                            <computed>
      metadata.0.name:                                  "dogs"
      metadata.0.resource_version:                      <computed>
      metadata.0.self_link:                             <computed>
      metadata.0.uid:                                   <computed>

  + kubernetes_service_account.dogs
      id:                                               <computed>
      default_secret_name:                              <computed>
      metadata.#:                                       "1"
      metadata.0.generation:                            <computed>
      metadata.0.name:                                  "dogs"
      metadata.0.namespace:                             "dogs"
      metadata.0.resource_version:                      <computed>
      metadata.0.self_link:                             <computed>
      metadata.0.uid:                                   <computed>


Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kubernetes_namespace.dogs: Creating...
  metadata.#:                                       "" => "1"
  metadata.0.annotations.%:                         "" => "3"
  metadata.0.annotations.openshift.io/description:  "" => "Dogs"
  metadata.0.annotations.openshift.io/display-name: "" => "Dogs"
  metadata.0.annotations.openshift.io/requester:    "" => "admin"
  metadata.0.generation:                            "" => "<computed>"
  metadata.0.name:                                  "" => "dogs"
  metadata.0.resource_version:                      "" => "<computed>"
  metadata.0.self_link:                             "" => "<computed>"
  metadata.0.uid:                                   "" => "<computed>"
kubernetes_namespace.dogs: Creation complete after 0s (ID: dogs)
kubernetes_service_account.dogs: Creating...
  default_secret_name:         "" => "<computed>"
  metadata.#:                  "" => "1"
  metadata.0.generation:       "" => "<computed>"
  metadata.0.name:             "" => "dogs"
  metadata.0.namespace:        "" => "dogs"
  metadata.0.resource_version: "" => "<computed>"
  metadata.0.self_link:        "" => "<computed>"
  metadata.0.uid:              "" => "<computed>"

Error: Error applying plan:

1 error(s) occurred:

* kubernetes_service_account.dogs: 1 error(s) occurred:

* kubernetes_service_account.dogs: Expected 1 generated default secret, 2 found: [{  dogs-dockercfg-4fh66    } {  dogs-token-xfkg2    }]

Both the namespace (which in OpenShift is a project) and the service account were created despite the error!

The project and service account also both show up in Terraform state:

$ terraform state list
kubernetes_namespace.dogs
kubernetes_service_account.dogs

$ oc projects
You have access to the following projects and can switch between them with 'oc project <projectname>':

  * cats-and-dogs - Cats and Dogs
    default
    dogs
    kube-public
    kube-system
    logging
    management-infra
    openshift
    openshift-infra
    openshift-node

$ kubectl get sa dogs -o yaml

```yaml
apiVersion: v1
automountServiceAccountToken: false
imagePullSecrets:

name: dogs-dockercfg-xlkd8
kind: ServiceAccount
metadata:
creationTimestamp: 2017-12-02T19:59:30Z
name: dogs
namespace: dogs
resourceVersion: "120113"
selfLink: /api/v1/namespaces/dogs/serviceaccounts/dogs
uid: 4ef2204f-d79b-11e7-817a-0277a1fe5d6e
secrets:
name: dogs-token-lbsbv
name: dogs-dockercfg-xlkd8


However, the project did not show up in the OpenShift Console until I ran `oadm policy add-role-to-user admin admin` after first selecting the new project with `oc project dogs`.

Note that the Terraform state does not have the default_secret_name attribute:

$ terraform state show kubernetes_service_account.dogs
id = dogs/dogs
metadata.# = 1
metadata.0.annotations.% = 0
metadata.0.generate_name =
metadata.0.generation = 0
metadata.0.labels.% = 0
metadata.0.name = dogs
metadata.0.namespace = dogs
metadata.0.resource_version =
metadata.0.self_link =
metadata.0.uid =


In contrast, when I created a service account with the Kubernetes Provider against Azure Container Service and Google Container Engine, the state in TFE did show an item like "default_secret_name": "cats-and-dogs-token-2f0b2". 

### Steps to Reproduce

1. `terraform apply`

### Important Factoids

The problem with OpenShift seems to be that a service account always has 2 secrets, one for service account authentication and one for Docker.  For example, here is the output from a service account I created with kubectl:

$ kubectl get sa cats-and-dogs -o yaml

```yaml
apiVersion: v1
imagePullSecrets:
- name: cats-and-dogs-dockercfg-56g9h
kind: ServiceAccount
metadata:
  creationTimestamp: 2017-12-02T16:30:00Z
  name: cats-and-dogs
  namespace: cats-and-dogs
  resourceVersion: "100176"
  selfLink: /api/v1/namespaces/cats-and-dogs/serviceaccounts/cats-and-dogs
  uid: 0a923ee9-d77e-11e7-817a-0277a1fe5d6e
secrets:
- name: cats-and-dogs-token-rwdxt
- name: cats-and-dogs-dockercfg-56g9h

The token/secret that I needed to use for authenticating against the Vault Kubernetes auth backend was the first. I think the Kubernetes Provider is unable to decide which token to assign to the default_secret_name computed attribute.

bug

Source