Terraform-provider-helm: RBAC pb

[obeyler]

Hi I use Kubernetes with RBAC, I've try to use the terraform helm provider without success. I suppose it's due to rbac (see behavior)
Do you have some example how to use these provider in case of RBAC?

Terraform Version

Terraform v0.11.7
provider.helm v0.5.1

Affected Resource(s)

helm release

Terraform Configuration Files

resource "helm_release" "nginx-ingress" {
  name      = "nginx-ingress"
  chart     = "stable/nginx-ingress"  
}

Expected Behavior

ingress nginx controller should be installed

Actual Behavior

the output is :
helm_release.nginx-ingress: rpc error: code = Unknown desc = configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

[hounded]

Same problem with stable/jenkins, did you find a solution?

[obeyler]

Not yet :-( 

I supposed we need to create a service account before but I'm not sure

[aaronmell]

@obeyler I found this https://blog.stigok.com/2018/06/25/aws-eks-kubernetes-terraform-system-anonymous-service-account.html it seems to work for me.

[RothAndrew]

provider "helm" {
    install_tiller = "true"
    service_account = "tiller"
    kubernetes {
        config_context = "tiller"
    }
}

resource "helm_release" "ghost" {
    name = "ghost"
    chart = "stable/ghost"
}

Error: Error applying plan:

1 error(s) occurred:

* helm_release.ghost: 1 error(s) occurred:

* helm_release.ghost: rpc error: code = Unknown desc = configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Which is strange because:

$ kubectl auth can-i --context=tiller list configmaps
yes

[Legogris]

I get the same issue as above when deploying helm charts to a fresh cluster on GKE. Will update here if I have any progress, but pretty stumped ATM.

[PsyonixBarrett]

@Legogris I had the same issue but solved it by creating a service account on the cluster for tiller. Something along these lines...

resource "kubernetes_service_account" "tiller_service_account" {
  metadata {
    name = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller_cluster_role_binding" {
  metadata {
    name = "tiller"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "default"
    namespace = "kube-system"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "${kubernetes_service_account.tiller_service_account.metadata.0.name}"
    namespace = "kube-system"
  }
}

Then in your helm provider config use that newly created service account....

provider "helm" {
    service_account = "${kubernetes_service_account.tiller_service_account.metadata.0.name}"
    kubernetes {
        ...
    }
}

[rickard-von-essen]

@PsyonixBarrett I thin that works because you add the default service account to the cluster-admin role, probably something you didn't intend to do..

[seeruk]

@obeyler Is spot on. You need to have a service account with the correct role already bound before Tiller is installed in the cluster. If you've already got Tiller installed, remove it, and make sure it's set up to use a service account with the correct role.

[lukemarsden]

My read is that if you remove

  subject {
    kind      = "ServiceAccount"
    name      = "default"
    namespace = "kube-system"
  }

from https://github.com/terraform-providers/terraform-provider-helm/issues/77#issuecomment-474148274 then it only grants cluster-admin to service-account tiller, which is less insecure, and probably what @PsyonixBarrett intended?

It also seems to work for me.

[lukemarsden]

@obeyler IMO this should be reopened until a docs fix (in readme and terraform provider docs) makes it clear how to fix this... RBAC is everywhere now, this failed for me on AWS EKS until I unearthed this issue, for example.

[aaronmell]

@lukemarsden Does helm v3 change this?

[aramkarapetian]

Does not feel like RBAC problem. Added all necessary RBAC configs but still get the problem.
My understanding is helm_release does not respect provider values, it till tries to access using default service account.
Look again:
Error: rpc error: code = Unknown desc = configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "kube-system".
User "system:serviceaccount:kube-system:default.
Why default when I clearly set helm provider to use tiller service account?
What is not clear from the message if it is terraform having problems installing tiller or tiller installing chart.
For me too, works if I do it using helm binary from command line and only fails for terraform.

Is there a way to reopen this issue?

[rtrive]

same problem, is there a way to fix that?

[jonasdebeukelaer]

I had this issue too, I think because I selectively installed tiller resources using terraform apply -target.

I ended up manually deleting my tiller install namespace (which was not kube-system) and all its contents, plus the helm charts namespace.

After that I
1) ran my whole tiller module and
2) ran my helm charts module
which sorted out the problem.

[fsadykov]

I had the same issue with this configuration

provider "helm" {
  version = "0.10.4"
  install_tiller = "true"
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  namespace = "${kubernetes_service_account.tiller.metadata.0.namespace}"
  tiller_image = "gcr.io/kubernetes-helm/tiller:${var.tiller_version}"
}

## Service account for tiller
resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "${var.tiller_namespace}"
  }
  secret {
    name = "${kubernetes_secret.tiller.metadata.0.name}"
  }
  automount_service_account_token = true
}

## Secret for tillers service account
resource "kubernetes_secret" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "${var.tiller_namespace}"
  }
}

## Cluster role binding for tiller
resource "kubernetes_cluster_role_binding" "tiller_cluster_rule" {
    depends_on = [
      "kubernetes_service_account.tiller",
      "kubernetes_secret.tiller"
    ]
    metadata {
        name = "tiller-cluster-rule"
    }
    role_ref {
        api_group = "rbac.authorization.k8s.io"
        kind      = "ClusterRole"
        name      = "cluster-admin"
    }
    subject {
        kind      = "ServiceAccount"
        name      = "${kubernetes_service_account.tiller.metadata.0.name}"
        namespace = "${kubernetes_service_account.tiller.metadata.0.namespace}"
    }
}

Then I change to helm init local exec it's working now.

resource "null_resource" "helm_init" {
  depends_on = [
    "kubernetes_service_account.tiller",
    "kubernetes_secret.tiller",
    "kubernetes_cluster_role_binding.tiller_cluster_rule"
  ]
  provisioner "local-exec" {
    command = <<EOF
    kubectl delete deploy -n kube-system  tiller-deploy
    helm init --service-account ${kubernetes_service_account.tiller.metadata.0.name} --tiller-namespace ${kubernetes_service_account.tiller.metadata.0.namespace}
    EOF
  }
}

## Service account for tiller
resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "${var.tiller_namespace}"
  }
  secret {
    name = "${kubernetes_secret.tiller.metadata.0.name}"
  }
  automount_service_account_token = true
}

## Secret for tillers service account
resource "kubernetes_secret" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "${var.tiller_namespace}"
  }
}

## Cluster role binding for tiller
resource "kubernetes_cluster_role_binding" "tiller_cluster_rule" {
    depends_on = [
      "kubernetes_service_account.tiller",
      "kubernetes_secret.tiller"
    ]
    metadata {
        name = "tiller-cluster-rule"
    }
    role_ref {
        api_group = "rbac.authorization.k8s.io"
        kind      = "ClusterRole"
        name      = "cluster-admin"
    }
    subject {
        kind      = "ServiceAccount"
        name      = "${kubernetes_service_account.tiller.metadata.0.name}"
        namespace = "${kubernetes_service_account.tiller.metadata.0.namespace}"
    }
}

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!