Terraform-provider-helm: Tiller does not install correctly

[shamsalmon]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.
Terraform v0.11.10

Affected Resource(s)

Please list the resources as a list, for example:

• tiller

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

provider "helm" {
  install_tiller  = true
  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
  home            = "./.helm"

  kubernetes {
    config_path = "./kubeconfig_monitoring"
  }
}

Expected Behavior

Helm install works after tiller installation

Actual Behavior

Helm returns Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 127.0.0.1:8080: connect: connection refused

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

I created the cluster with the EKS module which created the kubernetes config. When I run helm init --service-account tiller then everything works correctly. When configured with the provider it is broken.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• GH-1234

[shamsalmon]

As a workaround I can do this

resource "null_resource" "helm_init" {
  provisioner "local-exec" {
    command = "helm init --service-account tiller --wait"
  }
}
provider "helm" {
  install_tiller  = false
  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
  home            = "./.helm"

  kubernetes {
    config_path = "./kubeconfig_monitoring"
  }
}

resource "helm_release" "prometheus_operator" {
  name  = "monitoring"
  chart = "stable/prometheus-operator"

  values = [
    "${file("${path.module}/helm/prometheus-operator/values.yml")}",
  ]

  depends_on = ["kubernetes_service_account.tiller", "kubernetes_cluster_role_binding.tiller", "null_resource.helm_init"]
}

[olib963]

Hey @shamsalmon I had a similar problem, I think the issue is that tiller isn't automounting the token for the service account, there is a PR open that will hopefully fix this soon by upgrading to helm 2.11.

https://github.com/terraform-providers/terraform-provider-helm/issues/122 might be related. Sorry if it is a different issue.

[Stelminator]

I traced this to what seems to be the most recent related change to helm: https://github.com/helm/helm/pull/4589 so, I think @olib963 is correct that 2.11.0+ should do the trick.

[gaui]

Yes if you add automountServiceAccountToken: true to the tiller deployment, it works.

[Jeeppler]

@gaui How do you add automountServiceAccountToken: true in the Helm provider?

[gaui]

@Jeeppler You can't because 6.0 depends on Helm 2.10, but Helm 2.11 has it enabled - so we have to wait for the new version. But you can bypass it by doing what @shamsalmon suggested above. https://github.com/terraform-providers/terraform-provider-helm/issues/148#issuecomment-436063369

[gaui]

@Jeeppler or just manually set the flag...

kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":true}}}}'

[meyskens]

This will be fixed by #143

[shamsalmon]

This should be fixed by new release that just came out (0.7.0). Just wondering if you know @meyskens when does terraform publish provider changes? Terraform does not seem to be able to grab it.

[Jeeppler]

@shamsalmon This is fixed, see #171

[Johnch9]

Has anyone got this to work - I'm running 0.7.0, but tiller is not installed if I use the following definition. Is there anything else I need to set?

provider "helm" {
  install_tiller  = true
  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"

  kubernetes {
    config_path = "${var.kube_config_path}"
  }
}

[Jeeppler]

@Johnch9 I was unable to get it to work either.

[alexsomesan]

Make sure Tiller is using a service account with appropriate permissions.

Check the Tiller pod logs for errors.

[Jeeppler]

@alexsomesan can you elaborate a little bit more what you mean with appropriate permissions? Do we have to create the service account "tiller" before using it in the Helm provider?

I currently have the following configuration:

provider "helm" {
    service_account = "tiller"
    namespace = "kube-system"
    install_tiller = true
    debug = true
    insecure = true
    tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"

    kubernetes {
      config_path = "/k8s/kubeconfig_${var.cluster_name}"
    }
}

and this is the output or Terraform apply:

$ terraform apply       

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Releasing state lock. This may take a few moments...

I do not have any Tiller pod running, so I am unable to check the Tiller pod logs for errors:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY     STATUS    RESTARTS   AGE
kube-system   aws-node-b5rw2             1/1       Running   0          39m
kube-system   coredns-7bcbfc4774-98gx7   1/1       Running   0          42m
kube-system   coredns-7bcbfc4774-9b7b4   1/1       Running   0          42m
kube-system   kube-proxy-7jfm7           1/1       Running   0          39m

[jhoblitt]

It hasn't yet been updated for helm 2.11.0, but I hacked up a simple module for installing tiller with rbac (and without tiller being accessible to other pods).

https://github.com/lsst-sqre/terraform-tinfoil-tiller

[hemantseth0210]

I tried with 0.7.0 but it still doesn't work. I hope I am not missing any configuration

[jhoblitt]

I've updated https://github.com/lsst-sqre/terraform-tinfoil-tiller to work with 0.7.0. Eg.,

module "tiller" {
  source = "git::https://github.com/lsst-sqre/terraform-tinfoil-tiller.git//?ref=master"

  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
}

provider "helm" {
  version = "~> 0.7.0"

  debug           = true
  service_account = "${module.tiller.service_account}"
  namespace       = "${module.tiller.namespace}"
  install_tiller  = false

  kubernetes {
    ...
  }
}

[Jeeppler]

@hemantseth0210 I could not get it to work either.

[Stelminator]

@Jeeppler, there are couple possibly unecessary bits here, but I left them in anyway.

1. automount_service_account_token = true should only be necessary for helm provider version < 0.7.0
2. tiller_image = "gcr.io/kubernetes-helm/tiller:v2.11.0" since this doesn't seem to have been updated. Whoops. We should do that for 2.12.0/0.8.0.

TL;DR: yes, the service account needs to be created first.

provider "kubernetes" {
  version = ">= 1.4.0"
  ...
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "terraform-tiller"
    namespace = "kube-system"
  }

  automount_service_account_token = true
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "terraform-tiller"
  }

  role_ref {
    kind      = "ClusterRole"
    name      = "cluster-admin"
    api_group = "rbac.authorization.k8s.io"
  }

  subject {
    kind = "ServiceAccount"
    name = "terraform-tiller"

    api_group = ""
    namespace = "kube-system"
  }
}

provider "helm" "cluster_helm" {
  version = "~> 0.7.0"

  kubernetes {
    ...
  }

  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  namespace       = "${kubernetes_service_account.tiller.metadata.0.namespace}"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
}

@jhoblitt Do I understand correctly that the main point of https://github.com/lsst-sqre/terraform-tinfoil-tiller is that it only listens on localhost?

[jhoblitt]

@Stelminator That's correct. It avoids the need to configure x509 certs, network policy, etc.

[jiphex]

Just confirming that this seems to work when using Terraform 0.1.11, Helm Provider 0.7.0 and Helm 2.12.1 with the instructions provided by @Stelminator.

In case it's useful to anyone else coming across this: I'd previously got this working by shell-ing out to helm init and I had to manually clear some old state referring to helm_releases using helm state _before_ I could successfully do a terraform plan using the method described above.

[btai24]

@jiphex I don't seem to have it working with Terraform 0.11.11, Helm Provider 0.8.0, Helm 2.11.0. I'm adding the helm provider with install_tiller set to true, but I don't see a tiller pod when I spin up the cluster.

Here are the relevant HCL pieces of this:

data "aws_eks_cluster_auth" "eks" {
  name = "${local.name}"
}


resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }

  automount_service_account_token = true
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller-cluster-admin"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

provider "helm" {
  install_tiller  = true
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  namespace       = "${kubernetes_service_account.tiller.metadata.0.namespace}"

  kubernetes {
    host                   = "${module.eks_cluster.endpoint}"
    cluster_ca_certificate = "${base64decode(module.eks_cluster.certificate_authority_data)}"
    token                  = "${data.aws_eks_cluster_auth.eks.token}"
    load_config_file       = false
  }
}

[syst0m]

@Stelminator Using your config, I still cannot install Tiller:

  + module.habito-eks.kubernetes_cluster_role_binding.tiller
      id:                              <computed>
      metadata.#:                      "1"
      metadata.0.generation:           <computed>
      metadata.0.name:                 "terraform-tiller"
      metadata.0.resource_version:     <computed>
      metadata.0.self_link:            <computed>
      metadata.0.uid:                  <computed>
      role_ref.%:                      "3"
      role_ref.api_group:              "rbac.authorization.k8s.io"
      role_ref.kind:                   "ClusterRole"
      role_ref.name:                   "cluster-admin"
      subject.#:                       "1"
      subject.0.kind:                  "ServiceAccount"
      subject.0.name:                  "terraform-tiller"
      subject.0.namespace:             "kube-system"

  + module.habito-eks.kubernetes_service_account.tiller
      id:                              <computed>
      automount_service_account_token: "true"
      default_secret_name:             <computed>
      metadata.#:                      "1"
      metadata.0.generation:           <computed>
      metadata.0.name:                 "terraform-tiller"
      metadata.0.namespace:            "kube-system"
      metadata.0.resource_version:     <computed>
      metadata.0.self_link:            <computed>
      metadata.0.uid:                  <computed>

  + module.monitoring.helm_release.prometheus-operator
      id:                              <computed>
      chart:                           "stable/prometheus-operator"
      disable_webhooks:                "false"
      force_update:                    "false"
      keyring:                         "/home/tomislav/.gnupg/pubring.gpg"
      metadata.#:                      <computed>
      name:                            "prometheus-operator"
      namespace:                       "default"
      recreate_pods:                   "false"
      reuse:                           "false"
      reuse_values:                    "false"
      timeout:                         "300"
      verify:                          "false"
      version:                         "3.0.0"
      wait:                            "true"


Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.habito-eks.kubernetes_service_account.tiller: Creating...
  automount_service_account_token: "" => "true"
  default_secret_name:             "" => "<computed>"
  metadata.#:                      "" => "1"
  metadata.0.generation:           "" => "<computed>"
  metadata.0.name:                 "" => "terraform-tiller"
  metadata.0.namespace:            "" => "kube-system"
  metadata.0.resource_version:     "" => "<computed>"
  metadata.0.self_link:            "" => "<computed>"
  metadata.0.uid:                  "" => "<computed>"
module.habito-eks.kubernetes_cluster_role_binding.tiller: Creating...
  metadata.#:                  "" => "1"
  metadata.0.generation:       "" => "<computed>"
  metadata.0.name:             "" => "terraform-tiller"
  metadata.0.resource_version: "" => "<computed>"
  metadata.0.self_link:        "" => "<computed>"
  metadata.0.uid:              "" => "<computed>"
  role_ref.%:                  "" => "3"
  role_ref.api_group:          "" => "rbac.authorization.k8s.io"
  role_ref.kind:               "" => "ClusterRole"
  role_ref.name:               "" => "cluster-admin"
  subject.#:                   "" => "1"
  subject.0.kind:              "" => "ServiceAccount"
  subject.0.name:              "" => "terraform-tiller"
  subject.0.namespace:         "" => "kube-system"

Error: Error applying plan:

2 error(s) occurred:

* module.habito-eks.kubernetes_service_account.tiller: 1 error(s) occurred:

* kubernetes_service_account.tiller: Post http://localhost/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:80: connect: connection refused
* module.habito-eks.kubernetes_cluster_role_binding.tiller: 1 error(s) occurred:

* kubernetes_cluster_role_binding.tiller: Post http://localhost/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: dial tcp 127.0.0.1:80: connect: connection refused

Using terraform v0.11.11

[jiphex]

@syst0m your terraform errors at the end there are showing that it can't connect to Kubernetes - it's trying on localhost port 80 so I assume that's not where you're running Kubernetes

@btai24 I can't see any error output there, I can just confirm that this worked for me once I reset the state of the helm_release objects

[mkozjak]

Yeah, this is mostly a problem with unconfigured local environment, but I see the OP also had this issue, so it might be something else. Anyway, @syst0m, eg. if you're using GCP, you'd use gcloud to set credentials for your GKE cluster. Once Kubernetes cluster is brought up, you should have a kubeconfig file. You need your tiller setup to look at that file. Hope it helps.

[grumvalski]

Hi, I tried @Stelminator config on GKE cluster with
Terraform v0.11.11

• provider.helm v0.8.0
• provider.kubernetes v1.5.2
  terraform apply ends without errors but tiller is not installed.

Activating terraform debug log I can see the following message

[DEBUG] pruning unused provider my_module.provider.helm

In our deployment we don't use helm provider to deploy helm charts but we would like to have tiller installed by terraform. Am I missing something?

[previ]

[EDIT] Looks like the problem was due to a duplicate declaration of helm provider, in a nested module.
I also separated in two separate terraform "states" (to be applied sequentially) the ServiceAccount creation from the actual helm install, this to avoid the possible issue of the install job being executed before the ServiceAccout creation.

Same issue with AKS, my configuration:
azurerm v 1.22.1
kubernetes v1.5.2
helm 0.8.0

 resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }

  automount_service_account_token = true
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller-binding"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }

  role_ref {
    kind      = "ClusterRole"
    name      = "cluster-admin"
    api_group = "rbac.authorization.k8s.io"
  }
}

provider "helm" {

  kubernetes {
    host = "${azurerm_kubernetes_cluster.main.kube_config.0.host}"

    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)}"
  }

  install_tiller  = "true"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.13.0"
}

Tiller account is actually created and tiller itself does get installed but with wrong serviceaccount (default instead of tiller) and wrong version (v2.11.0 instead of specified 2.13.0).

[cwiggs]

I think the problem some people are having here is it seems like just including this block doesn't actually install tiller because terraform doesn't know to do anything to the remote k8 system just from a provider block.

provider "helm" {
  install_tiller  = true
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  tiller_image    = "${var.tiller_image}"
}

However if you include this block it looks like tiller does get installed (although terraform never really tells you it does)

resource "helm_release" "mydatabase" {
  name  = "mydatabase"
  chart = "stable/mariadb"

  set {
    name  = "mariadbUser"
    value = "foo"
  }

  set {
    name  = "mardiadbPassword"
    value = "qux"
  }
}

This is kinda a weird setup as I've never seen a terraform provider block actually install anything remotely. I think a better setup would be to remove the "install_tiller" argument and create a resource that installs tiller instead. Then if you try to use the helm provider and tiller isn't installed just throw an error.

just my 2 cents.

[previ]

I think the problem some people are having here is it seems like just including this block doesn't actually install tiller because terraform doesn't know to do anything to the remote k8 system just from a provider block.

provider "helm" {
  install_tiller  = true
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  tiller_image    = "${var.tiller_image}"
}

However if you include this block it looks like tiller does get installed (although terraform never really tells you it does)

resource "helm_release" "mydatabase" {
  name  = "mydatabase"
  chart = "stable/mariadb"

  set {
    name  = "mariadbUser"
    value = "foo"
  }

  set {
    name  = "mardiadbPassword"
    value = "qux"
  }
}

This is kinda a weird setup as I've never seen a terraform provider block actually install anything remotely. I think a better setup would be to remove the "install_tiller" argument and create a resource that installs tiller instead. Then if you try to use the helm provider and tiller isn't installed just throw an error.

just my 2 cents.

That was exactly my case: I configured the helm provider in two separate terraform modules, but the first one called was not actually installing tiller for the reason you told.
A specific resource to install tiller would be more predictable.

[rporres]

This is kinda a weird setup as I've never seen a terraform provider block actually install anything remotely.

@cwiggs It is indeed weird. The plan is to deprecate that as soon as we have a tiller resource that meets all the expectations. See https://github.com/terraform-providers/terraform-provider-helm/pull/203#issuecomment-474933490

[Sudharma]

Any dates when this will be fixed?

[Tenseiga]

doesnt work for me either... followed all the strategies..

[mcuadros]

Closing this issue since is making reference to a version based on Helm 2, if this is still valid to the master branch please reopen it. Thanks.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!