Terraform-provider-helm: Server not used from kubeconfig file

[ukphillips]

When using the kubeconfig file - it does not appear to use the server in the host in the file, and is using localhost. This may be intentional, but the documentation seems to imply that the kubeconfig can be used alone to connect to a cluster. Currently I need to set the host as well

Terraform Version

0.11.13

Affected Resource(s)

• helm_resource

Terraform Configuration Files

data "azurerm_key_vault_secret" "kubeconfig" {
  name      = "admin-kubeconfig"
  vault_uri = "https://${var.keyvault}.vault.azure.net/"
}
resource "local_file" "kubeconfig" {
    content     = "${base64decode(data.azurerm_key_vault_secret.kubeconfig.value)}"
    filename = "${path.module}/kubeconfig.yaml"
}



provider "helm" {
  kubernetes {
    config_path = "${path.module}/kubeconfig.yaml"
  }
  debug= true
  namespace       = "kube-system"
}

resource "helm_release" "mychart" {
  name          = "${var.release_name}"
  chart         = "${var.chart_name}"
  version       = "${var.chart_version}"
  namespace     = "${var.namespace}"
  recreate_pods = "${var.recreate_pods}"

}

Panic Output

Error: Error applying plan:

1 error(s) occurred:

• helm_release.mychart: 1 error(s) occurred:

• helm_release.mychart: error installing: Post http://localhost/apis/extensions/v1beta1/namespaces/kube-system/deployments: dial tcp 127.0.0.1:80: connect: connection refused

Expected Behavior

This would be executed against the URL in the kubeconfig

Actual Behavior

Executing against localhost

Steps to Reproduce

1. terraform apply

[dineshba]

Hi @ukphillips I tried to reproduce this error. But couldn't.

resource "local_file" "one" {
  content = "${file("/Users/dineshbalasubramanian/.kube/config")}"
  filename = "/Users/dineshbalasubramanian/projects/trails/helm/terraform/temp.yaml"
}


provider "helm" {
     kubernetes {
         config_path = "/Users/dineshbalasubramanian/projects/trails/helm/terraform/temp.yaml"
     }
 }

resource "helm_release" "myapp" {
  name      = "myapp"
  chart     = "stable/redis"
  namespace = "myapp"
}

Hope I am using similar tf file as yours. Are you able to reproduce the error?

[virmanv]

Hi @ukphillips I was trying to do the same thing and I see the same error

kubernetes_service_account.tiller: Post https://localhost:6443/api/v1/namespaces/kube-system/serviceaccounts: dial tcp [::1]:6443: connect: connection refused

provider "helm" {
    version = "~> 0.7"
  install_tiller  = true
  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"

  kubernetes {
    config_path = "~/.kube/config-eks"
  }
}

Its not picking up the config_path. But when I explicitlly add the environment variable export KUBECONFIG=~/..kube/config-eks then it works

[etheleon]

Hi @virmanv, it's a different error message as @ukphillips 's error

http://localhost/apis/extensions/v1beta1/namespaces/kube-system/deployments: dial tcp 127.0.0.1:80: connect: connection refused

kubernetes_service_account.tiller: Post https://localhost:6443/api/v1/namespaces/kube-system/serviceaccounts: dial tcp [::1]:6443: connect: connection refused

but both seem to sending the requests to localhost. I tried adding the ENV variable but am still connecting to localhost.

[adurai81]

I'm seeing the same issue as well. It connects to localhost.

error installing: Post http://localhost/apis/extensions/v1beta1/namespaces/kube-system/deployments: dial tcp [::1]:80: connect: connection refused

Any help on this ?

[virmanv]

Hello @adurai81

I have fixed this issue by adding kubernetes provider like this

provider "kubernetes" {
  token                  = "${data.aws_eks_cluster_auth.eks.token}"
  host                   = "${var.cluster_endpoint}"
  cluster_ca_certificate = "${base64decode(varr.cluster_certificate_authority_data)}"
}

This has fixed my issue.

[sanket-bengali]

@ukphillips @adurai81 @virmanv
Another way to add Kubernetes provider (without kubeconfig file) is :
(FYI, this is for AWS, there should be similar way for Azure and others.)

provider "kubernetes" {
  cluster_ca_certificate = "${var.cluster_ca_certificate}"
  host                   = "${var.cluster_endpoint}"
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws-iam-authenticator"
    args        = ["token", "-i", "${var.cluster_name}"]
  }
}

Helm provider works like this :

provider "helm" {
  install_tiller                  = true
  namespace                       = "kube-system"
  service_account                 = "tiller"
  tiller_image                    = "gcr.io/kubernetes-helm/tiller:v2.13.1"

  kubernetes {
    config_path            = "${var.kubeconfig_filename}"
    load_config_file       = false
  }
}

--------------- OR ---------------

1. Get token :

data "external" "aws_iam_authenticator" {
  program = ["sh", "-c", "aws-iam-authenticator token -i ${var.cluster_name} | jq -r -c .status"]
}

1. Use in Helm provider :

provider "helm" {
  install_tiller                  = true
  namespace                       = "kube-system"
  service_account                 = "tiller"
  tiller_image                    = "gcr.io/kubernetes-helm/tiller:v2.13.1"

  kubernetes {
    host                   = "${var.cluster_endpoint}"
    cluster_ca_certificate = "${var.cluster_ca_certificate}"
    token                  = "${data.external.aws_iam_authenticator.result.token}"
    load_config_file       = false
  }
}

[eskp]

Thanks @sanket-bengali 👍

In the second case token becomes token = "${data.external.aws_iam_authenticator.result.token}"

[sanket-bengali]

Thanks @sanket-bengali 👍

In the second case token becomes token = "${data.external.aws_iam_authenticator.result.token}"

Thank you for correction @eskp

[imranismail]

We should support exec authentication from helm provider

[rporres]

@ukphillips Without seeing the content of your kube config file it's difficult to know. I would inspect and test using helm command the file created to make sure that it's being correctly created. But as commented in this issue, config_path option works using a properly created kube config file.

As pointed in this issue, it's always better to avoid relying on intermediate files and configure things directly via certificates, tokens, etc

I'm closing this. Please feel free to reopen if you can provide further information to help us debugging.

[vrathore18]

@sanket-bengali @virmanv : I am also facing the same issue.

Get http://localhost/api/v1/namespaces/kube-system/serviceaccounts/tiller: dial tcp 127.0.0.1:80: connect: connection refused

Could you please help me. how you guy passing the values of ${var.cluster_endpoint} and ${var.cluster_ca_certificate} as it will be generated automatically from eks module.

my eks module.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "6.0.2"

  cluster_name = "${var.name}"
  subnets      = "${module.vpc.private_subnets}"
  vpc_id       = "${module.vpc.vpc_id}"

  kubeconfig_aws_authenticator_additional_args = ["-r", "arn:aws:iam::${var.target_account_id}:role/terraform"]

  worker_groups = [
    {
      instance_type        = "${var.eks_instance_type}"
      asg_desired_capacity = "${var.eks_asg_desired_capacity}"
      asg_max_size         = "${var.eks_asg_max_size}"
      key_name             = "${var.key_name}"
    },
  ]

  map_accounts = ["${var.target_account_id}"]

  map_roles = [
    {
      rolearn = "${format("arn:aws:iam::%s:role/admin", var.target_account_id)}"
      username = "${format("%s-admin", var.name)}"
      groups    = ["system:masters"]
    },
  ]

  #map_accounts_count = "1"
  #map_roles_count    = "1"

  # don't write local configs, as we do it anyway
  write_kubeconfig      = "false"

  # You need to write an aws_auth_config to let your nodes join the cluster !
  write_aws_auth_config = "true"
}

resource "local_file" "kubeconfig" {
  content  = "${module.eks.kubeconfig}"
  filename = "./.kube_config.yaml"
}

and provider.tf

provider "kubernetes" {
  config_path = "/.kube_config.yaml"
  version = "~> 1.10.0"
}

provider "helm" {
  version = "~> 0.10.4"
  install_tiller  = true
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  namespace       = "${kubernetes_service_account.tiller.metadata.0.namespace}"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"

  kubernetes {
    config_path = "/.kube_config.yaml"
  }
}

[sanket-bengali]

@vrathore18 :
You can access cluster_endpoint and cluster_ca_certificate from the output of aws_eks_cluster :

resource "aws_eks_cluster" "default" {
     ................
}

locals {
  certificate_authority_data_list          = "${coalescelist(aws_eks_cluster.default.*.certificate_authority, list(list(map("data", ""))))}"
  certificate_authority_data_list_internal = "${local.certificate_authority_data_list[0]}"
  certificate_authority_data_map           = "${local.certificate_authority_data_list_internal[0]}"
  certificate_authority_data               = "${local.certificate_authority_data_map["data"]}"
}

Here, it is accessed locally in the same module.
Alternatively, you can store in the output variables (and pass as input in another module) :

output "eks_cluster_certificate_authority_data" {
  description = "The base64 encoded certificate data required to communicate with the cluster"
  value       = "${local.certificate_authority_data}"
}

output "eks_cluster_endpoint" {
  description = "The endpoint for the Kubernetes API server"
  value       = "${join("", aws_eks_cluster.default.*.endpoint)}"
}

Reference : terraform-aws-eks-cluster

Additionally, note that the generated certificate is in base64encoded form, and to pass in the Kubernetes provider, it needs to be in decoded form :

provider "kubernetes" {
  cluster_ca_certificate = "${base64decode(var.cluster_ca_certificate)}"
  host                   = "${var.cluster_endpoint}"
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws-iam-authenticator"
    args        = ["token", "-i", "${var.cluster_name}"]
  }
}