Terraform-provider-kubernetes: Feature Request: Support exec Authentication (client-go Credential Plugins)

[bflad]

Kubernetes implemented support for an exec authentication provider, where the client can generically reach out to another binary to retrieve a valid authentication token. This is used for supporting authentication models such as LDAP credentials or AWS IAM credentials.

This feature is alpha in Kubernetes 1.10, however it is the required model for working with AWS EKS.

Documentation on client-go credential plugins can be found here: https://kubernetes.io/docs/admin/authentication/#client-go-credential-plugins

Presumably a few things will need to happen to land this support:

• Fix up and merge anything related to vendoring the Kubernetes 1.9 client libraries: #117
• Vendor the Kubernetes 1.10 client libraries, existing PR: #162

Optionally:

• Add attributes in the provider configuration to manage exec authentication instead of requiring an on-disk kubeconfig for this setup

[tristanpemble]

EDIT (Mar 2019) -- this hack is unnecessary now due to the eks_cluster_auth data source (see bflad's comment)!

---

I've gotten around this in a rather hacky way to get EKS initialized with workers:

resource "aws_eks_cluster" "default" {
  name = "my-cluster"

  # ...
}

data "external" "heptio_authenticator_aws" {
  program = ["bash", "${path.module}/authenticator.sh"]

  query {
    cluster_name = "my-cluster"
  }
}

provider "kubernetes" {
  host                   = "${aws_eks_cluster.default.endpoint}"
  cluster_ca_certificate = "${base64decode(aws_eks_cluster.default.certificate_authority.0.data)}"
  token                  = "${data.external.heptio_authenticator_aws.result.token}"
  load_config_file       = false
}

and the authenticator.sh:

#!/bin/bash
set -e

# Extract cluster name from STDIN
eval "$(jq -r '@sh "CLUSTER_NAME=\(.cluster_name)"')"

# Retrieve token with Heptio Authenticator
TOKEN=$(heptio-authenticator-aws token -i $CLUSTER_NAME | jq -r .status.token)

# Output token as JSON
jq -n --arg token "$TOKEN" '{"token": $token}'

with this method I was able to use the kubernetes_config_map resource to configure Kubernetes:

resource "kubernetes_config_map" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }

  data {
    mapRoles = <<YAML
- rolearn: ${module.iam.node_role_arn}
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
YAML
  }
}

so that my workers come online with the EKS cluster in a single terraform apply :)

[aaronmell]

@tristanpemble Whenever I run your example, I get an unauthorized error. Any idea what might be the cause of that? I am not creating a config map in my example though, I have already created the config map manually.

If I do something like this https://blog.stigok.com/2018/06/25/aws-eks-kubernetes-terraform-system-anonymous-service-account.html That works, and terraform works.

Would rather use heptio for authentication rather than a hard coded token.

[sean-abbott]

I already had my workers coming up with the terraform instructions, but I can't quite figure out how to actually be able to use terraform control kubernetes yet.

TBH, I am trying to learn both of these things at once and I don't understand the k8s auth strcture though, so this may not be the right place to ask.

[danmaas]

Tip for those having trouble: try running
heptio-authenticator-aws token -i $NAME_OF_YOUR_CLUSTER
and see if it returns a working token. The part you're looking for is the string value for "token" in the returned JSON structure.

Temporarily hard-code that token inside your Terraform provider "kubernetes" block to test it.

Once you verify that Heptio can give you a working token, then you can figure out the data "external" trick.

I was originally encountering some problems until I realized that, because I use a non-default AWS profile from my ~/.aws/credentials, I needed to set the AWS_PROFILE environment variable for all runs of Terraform associated with that cluster. (I forgot that Terraform is going to invoke heptio-authenticator as a subprocess, and that subprocess needs to be set up with AWS credentials to be able to pull a token).

I did get @tristanpemble's example working for my own case. Thanks!

Although, I'm curious why there isn't a dependency ordering issue since heptio-authenticator-aws can't return a result until the cluster has been created. I guess it "just works" because Terraform evaluates the external block lazily, and you don't actually need that token until a point in time after the cluster is running.

[cosmopetrich]

Although, I'm curious why there isn't a dependency ordering issue since heptio-authenticator-aws can't return a result until the cluster has been created.

The heptio-authenticator-aws token -i $NAME_OF_YOUR_CLUSTER command doesn't communicate with Kubernetes. It just uses the AWS IAM API to sign some data. That signed data is what you pass to Kubernetes as a token. There's a brief description in the authenticator's README.

[danmaas]

Aha, makes sense, thanks. So the authenticator still gets you a token even if the named cluster doesn't actually exist.

[moosahmed]

FYI: The Terraform tutorial tells you to use aws-iam-authenticator. You're going to have to use that instead of heptio-authenticator-aws in TOKEN=$(heptio-authenticator-aws token -i $CLUSTER_NAME | jq -r .status.token) in the bash script for this to work properly.

[neilhwatson]

I tried this myself, got a token, but any attempt to create a k8s resource results in an 'unauthoried' error.

provider "aws" {}

terraform {
   backend "s3" {
      # Maintained by ansible in ./state/
      bucket = "example-terraform-eks-gir01"
      key    = "k8s"
      region = "us-east-1"
   }
}

# Remote backend for EKS
   data "terraform_remote_state" "eks" {
   backend = "s3" 
   config = {
      bucket = "example-terraform-eks-gir01"
      key    = "eks"
      region = "us-east-1"
   }
}

provider "kubernetes" {
   host = "${data.terraform_remote_state.eks.cluster_endpoint}"
   token = "${data.external.aws_iam_authenticator.result.token}"
   load_config_file = false
   cluster_ca_certificate
      = "${base64decode(data.terraform_remote_state.eks.cluster_cert.0.data)}"
}

data "external" "aws_iam_authenticator" {
   program = ["${path.module}/get_token.sh", "cluster"
      , "${data.terraform_remote_state.eks.cluster_name}"]
}

resource "kubernetes_config_map" "aws_auth" {
   metadata {
      name = "aws-auth"
      namespace = "kube-system"
   }
   data {
        "mapRoles" = <<MAPROLES
          - rolearn: ${data.terraform_remote_state.eks.node_iam_role}
            username: system:node:{{EC2PrivateDNSName}}
            groups:
              - system:bootstrappers
              - system:nodes
MAPROLES
   }
}

resource "kubernetes_service_account" "tiller" {
   depends_on = [ "kubernetes_config_map.aws_auth" ]
   metadata {
      name = "tiller"
      namespace = "kube-system"
   }
}

The external script:

#!/bin/sh

set -e

eval "$(jq -r '@sh "cluster=\(.cluster)"')"

token=$(aws-iam-authenticator token -i $cluster |jq -r '.status.token')

jq -n --arg token "$token" '{"token":$token}'

Output from terraform apply:

Error: Error applying plan:

1 error(s) occurred:

* kubernetes_config_map.aws_auth: 1 error(s) occurred:

* kubernetes_config_map.aws_auth: Unauthorized

2018-10-02T18:11:42.107Z [DEBUG] plugin: plugin process exited: path=/nwatson/k8s/.terraform/plugins/linux_amd64/terraform-provider-external_v1.0.0_x4
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


2018-10-02T18:11:42.110Z [DEBUG] plugin.terraform-provider-kubernetes_v1.2.0_x4: 2018/10/02 18:11:42 [ERR] plugin: plugin server: accept unix /tmp/plugin989680574: use of closed network connection
2018-10-02T18:11:42.117Z [DEBUG] plugin: plugin process exited: path=/nwatson/k8s/.terraform/plugins/linux_amd64/terraform-provider-kubernetes_v1.2.0_x4

[moosahmed]

I am not entirely whether extracting the eks end point and cert auth from the data resource works like that. I provisioned my eks directly using the aws provider. I would also check if the correct iam role policy attachments are initiated. see here How I did it: https://github.com/moosahmed/Stateful_Symphony/blob/master/terraform/modules/eks/cluster/cluster.tf

this works fine for me. good luck!

Also check rolearn: ${data.terraform_remote_state.eks.node_iam_role} if you are feeding the correct arn.
https://github.com/moosahmed/Stateful_Symphony/blob/master/terraform/modules/k8s/config_map/main.tf

[neilhwatson]

@moosahmed I've already provisioned the EKS cluster resource and nodes. Now I'm trying to provision services inside k8s, such as service accounts. Using kubectl works, but it's not idempotent, so I'm trying to use Terraform instead. No matter what I try the Terraform provider fails to authenticate to k8s.

[aaronmell]

@neilhwatson I had to create a token for terraform to use. My provider looks like this

provider "kubernetes" { host = "${data.aws_eks_cluster.eks_cluster.endpoint}" cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.eks_cluster.certificate_authority.0.data)}" token = "${data.aws_ssm_parameter.terraform_token.value}" load_config_file = false version = "= 1.1.0" }

[soasme]

Thanks for the hard working. My issue was gone after upgrading to v1.3.0.

[mbarrien]

To modify @neilhwatson 's response, a work around without relying on an external script to be checked in (but still requiring jq to be installed):

data "external" "aws-iam-authenticator" {
  program = ["sh", "-c", "aws-iam-authenticator token -i `jq -r .cluster_name` | jq -r -c .status"]

  query = {
    cluster_name = "${data.terraform_remote_state.eks.cluster_name}"
  }
}

My org also sticks in AWS_PROFILE=${var.aws_profile} aws-iam-authenticator... for the command line when we use profiles in a multi-account setting.

[alexsn]

@mbarrien - What's the query for? Why not just do:

data "external" "aws_iam_authenticator" {
  program = ["sh", "-c", "aws-iam-authenticator token -i ${data.terraform_remote_state.eks.cluster_name} | jq -r -c .status"]
}

[stephen-dahl]

for windows users

data "external" "kubectlToken" {
  program = [
    "powershell",
    "-Command",
    "& {aws-iam-authenticator token -i ${aws_eks_cluster.eks.cluster_name} | jq -r -c .status.token}",
  ]
}

requires you have installed jq. choco install jq if you use chocolaty

[ostmike]

@aaronmell - would be interested to see what the code looks like for data.aws_ssm_parameter.terraform_token.value if possible. The idea of not relying on the aws-iam-authenticator is appealing

[aaronmell]

@ostmike I created this in kubernetes

`apiVersion: v1
kind: ServiceAccount
metadata:
name: terraform

namespace: kube-system

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: terraform
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:

• kind: ServiceAccount
  name: terraform
  namespace: kube-system
  `

then i store the token from the terraform SA in SSM

[omeid]

FYI, if you're using EKS, there is terraform-providers/terraform-provider-aws#4904 which is much more seamless and "terraform way".

[bflad]

Related to the above comment with EKS, https://github.com/terraform-providers/terraform-provider-aws/pull/7438 (a continuation of https://github.com/terraform-providers/terraform-provider-aws/pull/4904) has been merged and will release in version 1.58.0 of the Terraform AWS provider, which contains a new aws_eks_cluster_auth data source, e.g.

data "aws_eks_cluster" "example" {
  name = "example"
}

data "aws_eks_cluster_auth" "example" {
  name = "${data.aws_eks_cluster.example.name}"
}

provider "kubernetes" {
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  token                  = "${data.aws_eks_cluster_auth.example.token}"
  load_config_file       = false
}

[kcroaker]

@bflad Thank you. Worked a charm!

[bflad]

A work in progress pull request for the last part of this feature request can be found here: #396

It should enable something like the following for AWS EKS:

# Functionality under development - details may change during implementation
provider "kubernetes" {
  cluster_ca_certificate = "${base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)}"
  host                   = "${data.aws_eks_cluster.cluster.endpoint}"
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["token", "-i", "${data.aws_eks_cluster.cluster.name}"]
    command     = "aws-iam-authenticator"
  }
}

[bbc88ks]

@alexsomesan Can you release a new version that includes this PR, please?