Terraform-aws-eks: aws_auth doesn't seem to be applied to my EKS cluster

[johnraber]

I have issues

I'm submitting a...

*[X] possible bug

Hello,
I'm currently using v4.0.2 of this module on terraform v0.11.14. I'm running the EKS with private endpoint only. Passing in an IAM role via 'map_roles' and set 'map_roles_count'. The output 'config_map_aws_auth' is generated properly but when I check the k8's environment, I do not see the 'aws-auth' config map applied to the environment. Once I manually $kubectl apply -f config_map_aws_auth.yaml to the cluster, I'm able to use the bastion host with the IAM role that I passed in.
My expectation, is that the 'aws-auth' configmap would have been applied by this module?

module "cluster" {
  # https://github.com/terraform-aws-modules/terraform-aws-eks
  source          = "terraform-aws-modules/eks/aws"
  version         =  "4.0.2"
  cluster_name    = "${var.cluster_name}"
  cluster_version = "1.13"

  subnets = [
    "${var.private_subnet_ids}",
  ]

  vpc_id          = "${var.vpc_id}"
  manage_aws_auth = true
  write_aws_auth_config = "true"
  map_roles = "${var.map_roles}"
  map_roles_count = "${length(var.map_roles)}"

  cluster_create_security_group   = true
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false
.......
}

Thanks for all the hard work here.

[houqp]

This is due to cluster_endpoint_public_access set to false. When manage_aws_auth is enabled, it just runs a local kubectl command to add the config map, which means the machine where you run your terraform command needs have access to the internal VPC when public access is disabled.

I believe this is a bug and the aws_auth apply resource should throw out an error when kubectl command failed after retry.

[johnraber]

@houqp While debugging, I've also run an apply with cluster_endpoint_public_access = true and I still don't see the 'aws-auth' config map applied to the cluster.

[houqp]

From the output log, you should see the error from kubectl run if it was not created successfully, look for line prefixed with something like null_resource.update_config_map_aws_auth[0]: Provisioning with 'local-exec'. Might be helpful to paste the error here.

[johnraber]

Executed a clean apply of a new workspace with TF_LOG=TRACE and found no errors in terraform.log file.
Set my EKS endpoint to public (via console) and ran:

$ kubectl --kubeconfig kubeconfig -n kube-system get cm

Verified that the configmap 'aws-auth' was not written.

Next, executed a clean apply of a new workspace with TF_LOG=TRACE and found no errors in terraform.log file. However, this time setting 'cluster_endpoint_public_access' = true. Verified that the configmap 'aws-auth' was actually written to the kube-system namespace correctly.

Takeaway, the lack of applying the 'aws-auth' configmap ( when 'cluster_endpoint_public_access' = false ) is either a bug or is working as intended.

@houqp thx for the assistance

[max-rocket-internet]

Can you try this? https://github.com/terraform-aws-modules/terraform-aws-eks/issues/420#issuecomment-510956819

[johnraber]

@max-rocket-internet I will attempt to dig a little further but it may take a bit of time till I can revisit this. I'm in the middle of upgrading to TF v0.12 so this issue may go away. Let me know if you want me to close this?

[cmrust]

Hey @max-rocket-internet,

Love the project. Regarding the proposed solution, https://github.com/terraform-aws-modules/terraform-aws-eks/issues/420#issuecomment-510956819, those suggested values are already the defaults for the terraform-aws-vpc module. These changes didn't make a difference for me.

I've been running into the same issue, presumably as the others filing issues recently, where ASG instances aren't connecting to the cluster as nodes upon cluster standup. Flipping cluster_endpoint_public_access to true, from the default of false, then destroying and recreating the cluster resolves that.

Given that default value, I think option 1 here in https://github.com/terraform-aws-modules/terraform-aws-eks/issues/356#issuecomment-486676380, or something similar, should be documented in the README.md. As of now the defaults won't stand up a working cluster.

[max-rocket-internet]

@cmrust

Flipping cluster_endpoint_public_access to true, from the default of false,

As of now the defaults won't stand up a working cluster.

It is set to true though: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/variables.tf#L226

[cmrust]

My apologies @max-rocket-internet, testing too many configurations last week. 😓

[aimanparvaiz]

I am running in to the same issue. By default cluster_endpoint_public_access is true. From AWS console also I see:

API server endpoint access 
Private access   
Disabled

I am using the following as map_roles:

module "eks" {
  source = "modules/eks"
  cluster_name = "${var.team}-${var.env}-eks"
  vpc_id = "${var.vpc}"
  subnets = "${concat(var.private_subnets_ids, var.public_subnets_ids)}"
  config_output_path = "${pathexpand("~/.kube")}/"
  worker_group_count = 1
  #map_roles = "${var.map_roles}"
  map_roles = [
    {
    role_arn = "arn:aws:iam::XXXXXXXX:role/${var.team}DevOps"
    username = "${var.team}DevOps"
    group    = "${var.team}DevOps"
    },
    {
    role_arn = "arn:aws:iam::077854988703:role/${var.team}TeamDev"
    username = "${var.team}TeamDev"
    group    = "${var.team}TeamDev"
    },
    {
    role_arn = "arn:aws:iam::0XXXXXXXX:role/TeamDev"
    username = "${var.team}TeamDev"
    group    = "${var.team}TeamDev"
    }
  ]
  cluster_version = "${var.cluster_version}"
  worker_groups = "${local.worker_groups}"
  tags = {
    Terraform = "true"
    Environment = "${var.env}"
    Team = "${var.team}"
  }

}

I notice that a local config map does get created but it doesn't have the roles passed in the map_roles var.
Any suggestions would be appreciated.
Terraform v0.11.13

[dpiddockcmp]

Hi @aimanparvaiz
In your case, with Terraform 0.11 and version 2 of the module, you also need to set the map_roles_count variable to the number of roles. This is to work around some of the issues 0.11 had with the count attribute and variables referencing resources.

[dannyburke1]

Hi @aimanparvaiz
In your case, with Terraform 0.11 and version 2 of the module, you also need to set the map_roles_count variable to the number of roles. This is to work around some of the issues 0.11 had with the count attribute and variables referencing resources.

Been banging my head against a wall all morning to then discover this comment, thanks 😄