Terraform-aws-eks: How to launch worker in private subnet

[tonyxiao]

In the getting started example

module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"
  version            = "1.14.0"
  name               = "test-vpc"
  cidr               = "10.0.0.0/16"
  azs                = ["${data.aws_availability_zones.available.names[0]}", "${data.aws_availability_zones.available.names[1]}", "${data.aws_availability_zones.available.names[2]}"]
  private_subnets    = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets     = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
  enable_nat_gateway = true
  single_nat_gateway = true
  tags               = "${merge(local.tags, map("kubernetes.io/cluster/${local.cluster_name}", "shared"))}"
}

module "eks" {
  source             = "../.."
  cluster_name       = "${local.cluster_name}"
  subnets            = ["${module.vpc.public_subnets}", "${module.vpc.private_subnets}"]
  tags               = "${local.tags}"
  vpc_id             = "${module.vpc.vpc_id}"
  worker_groups      = "${local.worker_groups}"
  worker_group_count = "1"
  map_roles          = "${var.map_roles}"
  map_users          = "${var.map_users}"
  map_accounts       = "${var.map_accounts}"
}

Both private and public subnets are passed to eks module as a single variable. How does eks module determine which subnets are public and which subnets are private and thus launch worker into private subnets only?

[dpiddockcmp]

It doesn't. The list of subnets is passed directly to ASG. This is a poor example. Workers will be spun up randomly between public and private subnets.

My solution is to only pass the private subnets to the EKS module, if you only want private nodes. For kubenetes to create public load balancers, the public subnets must be tagged with kubernetes.io/cluster/$cluster: shared, as in the example using VPC module.

Alternatively, pass private subnets as a comma list to workers_group_defaults

[max-rocket-internet]

This is a poor example. Workers will be spun up randomly between public and private subnets

100%!

Thanks for the issue @tonyxiao and the PR @dpiddockcmp

[tonyxiao]

@dpiddockcmp to confirm, the master doesn't need to be inside a public subnet for eks to work? Or are masters always in the public subnets and isn't otherwise configurable?

[dpiddockcmp]

@tonyxiao EKS master instances are not created within your VPC. They are in a network controlled by AWS and are always publicly accessible.

From some experimenting (as it doesn't appear to be publicly documented) the EKS service creates two ENIs in your VPC, in different AZs, for nodes to communicate with the master. I'm not sure what decision process is used to pick the subnets or whether it's random. So as long as the master is given at least two subnets, in different AZs, it will work.

Kubernetes requires Tags on subnets in order to add them to ELBs. EKS does this for you when you are creating a cluster. We have to add the subnet tags to terraform configuration otherwise they may be removed on subsequent terraform applys.

[tonyxiao]

@dpiddockcmp something else I just realized. As you mentioned EKS will tag the subnets you assign to it with kubernetes.io/cluster/$clusterName = shared. and use those subnets when creating ELBs. If the subnets given to EKS are private subnets, that means ELBs are created in private subnets? Would that work?

[dpiddockcmp]

Hi @tonyxiao.

If you request a public LoadBalancer and there are no public subnets tagged then Kubernetes will not create a load balancer.

You can create internal load balancers with this annotation on the Service. It will be created on the tagged private subnets:

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

[nandeeshb09]

Hi Dpiddockcmp,

Please help on this ,I have EKS cluster with public and private subnets.
I want worker nodes should be in private subnets and have created.

But internal load balancer is picking up public subnet still even though if we have tagged.

I have created tag in private subnet with below tag,but still internal load balancer is picking up public subnet.
Key | Value
kubernetes.io/role/internal-elb | 1

please guide on this...Im stuck at this work from last week...Pleaseeee..