Eksctl: Provide a mechanism in eksctl to deploy a fully private EKS/VPC environment

[mreferre]

Before creating a feature request, please search existing feature requests to see if you find a similar one. If there is a similar feature request please up-vote it and/or add your comments to it instead

Why do you want this feature?
A lot of Enterprise customers (especially in the financial sector but not limited to that) want to deploy fully private EKS environments (with no Internet connectivity at-all). These environments (including the VPC hosting them) should not have external connectivity but should allow for hybrid scenarios (i.e. consuming both the EKS service as well as workloads running on it from on-prem via a VPN connection).

What feature/behavior/change do you want?
Ideally a new flag (or a set of flags) that, when used together, achieve the result above.

Today people are implementing this pattern using CloudFormation, bash scripts or a combination of those (e.g. this is an example).

There are a few existing issues that cover some of this (e.g. 1230 and 1358) but the idea about this feature request is to go a step further and not only deploy an EKS cluster that is accessible from within the VPC but create a complete self-contained specific end-to-end setup (including the VPC itself) that does not have Internet connectivity at-all.

[henrybravo]

+1 for this, will be very much appreciated by many users

[sayboras]

definitely this feature will help to check all security boxes. Just want to share my experience, I eventually went with custom resource to do all post cluster creation steps. However, one more issue still needs to be addressed https://github.com/aws/containers-roadmap/issues/607 (not related to eksctl though).

[mreferre]

Thanks @sayboras for chiming in. Yes https://github.com/aws/containers-roadmap/issues/607 is being worked on.

[kleemae]

+1

[maxbraun]

+1

[martina-if]

Related information in #80 and #1230

[martina-if]

Start by making a proposal on how we want to tackle this

[michaelbeaumont]

https://github.com/weaveworks/eksctl/issues/747 is potentially relevant

[mreferre]

Start by making a proposal on how we want to tackle this

Sorry Martina was this a comment to me? I have provided some background in the ticket but let me know if you want/need me to expand.

[martina-if]

Hi @mreferre No, sorry, that was unclear :sweat_smile: That was a quick note I added for ourselves in the team during a planning meeting.

[mreferre]

As an additional piece of information, we have just updated our documentation to call out more specifically the use case of how to run clusters in VPC w/out outbound connectivity. It's in here:

https://docs.aws.amazon.com/eks/latest/userguide/private-clusters.html
https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html
https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html

[cPu1]

We have started working on this feature, and the UX and implementation details are documented below.

To create a fully-private cluster, a top-level field privateCluster is added to ClusterConfig:

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"

privateCluster.enabled enables creation of a fully-private cluster. When this field is set, eksctl creates no public subnets, but only private subnets that have no outbound internet access (i.e., no NAT gateway or NAT instance). To enable worker nodes to access AWS services privately, eksctl creates VPC endpoints for the following services:

• Interface endpoints for ECR (both ecr.api and ecr.dkr) to pull container images (AWS CNI plugin etc)
• A gateway endpoint for S3 to pull the actual image layers
• An interface endpoint for EC2 because the aws-cloud-provider integration accesses it
• An interface endpoint for STS to support Fargate and IAM Roles for Services Accounts (IRSA)
• An interface endpoint for CloudWatch logging (logs) if CloudWatch logging is enabled

These VPC endpoints are essential for a functional private cluster, and as such, eksctl does not support configuring or disabling them. However, a cluster might need to access other AWS services privately (e.g., Autoscaling required by the Cluster Autoscaler). These services can be specified in privateCluster.additionalEndpointServices, which will instruct eksctl to create a VPC endpoint for each of them. The list of additional endpoint services that can be enabled for private access will be limited to a few services initially.

privateNetworking for both self-managed and EKS-managed nodegroups must be set explicitly. It's an error to leave privateNetworking unset, as it's not possible to create public nodegroups in a fully-private cluster as there's no public subnet. Rather than enabling this field by default for a fully-private cluster, we require users to explicitly set it to make the behaviour explicit and avoid confusion.

nodeGroups:
- name: ng1
  instanceType: m5.large
  desiredCapacity: 2
  # privateNetworking must be explicitly set for a fully-private cluster
  # Rather than defaulting this field to true for a fully-private cluster, we require users to explicitly set it
  # to make the behaviour explicit and avoid confusion.
  privateNetworking: true

Furthermore, it is an error to set either clusterEndpoints.publicAccess or clusterEndpoints.privateAccess, as a fully-private cluster can have private access only, and allowing modification of these fields can break the cluster.

vpc:
  clusterEndpoints:
#    It is an error to set either of the following two fields, as a fully-private cluster defaults to private-only access
#    publicAccess: true
#    privateAccess: false

Combining the above snippets, a ClusterConfig for a fully-private cluster would look like this:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: private-cluster
  region: us-west-2

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"


nodeGroups:
- name: ng1
  instanceType: m5.large
  desiredCapacity: 2
  # privateNetworking must be explicitly set for a fully-private cluster
  # Rather than defaulting this field to true for a fully-private cluster, we require users to explicitly set it
  # to make the behaviour explicit and avoid confusion.
  privateNetworking: true

managedNodeGroups:
- name: m1
  instanceType: m5.large
  desiredCapacity: 2
  privateNetworking: true

It's important to note that the initial implementation will have a limitation where eksctl will initially create the cluster with both public and private endpoints access enabled, because eksctl needs access to the Kubernetes API server to allow self-managed nodes to join the cluster and to support GitOps and Fargate. After these operations have completed, eksctl will switch the cluster endpoint access to private-only. This additional update does mean that creation of a fully-private cluster will take longer than for a standard cluster. In the future, we may consider switching to a VPC-enabled Lambda function to perform these API operations.

~It goes without saying that for subsequent operations on the cluster, eksctl must be run from within the cluster's VPC or a peered VPC (or some other means).~ This is obsoleted by this comment: https://github.com/weaveworks/eksctl/issues/1765#issuecomment-654263700

[mreferre]

The only thing we need to consider (but I doubt this will change the way you are designing this) is that customers that have the need for a fully private cluster also has a hybrid setup with on-prem by means of a Direct Connect. As I said, I don't think this changes the approach you are using but when we say "... subsequent operations on the cluster, eksctl must be run from within the cluster's VPC or a peered VPC (or some other means)."
.... that use case should definitely include/consider the ramifications of connecting to it from on-prem.

[opensourceonly]

@martina-if @mreferre when it comes to connecting to TGW, it can be nice to have TGW attachments created
as part of cluster creation so that we do not want to do them manually later .

[jebbens]

This is great news. Will this capability allow you to deploy the private cluster inside an existing subnet or will it only create the subnet from scratch?

[sayboras]

Agreed with @jebbens that creating the same feature with _existing VPC_ is quite important for adoption.

Just want to express my opinion, I feel like the current schema is missing top most level to describe _cluster_ object, so I feel with this effort, it's a good chance to define for the same (e.g. the below _privateCluster_ can be just changed to some generic object such as _Cluster_). One benefit is to clean up all the config options related to API server or cluster level.

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"

[cPu1]

This is great news. Will this capability allow you to deploy the private cluster inside an existing subnet or will it only create the subnet from scratch?

@jebbens yes, it will. You can supply an existing VPC and subnets, and eksctl will create VPC endpoints in the supplied VPC and modify route tables for the supplied subnets. There will be a usage doc to address this use case.

[cPu1]

A slight update to the usage of eksctl post cluster creation:

Post cluster creation, not all eksctl commands will be supported, especially commands that need access to the Kubernetes API server. Creating managed nodegroups will continue to work, however, creating self-managed nodegroups will not work as it needs access to the API server. Even if the command is run from within the cluster's VPC, a peered VPC or using some other means like AWS Direct Connect, some commands may fail because they'll need private access to the EKS API (DescribeCluster), and the AWS EKS service does not offer an interface endpoint. If eksctl can reach the EKS API server endpoint via its private address, and has outbound internet access (for EKS:DescribeCluster), all eksctl commands should work.