Cluster-api-provider-aws: [Umbrella] Support for EKS

[richardcase]

/kind feature

Describe the solution you'd like
I'd like to see the ability to create/manage EKS clusters using Cluster API. This is the parent issue for implementing EKS. There are sub tasks and issues listed below for implementation of specific functionality.

Anything else you would like to add:
This comes up quite a bit in discussions with customers around capi. Especially in hybrid on-prem/cloud type environments.

This relates to https://github.com/kubernetes-sigs/cluster-api/issues/980 and also the associated proposal https://github.com/kubernetes-sigs/cluster-api/pull/2045

Tasks:

• [x] Refactor ClusterScope and services that depend on it - issue #1796 (Now merged to main)
• [x] EKS controlplane support - issue #1787
• [x] EKS AMI lookup support - issue #1778
• [x] EKS bootstrap provider - issue #1779
• [x] EKS support in clusterawsadm- issue #1780
• [x] EKS unmanaged node support - issue #1784
• [ ] EKS managed node group support - issue #1785
• [ ] EKS Fargate profiles support - issue #1786
• [x] Create template flavours to support EKS - issue #1828
• [x] Update documentation for AWSManagedCluster kubeconfigs - issue #1851
• [ ] Support granting additional IAM roles access to the EKS cluster - issue #1850

[detiber]

/priority important-longterm

[richardcase]

/assign @richardcase

[detiber]

@richardcase I haven't followed along with the Managed proposal linked overly well, but I would expect that it should be possible to implement a Managed kubernetes provider leveraging a Control Plane provider and a MachinePool provider targeted towards EKS. See https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/482 and the related discussion there.

[richardcase]

Thanks @detiber - i was was just reading your comments on the PR in the Azure provider. I'll follow the approach you suggest.

Would you suggest that the managed control plan provider code be placed in controlplane\eks (or controlplane\managed) in this repo?

[detiber]

@richardcase lets do controlplane/eks since it's more aligned with the implementation.

[richardcase]

@detiber - do you think the eks control plane should reuse the services (for example) or should it have its own specific versions? If we re-use then we would have to use the cluster scope for example and that expects a AWSCluster, which we could create with the control plane controller or we could refactor to work with multiple scopes. Either way i'm adding an eks service.

[detiber]

@richardcase I think it would make sense to re-use where possible. We should probably consider refactoring common services (on an as needed basis) to remove the need for a scope to better accommodate the reuse

[richardcase]

@detiber - i've started to refactor the ec2 service so that networking side could be reused. Instead of removing the need for a scope i split out the networking into a _network service_ and associated _clusternetwork scope_......i misread the last part of your comment about removing the need for a scope completely.

Shall i change what i've done to remove the new scope and change ReconcileNetwork to accept aNetworkParams (or something similar) instead? It would only be a minor change.

[rudoi]

/cc

[richardcase]

Work is progressing on this:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: EksControlPlane
metadata:
  name: capi-managed-test-control-plane
  namespace: default
spec:
  roleArn: arn:aws:iam::12345678:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS
  version: v1.15
---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: Cluster
metadata:
  name: capi-managed-test
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    kind: EksControlPlane
    name: capi-managed-test-control-plane
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSCluster
    name: capi-managed-test

A couple of questions:

1. A reference to a infrastructure cluster is required. Should i create AWSManagedCluster or update AWSCluster to have a IsManaged property?
2. Can the new controllers run under the existing manager or does it have to be a new 'control plane manager' under controlplane/eks? I've started with the later as it was agreed above and is similar to the kubeadm control plane and but it would be simpler to have it part of the existing manager with flags to enable.
3. If we stick with a separate eks control plane manager which reconciles EksControlPlane should it also reconcile any additional resources like AWSManagedCluster as well or would they be handled by the existing manager?

@detiber - would be good to get your thoughts on the above.

[richardcase]

A new subnet stratety has been implemented, along with nat gateway/eip logic. We should probably rationalise this with the current provider (see #1643 and #1323) to avoid too much divergence.
Work has also started on the actual EKSControlPlane.

[sethp-nr]

Hi @richardcase – @dthorsen & myself are taking some time over the next couple of weeks to focus on EKS. I'm curious if you think this work's in a place where we could help with it?

[richardcase]

@sethp-nr - its not quite in a place where collaboration would be easy. I paused work on it these last 2 days whilst i made a change to the default subnetting strategy (something i originally added for this). I also realised that some of the changes i have made are not needed so i need to undo those as there is a better way to handle it. I'll make those changes and let you know.

[sethp-nr]

Cool, no rush here – I'll plan to review #1721 when it's ready, and otherwise let me know if there's anything you think we can do on the CAPA side.

One thing we stumbled into coming from the other direction (looking to join CAPA Machines to an existing EKS cluster) was that since nobody ran kubeadm init, we were missing a few key resources to make that go. Is that something you're already looking at?

[richardcase]

I hadn't got to the machines part yet, so thats interesting. It will be interesting how we handle an unmanaged nodes, managed nodes and also supporting fargate.

[rudoi]

Wanted to add some tracking details, so I will update this comment as things happen:

• [ ] EKS controlplane support - PR https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/1724
• [ ] EKS AMI lookup support - issue https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/1778
• [ ] EKS bootstrap provider - issue https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/1779

[richardcase]

And also:

• [ ] EKS support in clusterawsadm- issue #1780

[richardcase]

@rudoi - i've moved the list of items up into the description

[randomvariable]

Given we have issues for all of these, and a lot of these are done, going to close this one as it's not useful for milestone tracking.

/close

[k8s-ci-robot]

@randomvariable: Closing this issue.

In response to this:

Given we have issues for all of these, and a lot of these are done, going to close this one as it's not useful for milestone tracking.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.