Cluster-api-provider-aws: Add documentation for how to create a control plane that spans multiple AZs

Created on 26 Jul 2019 · 17Comments · Source: kubernetes-sigs/cluster-api-provider-aws

I generate the file which is controlplane-machine-ha, and find that the master machineList file no specified detail of HA.

in fact, the user want to create a cluster is high available, that means is the cluster is HA with whole region, whole VPC. not create three master with same subnet.

/kind feature

kindocumentation kinfeature prioritimportant-soon

Source

Sn0rt

Most helpful comment

It is also possible to define multiple subnets when creating a Cluster with a "managed VPC" by setting Cluster.Spec.ProviderSpec.NetworkConfig.Subnets[*].AvailabilityZone for each Availability zone you wanted created in the managed VPC. When paired with setting the AvailabilityZone on the Machine, this allows you to pre-create Cluster and Machine resources without needing to pre-define the VPC or know the Subnets in advance before creating the Machines.

detiber on 26 Jul 2019

👍2

All 17 comments

To accomplish this today, you'll need to have 3 different subnets in a VPC and specify the AvailabilityZone field manually for each Machine.

vincepri on 26 Jul 2019

FWIW, what @vincepri suggested is (almost) what we do – we specify the subnets directly – and it works really well.

We've found that the examples in this repo are a helpful starting point, but we pretty quickly ran into the need to template out our own configs that matched our needs.

sethp-nr on 26 Jul 2019

detiber on 26 Jul 2019

👍2

I successfully created a cluster of ha, but in a semi-manual manner.
I create a three node HA-Cluster, but the subnet and route table set is manually.

1: create the yaml

guohao@buffer ~/workspace $ export CLUSTER_NAME=ha0x02
guohao@buffer ~/workspace $ ./aws/generate-yaml.sh
Done generating /Users/guohao/workspace/aws/out/cluster.yaml
Done generating /Users/guohao/workspace/aws/out/machines.yaml
Done generating /Users/guohao/workspace/aws/out/controlplane-machines-ha.yaml
Done generating /Users/guohao/workspace/aws/out/controlplane-machine.yaml
Done generating /Users/guohao/workspace/aws/out/machine-deployment.yaml
Done copying /Users/guohao/workspace/aws/out/addons.yaml
Generated credentials
Done writing /Users/guohao/workspace/aws/out/provider-components.yaml
WARNING: /Users/guohao/workspace/aws/out/provider-components.yaml includes credentials

2: create the cluster

guohao@buffer ~/workspace $ kubectl --kubeconfig boots create -f aws/out/cluster.yaml
cluster.cluster.k8s.io/ha0x02 created

3: login AWS Consloe

add two subnets and set the route table as same as the private subnet ( the nat route table is import to check

4: modify the controlplane-machines-ha.yaml

add the subnet

apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineList
items:
  - apiVersion: "cluster.k8s.io/v1alpha1"
    kind: Machine
    metadata:
      name: ha0x02-controlplane-0
      labels:
        cluster.k8s.io/cluster-name: ha0x02
        set: controlplane
    spec:
      versions:
        kubelet: v1.14.4
        controlPlane: v1.14.4
      providerSpec:
        value:
          apiVersion: awsprovider/v1alpha1
          kind: AWSMachineProviderSpec
          instanceType: "t2.medium"
          iamInstanceProfile: "control-plane.cluster-api-provider-aws.sigs.k8s.io"
          keyName: "guohao"
          subnet:
              id: subnet-003e25dc137b7ce45          
  - apiVersion: "cluster.k8s.io/v1alpha1"
    kind: Machine
    metadata:
      name: ha0x02-controlplane-1
      labels:
        cluster.k8s.io/cluster-name: ha0x02
        set: controlplane
    spec:
      versions:
        kubelet: v1.14.4
        controlPlane: v1.14.4
      providerSpec:
        value:
          apiVersion: awsprovider/v1alpha1
          kind: AWSMachineProviderSpec
          instanceType: "t2.medium"
          iamInstanceProfile: "control-plane.cluster-api-provider-aws.sigs.k8s.io"
          keyName: "guohao"
          subnet:
              id: subnet-04288a5f4bc6f05fa
  - apiVersion: "cluster.k8s.io/v1alpha1"
    kind: Machine
    metadata:
      name: ha0x02-controlplane-2
      labels:
        cluster.k8s.io/cluster-name: ha0x02
        set: controlplane
    spec:
      versions:
        kubelet: v1.14.4
        controlPlane: v1.14.4
      providerSpec:
        value:
          apiVersion: awsprovider/v1alpha1
          kind: AWSMachineProviderSpec
          instanceType: "t2.medium"
          iamInstanceProfile: "control-plane.cluster-api-provider-aws.sigs.k8s.io"
          keyName: "guohao"
          subnet:
              id: subnet-080196f9a70a12e6e

5: apply the controlplane

guohao@buffer ~/workspace $ kubectl --kubeconfig boots create -f aws/out/controlplane-machines-ha.yaml
machine.cluster.k8s.io/ha0x02-controlplane-0 created
machine.cluster.k8s.io/ha0x02-controlplane-1 created
machine.cluster.k8s.io/ha0x02-controlplane-2 created

6: check and wait machine to be ready

guohao@buffer ~/workspace $ kubectl --kubeconfig boots get machine
NAME                       PROVIDERID                    PHASE
...
ha0x02-controlplane-0      aws:////i-03a64a241a0cf80dd
ha0x02-controlplane-1
ha0x02-controlplane-2

guohao@buffer ~/workspace $ kubectl --kubeconfig boots get machine
NAME                       PROVIDERID                    PHASE
...
ha0x02-controlplane-0      aws:////i-03a64a241a0cf80dd
ha0x02-controlplane-1      aws:////i-0ec0432f7c44e8723
ha0x02-controlplane-2      aws:////i-035694abfa2e237b2

7: get the cluster status

get the yaml of cluster, and to check the aws.cluster.sigs.k8s.io/control-plane-ready is true or not.

guohao@buffer ~/workspace $ kubectl --kubeconfig boots get cluster ha0x02 -o yaml
apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    aws.cluster.sigs.k8s.io/control-plane-ready: "true"
    aws.cluster.sigs.k8s.io/infrastructure-ready: "true"
  creationTimestamp: "2019-08-08T06:35:39Z"

8: get the kubeconfig

guohao@buffer ~/workspace $ clusterctl alpha phases get-kubeconfig --provider aws --cluster-name ha0x02 --kubeconfig boots

9: get the master node of ha0x02 cluster

guohao@buffer ~/workspace $ kubectl --kubeconfig kubeconfig get node --all-namespaces
NAME                                       STATUS     ROLES    AGE     VERSION
ip-10-0-0-242.us-east-2.compute.internal   NotReady   master   3m35s   v1.14.4
ip-10-0-2-92.us-east-2.compute.internal    NotReady   master   78s     v1.14.4
ip-10-0-3-142.us-east-2.compute.internal   NotReady   master   30s     v1.14.4

10: apply the ADDON

guohao@buffer ~/workspace $ kubectl --kubeconfig kubeconfig apply -f aws/out/addons.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
deployment.extensions/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

11: check the master node status

guohao@buffer ~/workspace $ kubectl --kubeconfig kubeconfig get node --all-namespaces
NAME                                       STATUS   ROLES    AGE     VERSION
ip-10-0-0-242.us-east-2.compute.internal   Ready    master   11m     v1.14.4
ip-10-0-2-92.us-east-2.compute.internal    Ready    master   9m15s   v1.14.4
ip-10-0-3-142.us-east-2.compute.internal   Ready    master   8m27s   v1.14.4

12: to check a work node

use machinedeployment to scale a node.

guohao@buffer ~/workspace $ kubectl --kubeconfig boots apply -f aws/out/machine-deployment.yaml
machinedeployment.cluster.k8s.io/ha0x02-machinedeployment created

13: the last one

guohao@buffer ~/workspace $ kubectl --kubeconfig kubeconfig get node --all-namespaces
NAME                                       STATUS     ROLES    AGE   VERSION
ip-10-0-0-223.us-east-2.compute.internal   NotReady   node     29s   v1.14.4
ip-10-0-0-242.us-east-2.compute.internal   Ready      master   20m   v1.14.4
ip-10-0-2-92.us-east-2.compute.internal    Ready      master   18m   v1.14.4
ip-10-0-3-142.us-east-2.compute.internal   Ready      master   17m   v1.14.4

Sn0rt on 8 Aug 2019

It is also possible to define multiple subnets when creating a Cluster with a "managed VPC" by setting Cluster.Spec.ProviderSpec.NetworkConfig.Subnets[*].AvailabilityZone for each Availability zone you wanted created in the managed VPC. When paired with setting the AvailabilityZone on the Machine, this allows you to pre-create Cluster and Machine resources without needing to pre-define the VPC or know the Subnets in advance before creating the Machines.

is it means the HA info is cluster level not control-panel machine level ?

Sn0rt on 8 Aug 2019

It is both, since the shared infrastructure is created at the "Cluster level" and each control plane Machine needs to specify which subnet/AZ it should belong in.

detiber on 8 Aug 2019

/assign

Sn0rt on 16 Aug 2019

Trying to close out v0.3.x as we are no longer maintaining that milestone. Going to clear the milestone so we can retriage at the next meeting (or async works too!).

/milestone clear

ncdc on 10 Oct 2019

/milestone v0.4.x
/priority important-soon

chuckha on 4 Nov 2019

This is something that was brought up in the meeting: https://blog.scottlowe.org/2019/09/05/highly-available-kubernetes-clusters-on-aws-with-cluster-api/

We could adopt some of this for the v0.4 milestone and rework them for the v0.5 milestone

chuckha on 4 Nov 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 2 Feb 2020

/remove-lifecycle stale

detiber on 3 Feb 2020

Bumping this to v0.5.x, since we are nearing the v0.5.0 release
/milestone v0.5.x

detiber on 3 Mar 2020

With v1alpha3 and the KubeadmControlPlane, it looks like the only way to accomplish this is to provide a networkSpec laying out the subnets and AZs, and then the KCP will "do the right thing" and spread control plane nodes across AZs. Should I write-up a document to that end?