Cluster-api-provider-azure: HA Clusters in Non-AZ enabled Regions

[jseely]

⚠️ Cluster API Azure maintainers can ask to turn an issue-proposal into a CAEP when necessary. This is to be expected for large changes that impact multiple components, breaking changes, or new large features.

Goals

1. Provide a mechanism to create HA clusters in regions that do not support AZs
2. Investigate VMSS Orchestration Mode (Preview Feature) and Availability Sets
3. Investigate interoperability between Availability Zones and (VMSS Orchestration Mode or Availability Sets)

Non-Goals/Future Work

1. Core API CRD support for VMSS Orchestration Mode

User Story

As a user I would like to highly available clusters in all regions including regions without AZs so my clusters can serve workloads consistently.

Detailed Description

We need to first investigate the edge cases around HA clusters backed by VMSS Orchestration Mode and Availability Sets to understand trade-offs in either implementation. Notably, Availability Sets pin resources to 1 of 3 fault domains which are selected on the first VM being assigned to the availability set. This means that all VMs assigned to that Availability Set must reside within the same set of hardware. For instance a single AvailSet can not support a Standard_DS_v2 VM at the same time as a Standard_DS_v3 VM as they reside on independent hardware. Also Availability Sets have scaling constraints due to max # of vms per AvailSet as well as capacity constraints related to the capacity of the three Fault Domains selected and not the full capacity of that region.

VMSS Orchestration Mode - Docs
Availability Sets - Docs

Contract changes [optional]

There may be some changes required in Cluster API to support Failure Groups that are managed by the provider. While AZs can be manually managed by the User Fault Domains aren't exposed and are internally managed by Availability Sets.

/kind proposal

[jseely]

^ @CecileRobertMichon @devigned @justaugustus @nader-ziada @rbitia from today's office hours

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[voor]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[voor]

/remove-lifecycle stale

[CecileRobertMichon]

Looked into this a bit.

1. Provide a mechanism to create HA clusters in regions that do not support AZs

we'll need to find a mechanism to only add new VMs to availability sets (when zones are not supported) as availability sets do not support adding existing VMs. For now, we should focus on control plane VMs (ref https://github.com/kubernetes-sigs/cluster-api/issues/3358 for worker nodes).

1. Investigate VMSS Orchestration Mode (Preview Feature) and Availability Sets

VM orchestration modes are not available yet (preview is now closed) so we can revisit if/when they become available. I'd expect the process to be fairly similar anyway.

1. Investigate interoperability between Availability Zones and (VMSS Orchestration Mode or Availability Sets)

Zones and availability sets are mutually exclusive. We should pick zones when they are available and fallback to availability sets when they aren't.

For instance a single AvailSet can not support a Standard_DS_v2 VM at the same time as a Standard_DS_v3 VM as they reside on independent hardware.

I don't expect this to be an issue as we shouldn't put VMs from different "pools" in the same AS. All KCP control plane VMs have the same VM size and VM definition. If we extend to worker VMs, this should work per MachineDeployment / MachinePool so all VMs in an AS will have the same hardware characteristics.

Also Availability Sets have scaling constraints due to max # of vms per AvailSet as well as capacity constraints related to the capacity of the three Fault Domains selected and not the full capacity of that region.

For control planes this won't be a problem. For worker nodes it will be a constraint but I would just guide users towards using scale sets for large pool needs. For the record, current limit is 200 VMs per availability set. Compare to 600 per scale set with a custom image (1000 without custom image but I'd expect most users to rely on custom images).

[CecileRobertMichon]

/help

This issue is ready to be picked up if someone wants to give it a go. Ask is to implement availability sets for control plane virtual machines in regions that do not support zones. This should _only_ be applied to new VMs created, not existing VMs (should be okay since we don't update VMs after they are created).

[k8s-ci-robot]

@CecileRobertMichon:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

This issue is ready to be picked up if someone wants to give it a go. Ask is to implement availability sets for control plane virtual machines in regions that do not support zones. This should _only_ be applied to new VMs created, not existing VMs (should be okay since we don't update VMs after they are created).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[shysank]

/assign

[shysank]

@CecileRobertMichon A few clarifications:

1. Should we manage the lifecycle of availability sets or is it expected to be created beforehand?
2. If both failure domain and availability sets are present (region supports azs), failure domain takes precedence over availability sets?
3. I'm assuming this would be a change in AzureCluster spec. If we don't have to manage the availability set, it could also be set in the machine template spec, and potentially have more flexibility in what set of machines we want to make ha?
4. A followup on the above point, I don't fully understand why we have to restrict it to control plane nodes. I understand the limitation of failure domains in md, but in this case, can we not just assign all the nodes to the availability set, and let it take care of the distribution?

[CecileRobertMichon]

@shysank

1. Yes, CAPZ should manage lifecycle of availability sets. It should be a new resource that is created and deleted by AzureCluster. (edit: that's if we do control planes only, see point 4) below. It might make sense to reconcile as part of AzureMachine for control planes too actually).
2. Correct. You can inspect AzureCluster.Status.FailureDomains. If len == 0, then the location does not have AZs and we should create an availability set. If the region has AZs, no availability set should be created.
3. Not necessarily. I don't think this needs to be configurable for now and if we're not supporting BYO it doesn't need to be exposed in the API. This should be mostly hidden from the users, IMO. But happy to discuss if you think otherwise.
4. This was to try and simplify the issue as first step. Eventually we'd want all machines to be HA (whether that's with zones or scale sets, or availability sets). The main difficulty with worker machines is that you need one availability set per "pool" (machine deployment or machine set). It needs to be created before the first machine is created and deleted after the last one is deleted (totally solvable though, for create we could do a createOrUpdate before each machine create so it just creates it if it doesn't exist. For delete we could check if it's still in use and only delete it if not in use by other VMs when we do a VM delete). With control planes it's easier because the lifecycle of the availability set can be tied to the lifecycle of the cluster. If you want to take on both go for it, was mostly trying to break up the issue into smaller chunks to help newer contributors.

[shysank]

Thanks @CecileRobertMichon for the responses. For (4). I actually thought it was going to be a single availability set for the entire cluster. I think you're right, it's better to start with the control plane nodes, and then iterate from there.

[shysank]

Not necessarily. I don't think this needs to be configurable for now and if we're not supporting BYO it doesn't need to be exposed in the API. This should be mostly hidden from the users, IMO. But happy to discuss if you think otherwise.

On second thought, we do a similar thing for resource groups, where we set the name to cluster name in defaults if not present, and create a group with that name. While deleting, we ignore the error if it's not owned by the cluster. Maybe we could do something similar for available sets? This will also be useful to get the name when deleting, otherwise we might need to infer using some kind of convention to determine the sets name. And it'll provide information about the sets in kubectl describe, in case, for debugging. wdyt?

[CecileRobertMichon]

For resource groups, it's important to allow users to bring their own resource groups and customize the name of the resource group since that needs to be unique in the subscription. I'm not sure there is a similar user need for availability sets. As a user, I'm not sure why you would want to bring your own availability set. You shouldn't share availability sets across clusters or with other non-CAPZ VMs. An availability set should be dedicated to a set of VMs, typically an "agent pool" or a "control plane pool".

I do agree about not making assumptions about the name when deleting. In that case, the availability set would need to be part of AzureMachineTemplate. But that also complicates things because we'd have to enforce validation to make sure the user doesn't specify _both_ a failure domain and an availability set name. Also, in the webhooks we can't make API calls so we won't know if failure domains are supported in that region (that's why we set failure domains in the controller), so it will be hard to set defaults. I'm not sure it's worth the extra effort. We have other resources (like public IPs, OS disks, load balancers, etc.) that don't have configurable names already. I would be okay with naming the availability sets as-<kcp-name> or as-<machine-deployment-name>, and not making them configurable, especially since AS naming scope is the resource group and I don't expect us relying on AS long term (https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/999 should be the long term solution).

[shysank]

/reopen

[k8s-ci-robot]

@shysank: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[shysank]

@CecileRobertMichon Should we reopen this issue for worker nodes? or create a new one?

[CecileRobertMichon]

/reopen

we can keep this one open since it has the discussion

[k8s-ci-robot]

@CecileRobertMichon: Reopened this issue.

In response to this:

/reopen

we can keep this one open since it has the discussion

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[shysank]

I was planning to implement this. Extending on 1099, the plan is to have another availability set for worker nodes, and assign all vms that are not control plane nodes to this new availability set. wdyt?

[CecileRobertMichon]

I think it would make more sense to have one availability set per MachineDeployment. Because 1) you will run into max number of VMs per AS (200) quickly with large clusters, 2) it makes more sense to spread similar machines across fault domains. For example, if you have one MD with 10 GPU nodes and one MD with 10 CPU nodes in a region with 2 fault domains, you'd want 5 GPU nodes in each fault domain. If there's only 1 availability set in the whole cluster, you could end up with all 10 GPU nodes in the same fault domain.