Cluster-api-provider-azure: Secure AKS Baseline with CAPI/CAPZ

[eedorenko]

⚠️ Cluster API Azure maintainers can ask to turn an issue-proposal into a CAEP when necessary. This is to be expected for large changes that impact multiple components, breaking changes, or new large features.

Goals

1. Be able to provision a production ready Secure AKS baseline cluster with CAPI/CAPZ

User Story
In order to be in compliance with AKS Secure Baseline Architecture an AKS cluster should have the following properties and configurations:

|Feature|Installed by Flux|Possible with CAPI/CAPZ|
|-------|--------------|---------|
|Different Subnets (system nodes, user nodes, ingress subnet)| | No. Not possible to specify different subnets for different agent pools|
|Azure CNI plugun| | Yes |
|AAD Pod Managed Identity| Yes| Yes|
|Azure RBAC for K8s Auth| | No. EnableRBAC and AadProfile profile properties are not set |
|Azure Key Vault CSI secret provider| Yes| Yes|
|Traefik + TLS| Yes | Yes|
|Azure private link to KV and ACR| | No. Not possible to configure a VNet in the yaml
|K8s network policy| Yes | Yes |
|Azure K8s policies| | Yes |
|ACR| | No. How do I define attachment to a private acr in the yaml? |
|Flux V2| | Yes
|Azure Monitor Prometheus Scraping| Yes| Yes |
|Kured| Yes| Yes |

That said, there are a few "NOs" in the "Possible with CAPI/CAPZ" column that currently prevent using CAPI/CAPZ for a Secure AKS cluster. None of these issues seem to be super complicated to implement, it just needs to be done.

[CecileRobertMichon]

/cc @nprokopic @alexeldeib

[nprokopic]

I could tackle this list :) but maybe refactoring from https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/757 should be done first?

[eedorenko]

FYI @craiglpeters @jackfrancis

[alexeldeib]

Thanks for making this list! Nicely highlights some of the areas for us to work on.

for private link, is all that’s needed a custom VNET? We do have that logic, the VNET is in the ManagedControlPlaneSpec: https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/7744b89fe0aae67ec3f282ffd957c1214dc33a52/exp/api/v1alpha4/azuremanagedcontrolplane_types.go#L42

I think we could potentially expand that to include subID and RG, to allow more flexibility with custom vnets in real deployments?

[alexeldeib]

private ACR would be a role binding to the kubelet identity used by the cluster. we're defaulting to System identities, but if we allow user-managed identities, we could "solve" this by asking users to provide a user-managed identity preconfigured with the desired RBAC to their ACR. There is no location to specify the ACR name AFAIK, kubelet's internal provider will figure out you are pulling an ACR image and try to auth with the credentials we supply.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[dtzar]

@alexeldeib are any of these gaps on the planned backlog?

[alexeldeib]

1418 and #1376 are the top issues for managed clusters and both in progress. I think we should resolve those before much more feature work. #1396 is also pretty straightforward and should be a reasonable follow up shortly after.

With those done, I'd want to flesh out the model we map to the AKS API, addressing some of the issues here. @LochanRn is already working towards AAD support (https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/1407), but needs #1418 and #1376 first.

Different subnets should be pretty straightforward. We also need to add user-assigned managed identities for Kubelet. I think then the whole table will be filled out?

I don't have an explicit timeline for all of that. But like I mentioned, #1418 and #1376 are in-progress en route to #1407 (thank you again @LochanRn!). If no one beats me to it, I'll probably pick up subnets and user-assigned managed identities.

/remove-lifecycle stale