Cluster-api-provider-azure: failed calling webhook "azuremanagedcontrolplane.kb.io"

[ams0]

/kind bug

What steps did you take and what happened:

First of all apologies for my complete lack of understanding of cluster-api-provider-azure, but I wanted to try the AKS provider like this:

#create Kind cluster
kind create cluster --name capi
clusterctl init --infrastructure azure

then patching the CAPI controller-managers (inspired by #787):

kubectl patch deployment  \
  capz-controller-manager \
  --namespace capz-system \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/args", "value": [
   "--metrics-addr=127.0.0.1:8080",
   "--enable-leader-election",
   "--feature-gates=MachinePool=true,AKS=true"
]}]'

kubectl patch deployment  \
  capi-controller-manager \
  --namespace capi-system \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/args", "value": [
   "--metrics-addr=127.0.0.1:8080",
   "--enable-leader-election",
   "--feature-gates=MachinePool=true,ClusterResourceSet=true"
]}]'

kubectl patch deployment  \
  capi-controller-manager \
  --namespace capi-webhook-system \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/args", "value": [
   "--metrics-addr=127.0.0.1:8080",
   "--webhook-port=9443",
   "--feature-gates=MachinePool=true,ClusterResourceSet=true"
]}]'

I'm not sure it's enough..because when I try to create a cluster:

$> clusterctl config cluster akscapi --flavor aks | kubectl apply -f -
cluster.cluster.x-k8s.io/akscapi unchanged
azuremanagedcluster.exp.infrastructure.cluster.x-k8s.io/akscapi unchanged
machinepool.exp.cluster.x-k8s.io/agentpool0 unchanged
azuremanagedmachinepool.exp.infrastructure.cluster.x-k8s.io/agentpool0 unchanged
machinepool.exp.cluster.x-k8s.io/agentpool1 unchanged
azuremanagedmachinepool.exp.infrastructure.cluster.x-k8s.io/agentpool1 unchanged
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "azuremanagedcontrolplane.kb.io": the server could not find the requested resource

I do see the azuremanagedcontrolplane.kb.io when I run

kubectl get mutatingwebhookconfiguration capz-mutating-webhook-configuration -o yaml | grep   azuremanagedcontrolplane.kb.io
        k:{"name":"azuremanagedcontrolplane.kb.io"}:
        k:{"name":"azuremanagedcontrolplane.kb.io"}:
  name: azuremanagedcontrolplane.kb.io

What am I missing?

What did you expect to happen:

A cluster created with AKS provider.

Anything else you would like to add:

$> kgpoall
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-6b6579d56d-sqzvx       2/2     Running   0          48m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-6d878bb599-r5zt9   2/2     Running   0          48m
capi-system                         capi-controller-manager-69f474dd87-hwpnw                         2/2     Running   0          30m
capi-webhook-system                 capi-controller-manager-6c9b6f5dc4-968m4                         2/2     Running   0          30m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-56f98bc7f9-dhfzl       2/2     Running   0          48m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-85bcfd7fcd-5pbm8   2/2     Running   0          48m
capi-webhook-system                 capz-controller-manager-7b7f49c946-qf6tk                         2/2     Running   0          48m
capz-system                         capz-controller-manager-55ffb5766-k8tdc                          2/2     Running   0          42m
cert-manager                        cert-manager-cainjector-fc6c787db-ncf7w                          1/1     Running   0          48m
cert-manager                        cert-manager-d994d94d7-hjkkt                                     1/1     Running   0          48m
cert-manager                        cert-manager-webhook-845d9df8bf-8rcfx                            1/1     Running   0          48m
kube-system                         coredns-f9fd979d6-5v9j6                                          1/1     Running   0          60m
kube-system                         coredns-f9fd979d6-p8djp                                          1/1     Running   0          60m
kube-system                         etcd-capi-control-plane                                          1/1     Running   0          60m
kube-system                         kindnet-bplkb                                                    1/1     Running   0          60m
kube-system                         kube-apiserver-capi-control-plane                                1/1     Running   0          60m
kube-system                         kube-controller-manager-capi-control-plane                       1/1     Running   0          60m
kube-system                         kube-proxy-6ddfb                                                 1/1     Running   0          60m
kube-system                         kube-scheduler-capi-control-plane                                1/1     Running   0          60m
local-path-storage                  local-path-provisioner-78776bfc44-gqtbg                          1/1     Running   0          60m

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version): 1.19.1
• OS (e.g. from /etc/os-release):

[alexeldeib]

I believe you need to set env vars EXP_AKS=true and EXP_MACHINE_POOL=true before running clusterctl init. We probably should do a doc update for this in [0]. It's briefly mentioned that those are the required feature gates at the top, but you'd need to put it together with the knowledge from [1].

[alexeldeib]

fyi, it gets templated into the manager + webhook deployment: https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/master/config/manager/manager.yaml#L21

You're hitting that issue because the CAPZ webhook pod doesn't have the feature gate enabled so even with the webhook defined, calls to the pod will fail.

Your patches almost got there, but missed the capz webhook I think.

[alexeldeib]

/reopen

sorry, didn't mean to close, just mention

[k8s-ci-robot]

@alexeldeib: Reopened this issue.

In response to this:

/reopen

sorry, didn't mean to close, just mention

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ams0]

Thanks @alexeldeib ! That unblocked me considerably.

So what I am doing now:

export EXP_AKS=true
export EXP_MACHINE_POOL=true
export EXP_CLUSTER_RESOURCE_SET=false #not sure about this, but it works

kind create cluster --name capi
clusterctl init --infrastructure azure

Then create the clsuter

az group create -n capz-clusters
clusterctl config cluster akscapi --flavor aks  -i azure:v0.4.9 | kubectl apply -f -

A few more questions:

• Right after creation, the only cluster nodepool undergoes an upgrade operation:

az aks nodepool list -g capz-clusters --cluster-name akscapi-control-plane
Name        OsType    VmSize           Count    MaxPods    ProvisioningState    Mode
----------  --------  ---------------  -------  ---------  -------------------  ------
agentpool0  Linux     Standard_DS4_v2  2        30         Updating             User

Why is that?

• The cluster is working fine:

kubectl get secret akscapi-kubeconfig -o yaml -o jsonpath={.data.value} | base64 --decode > akscapi.kubeconfig
kubectl get node --kubeconfig=akscapi.kubeconfig
NAME                                 STATUS   ROLES   AGE     VERSION
aks-agentpool0-38263547-vmss000000   Ready    agent   9m32s   v1.19.3
aks-agentpool0-38263547-vmss000001   Ready    agent   9m17s   v1.19.3

but the cluster object stays in provisioning state:

kubectl get clusters
NAME      PHASE
akscapi   Provisioning

Is it because of this error in the logs of capz-controller?

I1104 09:22:07.070326       1 helpers.go:215] controllers/AzureMachinePool "msg"="gk does not match" "controller"="AzureMachinePool" "gk"={"Group":"exp.infrastructure.cluster.x-k8s.io","Kind":"AzureMachinePool"} "infraGK"={"Group":"exp.infrastructure.cluster.x-k8s.io","Kind":"AzureManagedMachinePool"}

• A second agentpool is created by clusterctl config

 kubectl get  AzureManagedMachinePool
NAME         AGE
agentpool0   23m
agentpool1   23m

kubectl get  MachinePool
NAME         REPLICAS   PHASE          VERSION
agentpool0   2          Running        v1.19.3
agentpool1              Provisioning   v1.19.3

but I don't understand why (perhaps that's why the cluster is in provisioning state?). In general how can I influence the behavior of clusterctl config cluster --flavor aks? Is there a list of options (like an Helm chart Readme?)

Thanks and sorry for unloading all the questions here, happy to open separate issues or continue on Slack.

[alexeldeib]

So a couple things

az group create -n capz-clusters

btw, no longer necessary (used to be). we'll create the group if it doesn't exist. I'll check the docs and update if that doesn't match

In general how can I influence the behavior of clusterctl config cluster --flavor aks

clusterctl is a fancy templating tool. The names of flavors are derived from the names of yaml files here: https://github.com/kubernetes-sigs/cluster-api-provider-azure/tree/master/templates (e.g. flavor=aks -> cluster-template-aks.yaml). You can take the raw yaml for any of these and kubectl apply them like any Kubernetes CRD. Clusterctl envsubsts a bunch of values and uses some defaults in the yaml you can see.

Is there a list of options (like an Helm chart Readme?)

Probably the best thing is the godoc for the api types right now: https://godoc.org/sigs.k8s.io/cluster-api-provider-azure/exp/api/v1alpha3#AzureManagedControlPlaneSpec. But really it's whatever bash-style env vars are in the yaml you pass it. It is just an envsubst with https://github.com/drone/envsubst I believe.

A second agentpool is created by clusterctl config

cluster-template-aks.yaml, as deployed by flavor=aks, contains both a required default nodepool and an optional second nodepool. I created the example like this because AKS requires at least one system nodepool which is not a CAPI requirement. However to use AKS as a control plane provider in CAPI, we need to know how the user wants to provision that first system node pool. I solved this by adding a reference to a CAPI machine pool on the AzureManagedControlPlane object and requiring it to be present to build the system node pool spec for AKS.

Since this first pool is a little bit special, I added a second machine pool to the example using the "normal" approach.

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/2dfc5ea4acb9bc037349516d6a75a262606963f1/templates/cluster-template-aks.yaml#L70-L98

but the cluster object stays in provisioning state:

tbh this could be a bug, the managed control plane functionality is admittedly still a bit rough. I'd like to work on https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/611 to get us some more confidence there. It's mostly manual sanity testing right now + unit tests where applicable. LMK if it never settles and I can take a closer look.

Right after creation, the only cluster nodepool undergoes an upgrade operation

The cluster reconciler and the agentpool reconcilers are separate at the moment. What you're seeing is we reconcile the cluster (including the default node pool), but then also need to reconcile each node pool individually. So after cluster creation we immediately trigger a no-op PUT on the first agent pool. We should be checking desired vs. current state and avoid sending no-op PUTs to AKS. We also should check MC provisioningState is terminal before initiating agent pool operations. See https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/847

Is it because of this error in the logs of capz-controller?

no, that's a valid and expected error logged for debug/info iirc (maybe we can drop the log level). AzureMachinePool and AzureManagedMachinePool both trigger reconciles from watching MachinePool, but only when the associated infrastructure resource matches for that type. So AzureMachinePool should ignore requests to reconcile MachinePools which are associated with AzureManagedMachinePool and vice versa.

Thanks and sorry for unloading all the questions here, happy to open separate issues or continue on Slack.

no worries, feel free to keep them coming 😃 happy to answer in slack as well.

[ams0]

Thanks so much, I kept playing with it today and I was just too impatient - eventually the cluster reconciles and both nodepools come up fine like the cluster. This is great stuff! Now working on putting all the templates into a handy Helm chart to deploy clusters programmatically (with Flux).

I'd love to contribute to the aks provider, adding flags like private-cluster, custom-vnet and so on. I'm closing this and thanks again for the support.

[alexeldeib]

Awesome! Glad I could help. Feel free to ping me if you have any questions/comments 🙂