Cluster-api: kcp keeps trying init cluster creating 20+ controlplane nodes

Created on 4 May 2020 · 14Comments · Source: kubernetes-sigs/cluster-api

What steps did you take and what happened:

See this with capz, kcp try to init cluster, creating machines, but somehow in later reconcile does not see the created machine "Desired"=1 "Existing"=0, and then try to init again. This keep going till kcp pod somehow lose leader election and got restarted (after around 20 minutes), then it can see all the machines (24 replicas) and start to scale down.

This happens when 3 cluster are launched around the same time, in different namespaces with controllers running independently in each namespaces. All 3 cluster see similar issue. Not able to reproduce after this run, but still curious to know what could be the possible root cause.

The fact that after kcp restart it can see all the replicas makes me wonder if possible because kcp somehow can not read the data from the cache client?

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

logs in gist: https://gist.github.com/jzhoucliqr/5a5989421b9ab92f6fa1f1fea1805457

kcp log before restart:

I0503 18:11:07.055730       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I0503 18:11:07.174735       1 main.go:143] setup "msg"="starting manager"  "version"=""
I0503 18:11:07.283549       1 leaderelection.go:242] attempting to acquire leader lease  cluster-5eaf02d88525ec2c177c8194/kubeadm-control-plane-manager-leader-election-capi...
I0503 18:11:07.318858       1 internal.go:356] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0503 18:11:07.403446       1 leaderelection.go:252] successfully acquired lease cluster-5eaf02d88525ec2c177c8194/kubeadm-control-plane-manager-leader-election-capi
I0503 18:11:07.411352       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"version":"","infrastructureTemplate":{},"kubeadmConfigSpec":{}},"status":{"initialized":false,"ready":false}}}
I0503 18:11:07.521060       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:11:07.727868       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:11:07.869003       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="kubeadmcontrolplane"
I0503 18:11:07.869863       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="kubeadmcontrolplane" "worker count"=10
W0503 18:11:10.524651       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189143274 (189143635)
W0503 18:11:13.315985       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189143260 (189143783)
W0503 18:11:13.569922       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189142558 (189143912)
W0503 18:11:27.431298       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189143783 (189144879)
W0503 18:11:29.080846       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189143912 (189144983)
W0503 18:11:30.833610       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189144879 (189145123)
W0503 18:11:32.200927       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189144983 (189145239)
W0503 18:11:43.326595       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189143787 (189146164)
I0503 18:12:00.101455       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
W0503 18:12:02.606307       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189145123 (189147334)
I0503 18:12:06.432680       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
W0503 18:12:27.373628       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AzureMachineTemplate ended with: too old resource version: 189147288 (189148818)
E0503 18:12:40.066053       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:12:40.637723       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:12:40.638257       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:40.638712       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:13:13.172003       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:13:13.373848       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:13:13.374333       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:13.374747       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:13:44.176878       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:13:44.177436       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:13:44.177805       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:44.178243       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:14:16.525902       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:14:16.526436       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:14:16.526799       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:14:16.527256       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:14:48.125634       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:14:48.126167       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:14:48.126575       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:14:48.127097       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:15:19.235634       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:15:19.236164       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:15:19.236525       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:15:19.236932       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:15:49.328025       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:15:49.328485       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:15:49.329095       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:15:49.329567       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:16:19.391199       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:16:19.391684       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:16:19.392064       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:16:19.392526       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:16:49.473964       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:16:49.474851       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:16:49.484484       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:16:49.484946       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:17:19.532691       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:17:19.533241       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:17:19.533626       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:17:19.533964       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:17:51.523432       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:17:51.524891       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:17:51.525275       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:17:51.525730       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:18:23.052235       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:18:23.052571       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:18:23.052850       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:18:23.053281       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:18:56.833777       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:18:56.834341       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:18:56.834772       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:18:56.835224       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:19:27.447739       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:19:27.448200       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:19:27.448611       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:19:27.449014       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:19:58.404637       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:19:58.405538       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:20:08.409213       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:20:08.410330       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:20:38.464228       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:20:38.464577       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:20:38.464843       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:20:38.465174       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
W0503 18:21:08.156990       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189149596 (189183041)
E0503 18:21:08.523908       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:21:08.524458       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:21:08.524874       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:21:08.525326       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
W0503 18:21:21.278714       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189153318 (189183969)
W0503 18:21:28.650735       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AzureMachineTemplate ended with: too old resource version: 189148818 (189184648)
E0503 18:21:38.692514       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:21:38.692912       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:21:38.693215       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:21:38.693532       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:22:08.992452       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:22:08.992977       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:23:19.122553       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:23:19.123067       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:23:52.652996       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:23:52.653555       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:23:52.653952       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:23:52.654393       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:24:24.138932       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:24:24.139642       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:25:28.801348       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:25:28.801828       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:25:58.885516       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:25:58.886011       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:25:58.886326       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:25:58.886732       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
E0503 18:26:28.948185       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:26:28.949083       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
W0503 18:27:10.487397       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189183041 (189208356)
I0503 18:27:11.517105       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:27:11.517866       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
W0503 18:27:25.334120       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.KubeadmControlPlane ended with: too old resource version: 189183969 (189209362)
I0503 18:27:39.296481       1 trace.go:116] Trace[772950856]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105 (started: 2020-05-03 18:27:26.334435947 +0000 UTC m=+983.101753221) (total time: 12.962012296s):
Trace[772950856]: [12.961989176s] [12.961989176s] Objects listed
E0503 18:27:41.654161       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:27:41.654628       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: dial tcp 52.186.44.52:6443: i/o timeout"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:27:41.654955       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:27:41.655367       1 controller.go:237] controllers/KubeadmControlPlane "msg"="Initializing control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=0
I0503 18:27:44.082259       1 leaderelection.go:288] failed to renew lease cluster-5eaf02d88525ec2c177c8194/kubeadm-control-plane-manager-leader-election-capi: failed to tryAcquireOrRenew context deadline exceeded
E0503 18:27:44.082333       1 main.go:145] setup "msg"="problem running manager" "error"="leader election lost"

kcp log after restart:

I0503 18:27:46.795790       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I0503 18:27:46.796180       1 main.go:143] setup "msg"="starting manager"  "version"=""
I0503 18:27:46.796299       1 leaderelection.go:242] attempting to acquire leader lease  cluster-5eaf02d88525ec2c177c8194/kubeadm-control-plane-manager-leader-election-capi...
I0503 18:27:46.796625       1 internal.go:356] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0503 18:28:05.964877       1 leaderelection.go:252] successfully acquired lease cluster-5eaf02d88525ec2c177c8194/kubeadm-control-plane-manager-leader-election-capi
I0503 18:28:05.965826       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"version":"","infrastructureTemplate":{},"kubeadmConfigSpec":{}},"status":{"initialized":false,"ready":false}}}
I0503 18:28:06.066624       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:28:06.167279       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="kubeadmcontrolplane" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:28:06.267676       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="kubeadmcontrolplane"
I0503 18:28:06.267712       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="kubeadmcontrolplane" "worker count"=10
I0503 18:28:06.280263       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:28:06.483142       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
W0503 18:28:10.787377       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AzureMachineTemplate ended with: too old resource version: 189209879 (189212290)
E0503 18:29:06.486307       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:29:06.514411       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: context deadline exceeded"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:29:06.514870       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:29:06.515780       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
E0503 18:30:06.518128       1 controller.go:147] controllers/KubeadmControlPlane "msg"="Failed to update KubeadmControlPlane Status" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:30:06.531910       1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to create remote cluster client: failed to create client for workload cluster cluster-5eaf02d88525ec2c177c8194/azyre-cos-0503-3: Get https://azyre-cos-0503-3-16a53890.eastus.cloudapp.azure.com:6443/api?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"  "controller"="kubeadmcontrolplane" "request"={"Namespace":"cluster-5eaf02d88525ec2c177c8194","Name":"azyre-cos-0503-3-cp"}
I0503 18:30:06.532362       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:30:06.532944       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
I0503 18:30:14.378468       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:30:14.379407       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
I0503 18:30:15.431924       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:30:15.432600       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
I0503 18:30:16.367831       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:30:16.373071       1 controller.go:246] controllers/KubeadmControlPlane "msg"="Scaling down control plane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" "Desired"=1 "Existing"=24
I0503 18:30:34.379277       1 controller.go:179] controllers/KubeadmControlPlane "msg"="Reconcile KubeadmControlPlane" "cluster"="azyre-cos-0503-3" "kubeadmControlPlane"="azyre-cos-0503-3-cp" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I

capz log:

I0503 18:11:07.254735       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I0503 18:11:07.255119       1 main.go:222] setup "msg"="starting manager"  
I0503 18:11:07.255276       1 leaderelection.go:242] attempting to acquire leader lease  cluster-5eaf02d88525ec2c177c8194/controller-leader-election-capz...
I0503 18:11:07.255613       1 internal.go:356] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0503 18:11:07.310453       1 leaderelection.go:252] successfully acquired lease cluster-5eaf02d88525ec2c177c8194/controller-leader-election-capz
I0503 18:11:07.311199       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="azurecluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"networkSpec":{"vnet":{"name":""}},"resourceGroup":"","location":"","controlPlaneEndpoint":{"host":"","port":0}},"status":{"network":{"apiServerLb":{"frontendIpConfig":{},"backendPool":{}},"apiServerIp":{}},"bastion":{"image":{},"osDisk":{"osType":"","diskSizeGB":0,"managedDisk":{"storageAccountType":""}}},"ready":false}}}
I0503 18:11:07.311515       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="azuremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"vmSize":"","availabilityZone":{},"osDisk":{"osType":"","diskSizeGB":0,"managedDisk":{"storageAccountType":""}},"location":"","sshPublicKey":""},"status":{"ready":false}}}
I0503 18:11:07.411542       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="azurecluster"
I0503 18:11:07.411603       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="azurecluster" "worker count"=10
I0503 18:11:07.412196       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="azuremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:11:07.513768       1 azurecluster_controller.go:110] controllers/AzureCluster "msg"="Reconciling AzureCluster" "azureCluster"="azyre-cos-0503-3" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:11:07.514470       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="azuremachine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"networkSpec":{"vnet":{"name":""}},"resourceGroup":"","location":"","controlPlaneEndpoint":{"host":"","port":0}},"status":{"network":{"apiServerLb":{"frontendIpConfig":{},"backendPool":{}},"apiServerIp":{}},"bastion":{"image":{},"osDisk":{"osType":"","diskSizeGB":0,"managedDisk":{"storageAccountType":""}}},"ready":false}}}
I0503 18:11:07.514514       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="azuremachine"
I0503 18:11:07.514537       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="azuremachine" "worker count"=10
W0503 18:11:09.469011       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189143274 (189143635)
W0503 18:11:11.889187       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189143635 (189143785)
W0503 18:11:13.319374       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189142559 (189143904)
W0503 18:11:19.984494       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureCluster ended with: too old resource version: 189143497 (189144354)
W0503 18:11:20.158485       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureMachine ended with: too old resource version: 189143112 (189144356)
W0503 18:11:28.690735       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189143904 (189144956)
W0503 18:11:32.104077       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189144956 (189145236)
W0503 18:11:34.370715       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureCluster ended with: too old resource version: 189144354 (189145423)
W0503 18:11:34.427233       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureMachine ended with: too old resource version: 189144356 (189145430)
W0503 18:11:38.151291       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureCluster ended with: too old resource version: 189145423 (189145777)
W0503 18:11:38.251732       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureMachine ended with: too old resource version: 189145430 (189145781)
W0503 18:11:43.406341       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Cluster ended with: too old resource version: 189143785 (189146211)
I0503 18:11:58.479664       1 azurecluster_controller.go:110] controllers/AzureCluster "msg"="Reconciling AzureCluster" "azureCluster"="azyre-cos-0503-3" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:01.687268       1 azurecluster_controller.go:110] controllers/AzureCluster "msg"="Reconciling AzureCluster" "azureCluster"="azyre-cos-0503-3" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
W0503 18:12:06.900413       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.Machine ended with: too old resource version: 189145236 (189147650)
I0503 18:12:08.092836       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-g64lr" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:09.512033       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-g64lr" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
W0503 18:12:20.853541       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureCluster ended with: too old resource version: 189147331 (189148486)
W0503 18:12:20.903012       1 reflector.go:326] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: watch of *v1alpha3.AzureMachine ended with: too old resource version: 189147734 (189148491)
I0503 18:12:21.858761       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-g64lr" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:21.858986       1 azurecluster_controller.go:110] controllers/AzureCluster "msg"="Reconciling AzureCluster" "azureCluster"="azyre-cos-0503-3" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:21.905748       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-g64lr" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:41.600876       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-tcwz8" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:12:42.431844       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-tcwz8" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:13.724652       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-2gtnd" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:14.166602       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-2gtnd" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:45.783152       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-6q7j5" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:46.422320       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-6q7j5" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:13:57.482167       1 azuremachine_controller.go:94] controllers/AzureMachine "msg"="Machine Controller has not yet set OwnerRef" "azureMachine"="azyre-cos-0503-3-cp-59566-tcwz8" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I....

capi log. This log timestamp seems start from when kcp restarts. The older logs are lost.

I0503 18:27:41.370226       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I0503 18:27:41.371199       1 main.go:175] setup "msg"="starting manager"  "version"=""
I0503 18:27:41.372264       1 leaderelection.go:242] attempting to acquire leader lease  cluster-5eaf02d88525ec2c177c8194/controller-leader-election-capi...
I0503 18:27:41.372534       1 internal.go:356] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0503 18:27:58.019593       1 leaderelection.go:252] successfully acquired lease cluster-5eaf02d88525ec2c177c8194/controller-leader-election-capi
I0503 18:27:58.020281       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"unhealthyConditions":null},"status":{}}}
I0503 18:27:58.020603       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:27:58.020960       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:27:58.021058       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0503 18:27:58.021150       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machine" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:27:58.021519       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:27:58.021840       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0503 18:27:58.121456       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinehealthcheck" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:27:58.121645       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="machinehealthcheck"
I0503 18:27:58.121769       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="machinehealthcheck" "worker count"=10
I0503 18:27:58.123935       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:27:58.124093       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="cluster"
I0503 18:27:58.124211       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="cluster" "worker count"=10
I0503 18:27:58.125211       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:27:58.125394       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}},"status":{"bootstrapReady":false,"infrastructureReady":false}}}
I0503 18:27:58.125584       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machineset" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:27:58.125733       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="machineset"
I0503 18:27:58.126602       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="machine"
I0503 18:27:58.126664       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0503 18:27:58.126856       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0503 18:27:58.126987       1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0503 18:27:58.127128       1 controller.go:171] controller-runtime/controller "msg"="Starting Controller"  "controller"="machinedeployment"
I0503 18:27:58.326100       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="machineset" "worker count"=10
I0503 18:27:58.329123       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="machine" "worker count"=10
I0503 18:27:58.330433       1 controller.go:190] controller-runtime/controller "msg"="Starting workers"  "controller"="machinedeployment" "worker count"=10
I0503 18:27:58.429149       1 tracker.go:54] controllers/Cluster "msg"="Adding watcher on external object" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AzureCluster"
I0503 18:27:58.429220       1 controller.go:138] controller-runtime/controller "msg"="Starting EventSource"  "controller"="cluster" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"AzureCluster"}}
I0503 18:27:58.531404       1 tracker.go:54] controllers/Cluster "msg"="Adding watcher on external object" "cluster"="azyre-cos-0503-3" "namespace"="cluster-5eaf02d88525ec2c177c8194" "GroupVersionKind"="controlplane.cluster.x-k8s.io/v1alpha3, Kind=KubeadmControlPlane"
I0503 18:27:58.531462       1 controller.go:138] controller-runtime/controller "msg"="Starting EventSource"  "controller"="cluster" "source"={"Type":{"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha3","kind":"KubeadmControlPlane"}}
I0503 18:27:58.733413       1 tracker.go:54] controllers/Machine "msg"="Adding watcher on external object" "machine"="azyre-cos-0503-3-cp-trbwv" "namespace"="cluster-5eaf02d88525ec2c177c8194" "GroupVersionKind"="bootstrap.cluster.x-k8s.io/v1alpha3, Kind=KubeadmConfig"
I0503 18:27:58.733476       1 controller.go:138] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machine" "source"={"Type":{"apiVersion":"bootstrap.cluster.x-k8s.io/v1alpha3","kind":"KubeadmConfig"}}
I0503 18:27:58.821473       1 tracker.go:54] controllers/Machine "msg"="Adding watcher on external object" "machine"="azyre-cos-0503-3-cp-q5gpz" "namespace"="cluster-5eaf02d88525ec2c177c8194" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AzureMachine"
I0503 18:27:58.821530       1 controller.go:138] controller-runtime/controller "msg"="Starting EventSource"  "controller"="machine" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"AzureMachine"}}
I0503 18:27:58.821832       1 machine_controller_noderef.go:53] controllers/Machine "msg"="Machine doesn't have a valid ProviderID yet" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-q5gpz" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:27:58.821991       1 machine_controller.go:215] controllers/Machine "msg"="Reconciliation for Machine asked to requeue" "error"="Bootstrap provider for Machine \"azyre-cos-0503-3-cp-q5gpz\" in namespace \"cluster-5eaf02d88525ec2c177c8194\" is not ready, requeuing: requeue in 30s" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-q5gpz" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:27:58.822887       1 machine_controller_noderef.go:53] controllers/Machine "msg"="Machine doesn't have a valid ProviderID yet" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-cjd5b" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:27:58.822924       1 machine_controller.go:215] controllers/Machine "msg"="Reconciliation for Machine asked to requeue" "error"="Bootstrap provider for Machine \"azyre-cos-0503-3-cp-cjd5b\" in namespace \"cluster-5eaf02d88525ec2c177c8194\" is not ready, requeuing: requeue in 30s" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-cjd5b" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:27:58.825672       1 machine_controller_noderef.go:53] controllers/Machine "msg"="Machine doesn't have a valid ProviderID yet" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-bhrn4" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:27:58.825784       1 machine_controller.go:215] controllers/Machine "msg"="Reconciliation for Machine asked to requeue" "error"="Bootstrap provider for Machine \"azyre-cos-0503-3-cp-bhrn4\" in namespace \"cluster-5eaf02d88525ec2c177c8194\" is not ready, requeuing: requeue in 30s" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-bhrn4" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
I0503 18:27:58.827291       1 machine_controller_noderef.go:53] controllers/Machine "msg"="Machine doesn't have a valid ProviderID yet" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-z9j6b" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
E0503 18:27:58.828137       1 machine_controller.go:215] controllers/Machine "msg"="Reconciliation for Machine asked to requeue" "error"="Bootstrap provider for Machine \"azyre-cos-0503-3-cp-z9j6b\" in namespace \"cluster-5eaf02d88525ec2c177c8194\" is not ready, requeuing: requeue in 30s" "cluster"="azyre-cos-0503-3" "machine"="azyre-cos-0503-3-cp-z9j6b" "namespace"="cluster-5eaf02d88525ec2c177c8194" 
....

Environment:

Cluster-api version: capi v0.3.4, capz v0.4.2
Minikube/KIND version:
Kubernetes version: (use kubectl version): v1.17.3
OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

kinbug lifecyclactive

Source

jzhoucliqr

All 14 comments

/assign @benmoss

vincepri on 4 May 2020

@jzhoucliqr it's a little hard to read the logs, would you be able to paste them in a github gist instead?

vincepri on 4 May 2020

👍1

Looking through the logs a little more, it seems that indeed the control plane is being initialized multiple times, which comes from here:

https://github.com/kubernetes-sigs/cluster-api/blob/7cda00b20b09cdccf2ce4e5e952ecfdc0f87ccd5/controlplane/kubeadm/controllers/controller.go#L235-L238

which then calls this function: https://github.com/kubernetes-sigs/cluster-api/blob/7cda00b20b09cdccf2ce4e5e952ecfdc0f87ccd5/controlplane/kubeadm/controllers/scale.go#L33-L46 that actually clones and creates the first control plane machine.

The numMachine variables that we see above, comes from the number of owned machines returned from https://github.com/kubernetes-sigs/cluster-api/blob/7cda00b20b09cdccf2ce4e5e952ecfdc0f87ccd5/controlplane/kubeadm/controllers/controller.go#L215. I'm not sure why it would return an empty result in this case, but I've also noticed that the leader election was lost at the end, which could suggest some issues with the management cluster.

vincepri on 4 May 2020

I can't imagine how this is possible. It was doing a client.List on machines owned by this cluster and this control plane and apparently coming up with 0. I wonder what the state of etcd was, if there were machines in it and they were being filtered out, or whether like you suggest it was some kind of client caching issue. I know literally nothing about what a caching issue could be caused by.

benmoss on 4 May 2020

Thanks for looking into this. @benmoss @vincepri

gist: https://gist.github.com/jzhoucliqr/5a5989421b9ab92f6fa1f1fea1805457

Agree that could be some issue with mgmt cluster, especially when all 3 indepedent kcp controller have the same issue at the same time.

I was not able to check exactly what's the state when the issue happens, but from logs of other controllers seems machine objects are there. And the machine objects have creationTimestamp during that period.

jzhoucliqr on 4 May 2020

@jzhoucliqr Did the Machine in question have an OwnerReference under metadata pointing to the KCP controller? If so, I suspect the KCP controller manager cache was lagging behind and never received notifications for new Machines / updated the cache

vincepri on 4 May 2020

yes that's the first thing I checked and they have the right OwnerReference.

jzhoucliqr on 4 May 2020

We can potentially use a direct (uncached) client only for the initialization part to make sure we don't create multiple control planes, although I'd like to see what others think about this. It could be that we decide that we don't want to cover this case given that it seems that the management cluster wasn't in a healthy state. @detiber @ncdc thoughts?

vincepri on 4 May 2020

I think adding additional safety through the use of an uncached client for initialization only makes sense.

detiber on 4 May 2020

👍1

/milestone v0.3.6

vincepri on 4 May 2020

I have a feeling the MachineSet controller would experience a similar issue here. More broadly: anything that is doing a count against cached data that then proceeds to create or delete things could face this (given slow cache updates).

ncdc on 4 May 2020

Anything else (outside of KCP/init) should be able to reconcile back to the expected state in an eventually consistent way, though. Hard to recover from a split-brain scenario from multiple KCP inits, though.

detiber on 4 May 2020

Oh yeah, we should be most protective of KCP.

ncdc on 4 May 2020

/assign
/lifecycle active

vincepri on 4 May 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Bootstrap cluster cleaned up despite failed pivot

dlipovetsky · 5Comments

Adopt KIND as cluster bootstrapper

vincepri · 4Comments

MachineSet is looking at the wrong annotation for Machine deletion

detiber · 5Comments

node deletion can't delete node

chuckha · 4Comments

Can't install pre-release providers from repo with no "latest" release.

rsmitty · 5Comments