Cluster-api-provider-aws: Provisioning CentOS 7 control plane node fails for v1.18.2

[mehyedes]

/kind bug

What steps did you take and what happened:
Using the quick start guide, I tried to provision a new workload cluster using the CentOS 7 AMI ami-0d4f64f4bfd565fd2 for the kubernetes version v1.18.2.

$ clusterctl config cluster test-cluster  --control-plane-machine-count=1 --worker-machine-count=1 --from cluster-template-CENTOS.yaml --kubernetes-version v1.18.2 > centos-cluster.yaml
$ kubectl apply -f centos-cluster.yaml

Here, the cluster-template-CENTOS.yaml file is just a copy from cluster-template.yaml with the AMI set to ami-0d4f64f4bfd565fd2.
The control plane instance was created, but then failed to finish the init phase, which lead to the kubernetes API server being unavailable.

What did you expect to happen:
The control plane node to be successfully initialised and the kubernetes API server to be available.

Anything else you would like to add:
By checking the logs of the capi-controller-manager, found a lot of errors like:

E0729 15:51:02.132617       1 controller.go:248] controller-runtime/controller "msg"="Reconciler error" "error"="error creating dynamic rest mapper for remote cluster: Get https://internal-test-cluster-apiserver-473277167.eu-central-1.elb.amazonaws.com:6443/api?timeout=32s: EOF" "controller"="machine" "name"="test-cluster-control-plane-5fhdh" "namespace"="default"
E0729 15:52:24.008084       1 controller.go:248] controller-runtime/controller "msg"="Reconciler error" "error"="error creating dynamic rest mapper for remote cluster: Get https://internal-test-cluster-apiserver-473277167.eu-central-1.elb.amazonaws.com:6443/api?timeout=32s: EOF" "controller"="machine" "name"="test-cluster-control-plane-5fhdh" "namespace"="default"

The kubelet service also fails to start on the control plan node:

[root@ip-10-0-7-121 ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2020-07-29 15:59:51 UTC; 967ms ago
     Docs: https://kubernetes.io/docs/
  Process: 9196 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 9196 (code=exited, status=255)

Jul 29 15:59:51 ip-10-0-7-121.eu-central-1.compute.internal systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jul 29 15:59:51 ip-10-0-7-121.eu-central-1.compute.internal systemd[1]: Unit kubelet.service entered failed state.
Jul 29 15:59:51 ip-10-0-7-121.eu-central-1.compute.internal systemd[1]: kubelet.service failed.

And the reason seems to be a missing kubelet configuration

kubelet[9618]: F0729 16:07:32.556549    9618 server.go:199] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file ordirectory
systemd[1]: Unit kubelet.service entered failed state.
systemd[1]: kubelet.service failed.

The cloud-init.log file also showed some failures:

# grep FAIL /var/log/cloud-init.log
2020-07-29 15:50:45,462 - handlers.py[DEBUG]: finish: init-network/consume-user-data: FAIL: reading and applying user-data
2020-07-29 15:50:45,462 - handlers.py[DEBUG]: finish: init-network: FAIL: searching for network datasources
2020-07-29 15:50:54,533 - handlers.py[DEBUG]: finish: modules-final/config-scripts-user: FAIL: running config-scripts-user with frequency once-per-instance
2020-07-29 15:50:54,763 - handlers.py[DEBUG]: finish: modules-final: FAIL: running modules for final

Environment:

• Cluster-api-provider-aws version: v0.5.4
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release): Mac OSX

[EleanorRigby]

@codenrhoden @kkeshavamurthy
This is the exact same problem on RHEL 7 also.

[randomvariable]

/assign @codenrhoden
/lifecycle active
/priority important-soon

[codenrhoden]

ah, well I guess it's a good thing this is fixed in the current master of image-builder. I just need to build new CentOS 7 OVAs with the fix!

[randomvariable]

If I build new public CentOS images, this should be fixed?

[ncdc]

Waiting on a new tag of image-builder, then we can build updated AMIs

[sedefsavas]

Image-builder is tagged. Will create new AMIs with it and publish them soon.

[randomvariable]

All the new AMIs are in
https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/1892

I ran out of time to PR it in.

[randomvariable]

@mehyedes, this should be fixed now. feel free to reopen if you still have a problem.

[mehyedes]

Hi @randomvariable
apologies for the belated feedback. I have just tried provisioning a new cluster with the new CentOS 7 AMI and I can confirm that it works fine 👍
Thank you for the fix