Cluster-api-provider-aws: Root disk runs full, leads to evicted pods

[puja108]

/kind bug

What steps did you take and what happened:
Created a 3 worker single master cluster using the capi-quickstart. All machines were c5.xlarge (tried before also with t3.large and t3.xlarge, to check if it's instance size dependent). Once the cluster was up I ran sonobuoy upstream conformance tests using sonobuoy run --mode=certified-conformance --wait, which after a short while (in below case few seconds, but in some cases few minutes) resulted in:

ERRO[0620] error attempting to run sonobuoy: waiting for run to finish: failed to get status: pod has status "Failed"

Events of the Pod showed eviction due to ephemeral-storage:

  Normal   Scheduled  100s  default-scheduler  Successfully assigned sonobuoy/sonobuoy to ip-10-0-64-18.eu-central-1.compute.internal
  Normal   Pulled     100s  kubelet            Container image "sonobuoy/sonobuoy:v0.20.0" already present on machine
  Normal   Created    100s  kubelet            Created container kube-sonobuoy
  Normal   Started    99s   kubelet            Started container kube-sonobuoy
  Warning  Evicted    68s   kubelet            The node was low on resource: ephemeral-storage. Container kube-sonobuoy was using 48Ki, which exceeds its request of 0.
  Normal   Killing    68s   kubelet            Stopping container kube-sonobuoy

Checking the node it had DiskPressure:

  Normal   NodeHasDiskPressure   2m10s (x2 over 178m)  kubelet  Node ip-10-0-64-18.eu-central-1.compute.internal status is now: NodeHasDiskPressure
  Warning  EvictionThresholdMet  2m2s (x5 over 178m)   kubelet  Attempting to reclaim ephemeral-storage

What did you expect to happen:
Usually, sonobuoy should run through without issues, especially as it is used for upstream conformance instructions

Anything else you would like to add:

The issue is not with sonobuoy per-se, but that was the most reproducible (out of 6 tries, only once did it run quite long without being evicted, and for that one I had added a default storage class to the cluster, not sure if sonobuoy defaulted to using that)

In the past we've seen a lot of issues with the typical 8GB root disks of EC2 instances running full if nodes only use the root disk. Sometimes it was because of a lot of container images, other times because of extensive logging or containers writing too much to their ephemeral-storage.

At Giant Swarm, what we did to mitigate such issues, was to create EBS volumes and mount them for system components to use exclusively. We usually have a disk for docker, another for kubelet, and if etcd is on the master node a separate disk for that. Since we introduced these (and after some production experience increasing the sizes a bit), we didn't run into such issues anymore.

Most probably such a feature (setting a separate disk for e.g. docker or other components) should be optional and configruable, as some use cases might be ok with the root disk or smaller additional disks and depending on the provider the context is different. Not sure if this might need a way to manage the interaction between the infra provider and bootstrap provider in CAPI.

Environment:

• Cluster-api-provider-aws version: v0.6.3
• Kubernetes version: (use kubectl version): 1.19.5 (also tested on 1.18.9 and 1.19.2)
• OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS

[randomvariable]

You can resize the disk via the API and also add EBS volumes, in the RootVolume and NonRootVolumes fields of AWSMachine.

Is the issue that we should change the default disk size, which can be done in the https://github.com/kubernetes-sigs/image-builder project?

With regards to making easy to assign the extra disk for etcd, we don't have a decent UX, given that we need a combination of adding the disk here, and then also adding a KCP configuration that formats/mounts the disk to the right place. @vincepri has some ideas around an easy UX for this for v1alpha4, but in the meantime, may be worth trying to create a suitable cluster template for that.

FWIW, we are running conformance tests on CAPA, albeit with kubetest directly instead of Sonobuoy.

[randomvariable]

Thinking about it, maybe we should move this as a use case to the Cluster API repo?

[puja108]

Ah, didn't see that possibility to add it in the AWSMachine and then use it in the KCP. A better UX for that would be great. @vincepri is there already a proposal or something I can check out and maybe even help with?

Not sure if there needs to be a changing of the default disk size, as I believe keeping the root volume as is and add adding right-sized disks for things that need it (like docker for container images, kubelet for logs and ephemeral storage, and etcd for it's data) makes more sense and is a cleaner setup.

@randomvariable You might be right that as a use case this might make more sense in the CAPI repo as it's not very AWS specific. I just happened to try CAPA as a first thing.

Also, really appreciate your answers, I'm still learning a lot around CAPI and your answers have helped my understanding a lot already.

[randomvariable]

@puja108 happy to help any time.

There's some write up here about UX from @vincepri , but it's super early thoughts: https://hackmd.io/Z9tzIkGJQUaU9VbFuo6I7Q