Amazon-eks-ami: Kube/System Reserved resources should be enabled by default

[max-rocket-internet]

What would you like to be added:

I think a small amount of CPU and memory should be reserved for kubelet by default: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved

Why is this needed:

For stability.

I was recently running a load test and had some nodes go NotReady with error: Kubelet stopped posting node status. I know it's possible to add kubelet args manually but I think this should be the default.

I lot of people with related problems in https://github.com/awslabs/amazon-eks-ami/issues/79

For reference, a GKE node with 2 CPUs and 7.5GB of memory has this in /home/kubernetes/kubelet-config.yaml:

evictionHard:
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
kubeReserved:
  cpu: 70m
  ephemeral-storage: 41Gi
  memory: 1736Mi

Like many aspects of EKS, it would be great if we just copied GKE 😅

[whereisaaron]

I am guessing in the vast majority of cases users would prefer nodes not to go offline when they become oversubscribed, so let's have that as the default please 😄

[stefansedich]

Something that just bit us too (my fault ultimately for not setting better resource limits) but would be great to be safe by default!

@max-rocket-internet does GKE set anything for system-reserved by default?

[jnatten]

@stefansedich
Yes GKE reserves some amount of resources based on instance type.
https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu

[stefansedich]

@jnatten I took that was for kube-reserved, do they assign the same for both or split it up somehow?

[abelgvidal]

If you are having OOM errors, EKS support suggestion is:

--kubelet-extra-args "--kube-reserved memory=0.3Gi,ephemeral-storage=1Gi --system-reserved memory=0.2Gi,ephemeral-storage=1Gi --eviction-hard memory.available<200Mi,nodefs.available<10%"

and "adjust to your needs".

So while we wait for this, we are using the GKE doc as reference

[max-rocket-internet]

Resolved in https://github.com/awslabs/amazon-eks-ami/pull/350

[whereisaaron]

Related, eksctl has added support for specifying these kubelet options as part of your cluster config file.

...
    kubeletExtraConfig:
        kubeReserved:
            cpu: "300m"
            memory: "300Mi"
            ephemeral-storage: "1Gi"
        kubeReservedCgroup: "/kube-reserved"
        systemReserved:
            cpu: "300m"
            memory: "300Mi"
            ephemeral-storage: "1Gi"
        evictionHard:
            memory.available:  "200Mi"
            nodefs.available: "10%"
        featureGates:
            RotateKubeletServerCertificate: true
...