Kops: Reserve kube/system resources (Node Allocatable Resources)

[hjacobs]

We should reserve CPU and memory resources for Kubernetes and system daemons like described in the Node Allocatable Resources Design Proposal.

The necessary flags need to be passed to kubelet to get different values for allocatable and capacity:

• --kube-reserved mapStringString
  A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for kubernetes system components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. [default=none]
• --system-reserved mapStringString
  A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. [default=none]

The cluster scheduler should also honor these reservations: https://github.com/hjacobs/kube-aws-autoscaler/issues/11

NOTE: We are not using Kops right now, this is merely a copy of https://github.com/zalando-incubator/kubernetes-on-aws/issues/258 for users of Kops (as I think it's a valuable improvement for scheduling).

[blakebarnett]

This is now documented as Best Practice: https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#best-practices

[itskingori]

@justinsb @chrislovecnm just wondering ... is there a reason we don't do this? 🤔 Sounds pretty important that we should reserve some memory capacity for system daemons (kernel, kubelet, etc.) and evict pods at x% memory utilization to reduce thrashing and incidence of system OOM ... as per the best practices.

[chrislovecnm]

We would need new flags / API values https://github.com/kubernetes/kops/blob/master/pkg/apis/kops/componentconfig.go here

The API machinery would need to be regenerated.
Defaults added to kubelet model, I think that is in nodeup.

Docs added and e2e testing completed.

And bob is you uncle, aka we would have it done.

I believe this would be an excellent addition. Not a hard task, and a good task for someone to learn more about kops.

Is there any migration or k8s version specifics needed?

[chrislovecnm]

FYI - we have docs under the dev section about our API machinery.

[chrislovecnm]

Can anyone list some good default values to be used? System i/o is always tough to guard for headroom.

[itskingori]

I believe this would be an excellent addition. Not a hard task, and a good task for someone to learn more about kops.

@chrislovecnm I'm keen to give this a stab given some guidance.

[chrislovecnm]

Ping me on slack if you have questions. You will need a full dev build as well, make upload

[itskingori]

@chrislovecnm see https://github.com/kubernetes/kops/issues/2982.