Kubeapps: Review limits and requests defaults for our chart

[migmartri]

Currently we are not setting resources defaults for neither limits nor requests but we have commented out 128Mi in the values.yaml.

We should double check those values and offer a set of sensible defaults that can run the app under QOS K8s threshold.

[migmartri]

Bumping the priority since Monocular will start using these services and its other services are properly configured resources wise. https://github.com/helm/monocular/pull/509/files

[migmartri]

See below a table that contains some usage data from an instance of Kubeapps that I've had running for more than a month, along with the proposed memory requests and limits.

My goal, for this first pass, is to promote our containers from the BestEffort QOS class. That does not mean to go to the Guaranteed QOS for a couple of reasons:

1 - We do not know yet what good limits are without (under/over)committing
2 - I do not believe that the kind of workload that Kubeapps runs require of that tier (with the exception of MongoDB, although I decided to treat it like the rest of the components for now). All this taking into account that sooner than later we will have multiple replicas so being killed/re-scheduled should not be a huge deal https://github.com/kubeapps/kubeapps/issues/643
3 - Guaranteed QOS should be decided by the cluster op in my opinion.

That's why I decided to target Burstable in all our tiers except the recurring update jobs which I think it is reasonable to have them running as best effort for now.

Although most of the tiers seems to run consistently under/at 32Mi I decided to give them some leeway since my environment is not a production one so we should expect higher resources consumption. For the same reason the upper limit is way over any gathered runtime value.

As a second step (not in the scope of this issue), and once we have the requests and limits in place, we can optimize these ranges via cadvisor's kube_pod_container_resource_requests_memory_bytes, container_memory_usage_bytes.

Memory

Tier name | Current MAX | Current AVG | Targeted QOS | New resource request | New resource limit
------------ | ------------- | ------ |------ |------ |------ |
Frontend| 30MB| 15MB|Burstable|32Mi|128Mi
Dashboard|17MB|15MB|Burstable|32Mi|128Mi
TillerProxy|68MB|27MB|Burstable|32Mi|128Mi
Chartsvc|25MB|11MB|Burstable|32Mi|128Mi
MongoDB|365MB|305MB|Burstable|256MI|512Mi
ApprepositoryController|18MB|15MB|Burstable|32Mi|128Mi
App repo sync job|5MB|5MB|BestEffort|N/A|N/A

CPU

The CPU case is trickier, I've been looking at some default setups for Nginx Ingress, Monocular API server and others and they seems to stick to 100m as request. We could try this, more taking into account that the bulk of the CPU will be handled by the appSync job.

So my take for now (we can revisit it later once we get some comparative data) would be: request: 100m, limits: 500m

I want to make clear that this is a first pass and these numbers are for sure wrong, but the high limit boundary should reduce the risk. Once we put these values in place we will be able to see how our instances react and tweak it as suggested above.

@prydonius, @andresmgot what do you think?

[prydonius]

This is great @migmartri, perhaps we should avoid putting limits on CPU for now if we're not sure what makes sense? Also, I imagine MongoDB CPU usage will be higher?

[migmartri]

This is great @migmartri, perhaps we should avoid putting limits on CPU for now if we're not sure what makes sense? Also, I imagine MongoDB CPU usage will be higher?

That's a good point. I am ok with it. WRT mongo, I have noticed that upstream mongo chart has those CPU resources commented out, I am not sure if they have been tested https://github.com/helm/charts/blob/master/stable/mongodb/values.yaml#L113

[andresmgot]

The proposed values look good to me. Thanks for the investigation! (It's a pity that mongodb requires 8 times more resources than the rest)

[stormmore]

So I am currently using the following for all the resource sections:

    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi

I suspect in some places that might be overly conservative and will probably mix up the memory using the values in this issue.

[migmartri]

I've been running an instance with the proposed limits https://github.com/kubeapps/kubeapps/pull/666 and this is what I have under light usage.

Below you can see two graphs, on the left is the % of memory usage compared to its requests and on the right the same but compared to its new memory limits.

As you can see, all tiers except MongoDB use less memory than even the requested value (as expected). W.R.T the limits, all tiers seems to be quite far away from the ceiling, MongoDB is running at ~50% though. We should keep an eye on it, but so far I think this setup is low risk.

Regarding CPU, see below a similar pair of charts, which show that all are running far from its limits, and even requests, so 100m might be quite generous in some cases. Let's keep an eye on it.

[migmartri]

I have done some extra tweaking based on more observations. These tweaks affect the CPU requests values which have been reduced from 100m to 25m. As well as in the MongoDB case from 200m to 50m.

Even with those changes, it seems that the system runs well under the requested values. NOTE: I have not modified the limits, which are very generous. The goal is to reduce the requirements to run Kubeapps to a sensible default, more so now that we run multiple replicas in most tiers.

Using the new proposed CPU requests