Rh-che: Should we consider increasing memory limit for Che server?

[amisevsk]

Back when each user's namespace had its own Che server instance, it was necessary to have a conservative limit on the server's memory limit. Now that Che is multi-tenant and the server does not need to be replicated across namespaces, so its memory limit could likely be increased without significant cost.

Currently, the limit is set at 750MB. I recall the server commonly using near 700MB back when each user's namespace hosted an instance, so it's not unreasonable to suspect that this is no longer sufficient if the server is to handle many users at once.

Related: https://github.com/redhat-developer/rh-che/issues/898

[garagatyi]

+1 to increase it to avoid OOM but it is very important to set high priority to the investigation of a source of higher memory consumption. If it is thread/connections pool it is OK, but we might find a memory leak or something else.

[garagatyi]

@ibuziuk @davidfestal WDYT?

[ibuziuk]

sounds good to me, but what figure should we put there for now ? Do you happen to know what memory limits where put for codenvy (I suspect it was also 750 MB ) ?
What I suspect aggravate the situation with OOM is the enormous number of "Failed to store signature keys for ws" events (currently ~290.000) and I suspect it creates extra load to che-server. According to kibana there are ~ 20 workspaces against which failed to store signature keys for ws are performed.

kubernetes.pod_name:"rhche-18-8qk5d" AND kubernetes.namespace_name:"dsaas-production" AND "Failed to store signature keys for ws"

[garagatyi]

Codenvy is way too old to correlate this stuff. If there are no significant constraints from the SD team I would double this memory and plan OOM investigation as soon as possible. At the same time doubling this would show us how much memory consumption grows with the grow of the load.

[ibuziuk]

PR has been sent - https://github.com/redhat-developer/rh-che/pull/903

[garagatyi]

I do not understand something in out JVM config
Here is upstream Che default config:
-XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Xms20m
Here is RhCHe prod config
-XX:+UseParallelGC -XX:MinHeapFreeRatio=25 -XX:MaxHeapFreeRatio=40 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Xms50m -Xmx180m -Dfile.encoding=UTF8
The question is why they are different? Does anybody know?
In particular, here is difference in those configs:
CHE: -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -Xms20m -XX:MaxRAMFraction=2
RHCHE: -XX:MinHeapFreeRatio=25 -XX:MaxHeapFreeRatio=40 -Xms50m -Xmx180m -Dfile.encoding=UTF8
Xms doesn't matter here at all.
Difference in heap growth setting are interesting but let's look further.
In RhCHE we set maximum heap size to 180M whereas in Che MaxRAMFraction=2 sets it to half of the available memory in a container.
As far as I understand since we got OOM the problem was not in a heap space, but rather in overall consumption by heap + stack.
But it would be better to have analysis of JVM on crush to check what was consuming the memory

[ibuziuk]

here is the original PR for tuning memory limits to single-tenant che-server https://github.com/fabric8-services/fabric8-tenant-che/commit/ef748b6eb39017764fe489de278eb512642c8420
Those limits where ported to multi-tenant without changes - https://github.com/redhat-developer/rh-che/commit/195007157fdacce07f8e7322c562a564edad037f

In RhCHE we set maximum heap size to 180M whereas in Che MaxRAMFraction=2 sets it to half of the available memory in a container.

yeah, it looks like 180M for heap was just enough for single-tenant use-case, but for multi-tenant we might need to tune those options (probably fall back on the upstream ?)

[garagatyi]

+1 to reconsider the RAM policy

[ibuziuk]

looking into the original PR there is a comment:

Remove MaxRAMFraction=2 since this would turn out
to be too inflexible for a memory limit of 300MB (i.e. 150MB).
Explicitly set MaxHeapSize=180m instead.

Basically, it was planned that for single-tenant che-server 300MB limit would be sufficiently enough and this is definitely not the case for multi-tenant use-case. I would propose to set MaxRAMFraction=2 and remove -Xmx180m as the initial step.

[garagatyi]

+1

[ibuziuk]

PR has been sent - https://github.com/redhat-developer/rh-che/pull/906
Once this obvious enhancement got merged we could consider adding more JVM config customization if it would be required

[ibuziuk]

Closing: all PRs are merged and now new limits are available on prod for rh-che: