Runtime: Low memory notifications for GC on Unix
On Windows GC receives Low Memory notifications from the OS. Such functionality does not seem to exist on Unix. We need to investigate ideas how to deliver such information to the GC (user-mode polling strategy?) to help alleviate OOM and/or changes to allocation behaviors to reduce our OOM killer score on the process.
sergiy-k
All 9 comments
It looks like there is no low memory notification API in Linux and the only way is to poll /proc/meminfo regularly.
janvorli
on 12 Sep 2016
The Same thing is happening on LXD. I had a quick chat with @janvorli at WUG Brno last week and he had pointed this issue out. Should I report it as a different issue?
root@brnpreQA:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
root@brnpreQA:~# lxc version
2.12
itsjbe
on 7 Aug 2017
@jbednarik1 actually, I was thinking about your issue after we've met and I've realized in your case, it might be a different issue that was fixed for 2.0.0. But I am not sure if you've said you were hitting it with 2.0.0 or with previous versions.
janvorli
on 14 Aug 2017
Hi @janvorli. It seems to me we have solved our issue by setting GC to workstation mode.
Feel free to check https://github.com/dotnet/coreclr/issues/9719, if you'd be interested. Thank you anyway. Cheers!
itsjbe
on 27 Aug 2017
We can add the polling into the finalizer thread's FinalizerThread::WaitForFinalizerEvent where we check the low memory notification event for windows. On Linux, the WaitForMultipleObjectsEx already has a timeout that's used for heap dump trigger checks. This timeout is currently set to 10 seconds. We could change it to a lower value (e.g. 100ms - this would need to be determined experimentally based on the impact on the CPU load) and then poll the memory consumption each time it timeouts. We would also need to change the #ifdef to just FEATURE_PAL. If we find that the consumption is approaching the physical memory limit, we would do the same thing that we do for the kLowMemoryNotification event.
We would still want to keep the heap dump trigger to be checked with the original frequency of 10 secs (but only for defined(__linux__) && defined(FEATURE_EVENT_TRACE) to keep the behavior unchanged). This can be easily achieved by a counter counting the timeouts of the wait and checking the g_TriggerHeapDump only when the counter multiplied by the new timeout would reach the 10 secs. Or we can make it more responsive to the g_TriggerHeapDump - instead of checking it on a fixed interval, we would check it on each timeout, unless the previous heap dump was triggered less than 10 secs ago.
For polling the current memory consumption, we can use the PAL_GetWorkingSetSize function.
For getting the physical memory limit, we can use the PAL_GetRestrictedPhysicalMemoryLimit. This function works correctly even in docker container with physical memory limit being set. But it is quite heavy in parsing the cgroups files and so we should read it just once and keep it for the whole lifetime of coreclr.
As for the threshold to trigger GC, we should probably use a percentage of the physical memory limit. We could make the percentage configurable via a new COMPlus_xxxx variable. We can try to set the default to something like 75%.
To test the effect of the change, a simple application that allocates e.g. a byte array of a large size (e.g. 10 MB) in a loop and assigns it to the same variable, waiting for say 0.5 secs between the allocations would probably work. Running this application in a docker container with memory limited to e.g. 100MB on a host system with swap disabled and OOM killer enabled would result in the application getting killed without this change, but should result in its infinite survival with this change.
janvorli
on 25 May 2018
Is there any plan to assign a milestone for this? I believe low memory notifications would resolve issues I'm having while running services in Docker that deal with a high volume of requests.
I'm currently working around it by invoking the garbage collector manually on a schedule.
kriskalish
on 19 Nov 2018
Has anything changed around this in .NET Core 3.0?
Looking at the issue tracker I get the impression that some work has been done for docker, but what about regular servers/VMs?
Is there any special setting needed to have the GC invoked and avoid the memory to grow until it get OOM-killed?
alefranz
on 19 Aug 2019
OK, I was wrong, the GC is taking by default into accounts memory limits outside of containers as well, as pointed out to me by @Maoni0 on Twitter https://twitter.com/maoni0/status/1164939641352404992?s=20
alefranz
on 24 Aug 2019
been talking to @lpereira about the PSI stuff on linux and he believes we can implement the low mem notification on linux that way. CC @janvorli
Maoni0
on 18 Oct 2019
Related issues
yahorsi
路
3Comments
Timovzl
路
3Comments
jzabroski
路
3Comments
omariom
路
3Comments
aggieben
路
3Comments
Most helpful comment
We can add the polling into the finalizer thread's
FinalizerThread::WaitForFinalizerEventwhere we check the low memory notification event for windows. On Linux, theWaitForMultipleObjectsExalready has a timeout that's used for heap dump trigger checks. This timeout is currently set to 10 seconds. We could change it to a lower value (e.g. 100ms - this would need to be determined experimentally based on the impact on the CPU load) and then poll the memory consumption each time it timeouts. We would also need to change the#ifdefto justFEATURE_PAL. If we find that the consumption is approaching the physical memory limit, we would do the same thing that we do for thekLowMemoryNotificationevent.We would still want to keep the heap dump trigger to be checked with the original frequency of 10 secs (but only for
defined(__linux__) && defined(FEATURE_EVENT_TRACE)to keep the behavior unchanged). This can be easily achieved by a counter counting the timeouts of the wait and checking theg_TriggerHeapDumponly when the counter multiplied by the new timeout would reach the 10 secs. Or we can make it more responsive to theg_TriggerHeapDump- instead of checking it on a fixed interval, we would check it on each timeout, unless the previous heap dump was triggered less than 10 secs ago.For polling the current memory consumption, we can use the
PAL_GetWorkingSetSizefunction.For getting the physical memory limit, we can use the
PAL_GetRestrictedPhysicalMemoryLimit. This function works correctly even in docker container with physical memory limit being set. But it is quite heavy in parsing the cgroups files and so we should read it just once and keep it for the whole lifetime of coreclr.As for the threshold to trigger GC, we should probably use a percentage of the physical memory limit. We could make the percentage configurable via a new COMPlus_xxxx variable. We can try to set the default to something like 75%.
To test the effect of the change, a simple application that allocates e.g. a byte array of a large size (e.g. 10 MB) in a loop and assigns it to the same variable, waiting for say 0.5 secs between the allocations would probably work. Running this application in a docker container with memory limited to e.g. 100MB on a host system with swap disabled and OOM killer enabled would result in the application getting killed without this change, but should result in its infinite survival with this change.