Runtime: High virtual memory usage on Ubuntu 16

[shravan2x]

_This is a cross post from https://github.com/aspnet/Home/issues/2052 and https://github.com/aspnet/KestrelHttpServer/issues/2141. I am also posting here under the recommendation of @benaadams._

I run a number of ASP .NET Core 1.1 servers on Ubuntu. This is the htop view of one of the processes:

The process appears to be using 20G of VIRT. I understand that this is not actually allocated memory, but is the highest among all processes running on the entire system. This is view of a number of apps:

Every one of the 20.xG VIRTs is an ASP.NET Core app (and every other app in the 1000M range is mono). The actual ASP.NET server itself runs on Kestrel, is very simple and serves only static pages via MVC.

1. Is there a reason for such a high VIRT?
2. Why does an app that only serves static pages have 47M of _private_ memory? (I'm aware some other language like PHP or even plain HTML might be better suited for my use, but I use ASP since it is more customizable and fun to work with)

Please note that since this scenario lies on a production server, the version is 1.1. I apologize if it has already been resolved in another issue.

UPDATE: This issue still persists on .NET Core 2.0 and ASP.NET Core 2.0.

[janvorli]

@shravan2x please note that the high virtual memory usage is not an issue. It doesn't cost you anything (except for a bit of memory for the processor page tables), since this space is private to each process. So even if it was using hundreds of gigabytes of VM or more, it would still be ok. Is there any specific reason why you are worried about the VM space usage?

[shravan2x]

No, I am primarily just curious to know why it is so. Does .NET Core have some unusual memory usage pattern that leads to this?

EDIT: I was just thinking about this a little more, and it appears that almost 25MB of RAM are used simply to maintain the page tables (this was the result of some quick mental math, please correct me if I'm wrong).

Each page table can map 4Kb*1024=4Mb
Page tables needed 20G/4M=5120
So 5120*4K=20M for the depth 4 tables alone.
Of course, this assumes no page tables were reused. It also assumes the allocated memory was contiguous. Ignore this, I was confused.

[Suchiman]

I'm not sure if this is the same thing, but i can observe similiar behavior on my raspberry pi 2 with ubuntu 17.04 and 17.10. Virtual memory steadily increases until it dies due to OutOfMemory above 2GB (working set raising from 50MB initial to about 120MB when it dies). I couldn't really diagnose this since loading libsosplugin in lldb just segfaults but i've noticed that it appears to be leaking threads (starting with less than 20 and dying with over 300). I've since moved it to windows where the same .NET Core 2 version has a steady memory consumption and no steady increasing amount of threads

[janvorli]

@shravan2x GC reserves a continuous range of virtual memory for the heap. Actual allocations from that heap then just commit segments of that virtual memory. The GC heap needs to be in a continuous range of virtual memory so that we can quickly find if an address belongs to the heap or not. GC code maintains g_lowest_address and g_highest_address global variables that represent the GC heap range.
If we didn't reserve a virtual memory range at once, there would be no guarantee of continuous space where only GC heap can live.

[shravan2x]

@Suchiman I think your issue is unrelated, you should consider filing a separate issue.

@janvorli Thanks for explaining, that makes a lot of sense. So if only 20G of heap space is reserved, how does .NET Core handle applications that use 20GB+?

[Maoni0]

@Suchiman I agree with @shravan2x - please open a separate issue if you are still interested in debugging the problem you described.

@shravan2x GC heap actually isn't in a continuous range of VM - the g_lowest_address/g_highest_address just indicates the range which means we have segments and non GC managed memory inbetween. GC will acquire new segments (each segment is a continuous range of VM) if it needs to and will adjust g_lowest_address/g_highest_address if need to.

[shravan2x]

@Maoni0 I see. So in theory, are .NET apps hard limited to 20G? Or will g_highest_address move up if necessary?

[jkotas]

It moves up as necessary

[Maoni0]

and g_lowest_address will move down as necessary.

[shravan2x]

But could you tell me why this doesn't seem to be an issue with Mono?

[Maoni0]

this is not an issue - asp.net core chooses to use Server GC by default (which is a good choice) which reserves more and larger segments than Workstation GC. you can turn off Server GC or make it use fewer heaps if you have a good reason to make it not reserve as much VM.

[shravan2x]

Thanks for clarifying!

[janvorli]

@shravan2x I am sorry for confusing you before, I haven't realized that the range between g_lowest_address and g_highest_address can also contain stuff that's not part of the GC heap.

[vancem]

Just for reference you can control whether your .NET Core application's GC parameter using the
*.runtimeconfig.json file using these keys (merge it in with your existing keys.

{
  "runtimeOptions": {
    "configProperties": {
      "System.GC.HeapCount": 5,
      "System.GC.NoAffinitize": true, 
      "System.GC.Server": true
    }
  }
}

The default for HeapCount (only for server) is the number of processors, but you can cut it down here.
NoAffinitize means that GC threads do not HAVE to run on a particular processor (good if you are running other things on the machine that compute with the GC but less efficient when that is not the case).