Kestrelhttpserver: a problem when I dynamically add many dynamic instances of Kestrel

Created on 24 Jan 2018 · 7Comments · Source: aspnet/KestrelHttpServer

Hi, sorry for my Google English.

I am a problem when I dynamically add many dynamic instances of Kestrel.
At the moment I have 1200 ports of the same IP responding to different executables.
Every time I've passed a number of ports I have an error:

Exception of type 'System.OutOfMemoryException' was thrown. at System.Thr
eading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
at System.Threading.Thread.Start(StackCrawlMark& stackMark)
at System.Threading.Thread.Start(Object parameter)
at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.StartAsync()
at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Start(Int32 coun
t)
at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.StartTContext
at Microsoft.AspNetCore.Hosting.Internal.WebHost.Start()
at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host, Cancellat
ionToken token, String shutdownMessage)
at Alterdata.In.Infrastructure.Crosscutting.SystemServer.ServerManager.NewHost
ing(WebServer Server)

            if (!(new Ping(Server)).Active)
            {
                Server.Host.Instance = new WebHostBuilder()
                    .UseKestrel()
                    .UseUrls($"{Server.Host.Protocol}://{Server.Host.HostName}:{Server.Host.Port}")
                    .UseContentRoot(Server.ContentRoot)
                    .UseStartup(Server.StartupAssemblyName)
                    .Build();

                Server.Host.Instance.Run(Server.CancellationTokenSource.Token);
            }
            else if (!Server.Flag) 
                KillServer(Server);

Thank you

Source

fabiobraganet

All 7 comments

At the moment I have 1200 ports of the same IP responding to different executables.

Enquiring minds have to know.... Why?

mgravell on 24 Jan 2018

👍1

@fabiobraganet If possible, it's far more efficient to have a single Kestrel instance listen to all 1200 ports. I can't say for sure that this will fix your OOM (OutOfMemoryException), but it very well could considering that Thread.Start is throwing the exception.

Each Kestrel instance spawns a bunch (half the logical cores of the host CPU by default) of new IO/libuv threads outside of the threadpool. Each of these threads take a lot of memory for things like stack space.

If request to each of the 1200 ports need unique behavior, you can always look at the "Host" header in middleware and dispatch appropriately.

To bind Kestrel to a bunch of different ports at once, you can just use semicolons to delimit the addresses passed to UseUrls i.e:

    .UseUrls("{Server.Host[0].Protocol}://{Server.Host[0].HostName}:{Server.Host[0].Port}"
        + ";"
        + $"{Server.Host[1].Protocol}://{Server.Host[1].HostName}:{Server.Host[1].Port}");

Since you have so many endpoints, you'll probably want to use KestrelServerOptions.Listen(IPEndPoint) instead of UseUrls though.

halter73 on 25 Jan 2018

👍1

Thank you very much for your attention. Thank you! Thank you!

Hi mgravell - "Opserver (y)"

Enquiring minds have to know.... Why?

I have many business modules and many clients. The division is useful for planning N balances. And I take advantage to realize the multi tenant appropriate for the business.

Hi halter73

UseUrls

I can try this method again. But I need to shut down a specific host at any time. Define SLAs...
I will validate with the IT infrastructure. I can not guarantee that it is as before. It worked well with 600 hosts per application instance, I have how to initialize many instances of the same exe with arguments and define which ports I will generate on them.

My other great requirement is to create a new host or delete at runtime.

If request to each of the 1200 ports need unique behavior, you can always look at the "Host" header in middleware and dispatch appropriately.

I'll test this (y)

Thank you!

fabiobraganet on 25 Jan 2018

@fabiobraganet HttpSys is one of our other servers and it supports dynamically adding and removing address bindings: https://github.com/aspnet/HttpSysServer/blob/e40cf72fcefc5ab86678930f114821f4fdafe5a5/samples/HotAddSample/Startup.cs#L41

Tratcher on 25 Jan 2018

👍1

Quick rough calculation

1400 Instances
8 Logical Cores (4 core processor)
1/2 Thread per logical core
8mb (for Ubuntu Default) of stack per thread
= 44,800mb or 43gb of ram needed just to start the threads