Kestrelhttpserver: .net core slow upon first request

[cex-pnaik]

.net core is slow upon first request, and then subsequent requests are super fast.
Please refer to following stats:

First hit to application:

[18:18:55 INF] Request starting HTTP/1.1 GET http://localhost:5008/api/Print/Test
[18:19:08 INF] Executing action method Controller.Test (Service) with arguments (null) - ModelState is Valid
[18:19:09 INF] Executing JsonResult, writing value { Message = Test success }.
[18:19:13 INF] Executed action Controller.Test (Service) in 9159.67ms
[18:19:13 INF] Request finished in 18372.9308ms 200 application/json; charset=utf-8

Second hit:

[18:21:46 INF] Request starting HTTP/1.1 GET http://localhost:5008/api/Print/Test
[18:21:46 INF] Executing action method Controller.Test (Service) with arguments (null) - ModelState is Valid
[18:21:46 INF] Executing JsonResult, writing value { Message = Test success }.
[18:21:46 INF] Executed action Test (Service) in 64.5092ms
[18:21:46 INF] Request finished in 85.1069ms 200 application/json; charset=utf-8

Can someone please tell me, why it is taking so much time for first request?
How this delay can be avoided?

[davidfowl]

Can someone please tell me, why it is taking so much time for first request?

A profiler would tell you exactly what is taking time:

• https://www.jetbrains.com/profiler/
• https://blogs.msdn.microsoft.com/dotnet/2012/10/09/improving-your-apps-performance-with-perfview/

[davidfowl]

Also if you can share an application (preferably on the github repo) then we can take a look.

[cex-pnaik]

Link to the application is as follows:
https://github.com/punu31090/Test

For running it on my Linux machine, I followed below steps:
1)Published my application
2)Then on my Linux machine, run following command:
dotnet Test.dll

The stats obtained are as follows:
[13:05:29 INF] Request starting HTTP/1.1 GET http://localhost:5000/api/values/Test
[13:05:34 INF] Executing action method Test.Controllers.ValuesController.Test (Test) with arguments (null) - ModelState is Valid
[13:05:34 INF] Executing JsonResult, writing value { Message = Test success }.
[13:05:37 INF] Executed action Test.Controllers.ValuesController.Test (Test) in 4372.3698ms
[13:05:37 INF] Request finished in 9105.4474ms 200 application/json; charset=utf-8
[13:06:02 INF] Request starting HTTP/1.1 GET http://localhost:5000/api/values/Test
[13:06:02 INF] Executing action method Test.Controllers.ValuesController.Test (Test) with arguments (null) - ModelState is Valid
[13:06:02 INF] Executing JsonResult, writing value { Message = Test success }.
[13:06:02 INF] Executed action Test.Controllers.ValuesController.Test (Test) in 36.4619ms
[13:06:02 INF] Request finished in 44.6691ms 200 application/json; charset=utf-8

[benaadams]

@punu31090 might be runtime MVC view compilation and discovery; try precompilation (search for "View compilation" in https://blogs.msdn.microsoft.com/webdev/2016/11/16/announcing-asp-net-core-1-1/ for example on how to do this)

[davidfowl]

@benaadams There's no view compilation here. It's an API app

[davidfowl]

@punu31090 Did you publish it in release mode?

[davidfowl]

Here are my results for your application:

Publish

parallels@ubuntu:~/Test$ dotnet publish -c Release
Publishing Test for .NETCoreApp,Version=v1.0
Project Test (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
Configuring the following project for use with IIS: '/home/parallels/Test/bin/Release/netcoreapp1.0/publish'
Updating web.config at '/home/parallels/Test/bin/Release/netcoreapp1.0/publish/web.config'
Configuring project completed successfully
publish: Published to /home/parallels/Test/bin/Release/netcoreapp1.0/publish
Published 1/1 projects successfully

Navigate to the output folder

parallels@ubuntu:~/Test$ cd bin/Release/PublishOutput

Run

parallels@ubuntu:~/Test/bin/Release/PublishOutput$ dotnet Test.dll
Hosting environment: Production
Content root path: /home/parallels/Test/bin/Release/PublishOutput
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.

Hit http://localhost:5000/api/Values/Test

[08:30:20 INF] Request starting HTTP/1.1 GET http://localhost:5000/api/values/Test  
[08:30:21 INF] Executing action method Test.Controllers.ValuesController.Test (Test) with arguments (null) - ModelState is Valid
[08:30:21 INF] Executing JsonResult, writing value { Message = Test success }.
[08:30:21 INF] Executed action Test.Controllers.ValuesController.Test (Test) in 444.9514ms
[08:30:21 INF] Request finished in 1091.4302ms 200 application/json; charset=utf-8

[cex-pnaik]

@davidfowl I am publishing application in release mode only.
But in my case it took about 9 secs for first hit, and in your case its 1sec.
Where am I going wrong?

[davidfowl]

Can you reproduce it with those exact steps on another machine?

[cex-pnaik]

Ok let me check on another machine.
What Linux system you are testing with?
I am using Ubuntu 15.04 x86_64

[davidfowl]

I'm on 14.04

[cesarblum]

@halter73 looked into this some time ago. IIRC the delay comes from way down in the stack (OS level I think, out of our control). Let's wait for his input on this.

[cex-pnaik]

@davidfowl I tested my application on another machine.
On another machine, the time taken for first request is same like yours.

But on my machine, the same thing is taking 9 secs which is too much.
Now I am planning to migrate my application to .net core 1.1 and will do the testing. Will post my findings here.

[cex-pnaik]

I migrated my application to .net core 1.1.
But still no improvement.

Following are results of my application.
Now first hit took 16 secs.

[17:19:45 INF] Request starting HTTP/1.1 GET http://localhost:5000/api/values/Test  
[17:19:54 INF] Executing action method Test.Controllers.ValuesController.Test (Test) with arguments (null) - ModelState is Valid
[17:19:54 INF] Executing JsonResult, writing value { Message = Test success }.
[17:20:00 INF] Executed action Test.Controllers.ValuesController.Test (Test) in 7698.3497ms
[17:20:00 INF] Request finished in 16312.1185ms 200 application/json; charset=utf-8
-------------------------------------------------------------------------------------------------------------------
[17:20:22 INF] Request starting HTTP/1.1 GET http://localhost:5000/api/values/Test  
[17:20:22 INF] Executing action method Test.Controllers.ValuesController.Test (Test) with arguments (null) - ModelState is Valid
[17:20:23 INF] Executing JsonResult, writing value { Message = Test success }.
[17:20:23 INF] Executed action Test.Controllers.ValuesController.Test (Test) in 32.2122ms
[17:20:23 INF] Request finished in 43.4136ms 200 application/json; charset=utf-8

[ZOXEXIVO]

@punu31090 I have same "issue", when running my app on slow Virtual Machine with shared slow HDD.

[cex-pnaik]

@ZOXEXIVO how did you fixed it?

[ZOXEXIVO]

@punu31090 I moved to fast VM with ssd

[muratg]

@punu31090 did you try moving to an SSD too?

[muratg]

@punu31090 if you still have issues, please re-open or file another issue.

[skynyrd]

We are also suffering from this problem. When our .net core 1.1 API deployed, and we run our load tests, the first test always fails (6 req/sec) and after running the same test again and again, it becomes faster. (200 req/sec). There is no cache, and for the dependent microservices, the timeout value is 1 sec.

Could you find a solution @cex-pnaik ?

[HumanWannabe]

The only solution seems to be doing this first request yourself, before the user asks for it.

In my application, I am using ASP.NET Core for only part of the interface that is Web-based. When the main application loads, I fire-up Kestrel (as a background process) and then fire-up a BackgroundWorker that performs the first request. If the user makes a request after the BackgroundWorker is done, it is almost instant. If the user does the request before the BackgroundWorker is done, still part of the resources will be preloaded so it's faster. And since everything is done in a BackgroundWorker there is no application start-up cost.

I hope this helps.

[skynyrd]

Yes, the solution you have provided is useful, and it seems that we have no alternative option. Paper over the cracks...

[williamtetlow]

For anybody searching for solution for slow initial request when using .NET Core, Docker & Kubernetes: https://blog.markvincze.com/running-asp-net-core-in-auto-scaling-containers-warm-up/

Same basic idea, setup task to do initial request before users are able to make requests.

[taochenglin]

@cex-pnaik Have you fixed it?I'm suffering from this question too.

[briziomusic]

+1
I need this too. I need something managed. I can't discover all my api endpoints and run requests because all my apis requires authentication

[ryanlederman]

I am using curl to warm my server up–both when started manually and by systemd in case it crashes or the system reboots:

curl -so /dev/null -XGET --unix-socket /path/to/netcore/server/kestrel.sock http://images

I would still like to see this built in, though. See my repository for more information on configuring a .NET Core server as a systemd service.