Kestrelhttpserver: Graceful Shutdown Empty Response

[cphillips83]

I have attached a demo gist of the problem. I have tried shutting down Kestrel in several ways and they all produce the same result.

Test:

First run this command which will start a request that delays until the 2nd request is sent
curl http://localhost:5000/api/values/1

This request attempts to shutdown Kestrel
curl -X DELETE http://localhost:5000/api/values/1

Problem:

• Kestrel keeps running until the ShutdownTimeout duration elapses, even after all requests have finished. If you set this to something like 5 minutes it will just sit there the entrie time.
• Kestrel does not return a response even after return "value"; executes
• Curl finally exits with curl: (52) Empty reply from server after Kestrel ShutdownTimeout elapses

References:
https://github.com/aspnet/KestrelHttpServer/issues/1026 - Graceful shutdown and draining
https://github.com/aspnet/KestrelHttpServer/issues/247 - Hangs for 2-3 seconds when a request is made even after its served

test code

[cesarblum]

Investigating.

[cesarblum]

Your code causes a deadlock higher up in the stack.

Cancelling the token calls ApplicationLifetime.StopApplication(), on which WebHostExtensions.Run() waits while running. After that is signaled, WebHost is disposed. WebHost.Dispose() disposes DI ServiceProvider. ServiceProvider.Dispose() locks on its ResolvedServices property. ServiceProvider.Dispose() in turn calls KestrelService.Dispose(), which will wait for some time (ShutdownTimeout) until all connections are drained.

On the thread were your first request is being serviced, Get has returned a value and MVC is trying to execute the result of that action. ObjectResult.ExecuteResultAsync() calls ServiceProvider.GetRequiredService<>(), and that leads to a call to CallSiteRuntimeResolver.VisitScoped(), which will try to lock on ServiceProvider.ResolvedServices, which is already taken as a lock in ServiceProvider.Dispose() as outlined above.

Here are the call stacks:

Thread 1

    System.Private.CoreLib.ni.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
    System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.WaitAllBlockingCore(System.Collections.Generic.List<System.Threading.Tasks.Task> tasks, int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)    Unknown
    System.Private.CoreLib.ni.dll!System.Threading.Tasks.Task.WaitAll(System.Threading.Tasks.Task[] tasks, int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)   Unknown
    Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.Dispose()    Unknown
    Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose() Unknown
    Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceProvider.Dispose() Unknown
    Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose() Line 256   C#
    Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost host, System.Threading.CancellationToken token, string shutdownMessage) Line 96   C#
    Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(Microsoft.AspNetCore.Hosting.IWebHost host, System.Threading.CancellationToken token) Line 60   C#
    1501.dll!_1501.Program.Main(string[] args) Line 25  C#

Thread 2

    Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite scopedCallSite, Microsoft.Extensions.DependencyInjection.ServiceProvider provider)    Unknown
    Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceProvider.RealizeService.AnonymousMethod__0(Microsoft.Extensions.DependencyInjection.ServiceProvider provider)  Unknown
    Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(System.IServiceProvider provider, System.Type serviceType)   Unknown
    Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService<Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor>(System.IServiceProvider provider)    Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.ObjectResult.ExecuteResultAsync(Microsoft.AspNetCore.Mvc.ActionContext context)  Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeResultAsync(Microsoft.AspNetCore.Mvc.IActionResult result)   Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeResultAsync>d__15>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeResultAsync>d__15 stateMachine)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeResultAsync(Microsoft.AspNetCore.Mvc.IActionResult result)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.State next, ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Scope scope, ref object state, ref bool isCompleted)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextResultFilterAsync()   Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__18>(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResultFilterAsync>d__18 stateMachine) Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextResultFilterAsync()   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.State next, ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Scope scope, ref object state, ref bool isCompleted)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()    Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__20>(ref Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__20 stateMachine)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()    Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Next(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.State next, ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Scope scope, ref object state, ref bool isCompleted)    Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeNextResourceFilter() Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__18>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__18 stateMachine) Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeNextResourceFilter() Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Next(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.State next, ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.Scope scope, ref object state, ref bool isCompleted)    Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeFilterPipelineAsync()    Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__13>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__13 stateMachine)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeFilterPipelineAsync()    Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeAsync()  Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeAsync>d__11>(ref Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.<InvokeAsync>d__11 stateMachine)   Unknown
    Microsoft.AspNetCore.Mvc.Core.dll!Microsoft.AspNetCore.Mvc.Core.Internal.ResourceInvoker.InvokeAsync()  Unknown
    Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)    Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4>(ref Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4 stateMachine) Unknown
    Microsoft.AspNetCore.Routing.dll!Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext)    Unknown
    Microsoft.AspNetCore.StaticFiles.dll!Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)    Unknown
    Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)  Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6>(ref Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6 stateMachine) Unknown
    Microsoft.AspNetCore.Diagnostics.dll!Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context)  Unknown
    Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext httpContext) Line 53 C#
    Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.Internal.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context context) Line 86 C#
    Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()  Unknown
    System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__2>(ref Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.<RequestProcessingAsync>d__2 stateMachine) Unknown
    Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame<Microsoft.AspNetCore.Hosting.Internal.HostingApplication.Context>.RequestProcessingAsync()  Unknown
    Microsoft.AspNetCore.Server.Kestrel.dll!Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.Start() Unknown
    System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)    Unknown
    System.Private.CoreLib.ni.dll!System.Threading.QueueUserWorkItemCallbackDefaultContext.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()   Unknown
    System.Private.CoreLib.ni.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()   Unknown

I'm not sure what you're trying to do is a supported scenario. Paging @pakrym @davidfowl @NTaylorMullen @rynowak

[cphillips83]

This was an over simplification of a problem that I was trying to resolve but was able to reproduce with this example. I wanted to make sure it wasn't something deeper in our code base so I created an empty dotnet webapi project to run my tests in.

Our scenario is that we spin up new micro services to replace old ones as new versions become available (continuous delivery). We use Consul for service discovery and when the new ones come online they replace the old serviceID of the previous micro service. We want the old version of the microservice to drain and finish all its requests and exit.

The problem we kept running in to is when we requested kestrel to gracefully shutdown, all the active connections would fail and stall out rather than complete as intended.

NOTE: I tried requesting Kestrel to shutdown via an external thread rather than on a request and get the same result. This leads me to believe that the deadlock exists in all scenario's if you try to gracefully shutdown Kestrel.

Is there another way that I missed to ask Kestrel to drain all its connections properly and exit?

[cphillips83]

To be perfectly clear. I want to be able to signal Kestrel to exit and have it handle any active connections it has left before returning back to the command prompt. Even if those active connections take upwards of 60-120 seconds as a worst case scenario.

[cphillips83]

@cesarbs I've been looking through this all weekend and read your diagnoses probably 10 times. While it makes sense what you are saying, I still fail to see how to resolve this. You've mentioned this is possibly an unsupported scenario, but the scenario is nothing more than wanting to stop Kestrel and fulfilling the remaining requests without taking new ones.

What I'm not understanding is why are things being disposed and locked when you request Kestrel to be stopped. Shouldn't Kestrel simply stop the listening socket and enter a drain mode until all connections are completed (or shutdown timeout lapses) before anything is even disposed?

At a higher level I imagine something like this (pseudo code).

```

public void Stop(){
_running = false;
}

public void Run(){
if(_running) return;

_listenSocket.Start();

_running = true;
while(_running){
    ProcessActiveRequests();
    //sleep(0) or something?
}
_listenSocket.Stop();

//request to stop
var time = currentTime;
while(time + _shutdownTime > currentTime && _activeRequests.Count > 0){
    ProcessActiveRequests();
    //sleep(0) or something?
}

if(_activeRequests.Count > 0){
    //log that we aborted {_activeRequests.Count} requests
}


//Dispose resources

}

[cphillips83]

Since this is critical for me, I gave up on trying to determine how to handle this properly inside Kestrel and implemented my own version of drain. I'm putting this here in case anyone else needs a work around for the time being.

In program.cs, create and use a cancellation token source.

        public static readonly CancellationTokenSource CancellationTokenSource = new CancellationTokenSource();

Pass the token to Run

        host.Run(CancellationTokenSource.Token);

While this code doesn't belong here, I'm putting it here for illustration purposes. In Startup.cs, create some static variables to track state.

        private static int _activeConnections;
        private static bool _stopRequest = false;

Add a method for Stop.

        public static void Stop()
        {
            _stopRequest = true;
            while (_activeConnections > 0)
            {
                System.Threading.Thread.Sleep(1000);
                Console.WriteLine($"Connections: {_activeConnections}");
            }

            Console.WriteLine("Stopping");
            Program.CancellationTokenSource.Cancel(false);
        }

And finally in Configure, add a custom Use to track connections and to return 500 when the system is shutting down for all new connection attempts.

I believe these work as a chain (hence next), so this should be first before all other UseXYZ.

            app.Use(async (http, next) =>
            {
                if (_stopRequest)
                {
                    var response = http.Response;
                    response.StatusCode = 500;
                    return;
                }

                Interlocked.Increment(ref _activeConnections);
                try
                {
                    await next();
                }
                finally
                {
                    Interlocked.Decrement(ref _activeConnections);
                }
            });

And finally, you can call Startup.Stop() anywhere you like and it will drain all active connections, return 500 to any new connections and then cleanly exit instantly without any dead locks.

[avtc]

It works even without adding CancellationTokenSource in Program.
You just need to subscribe to applicationLifetime.ApplicationStopping in Configure.

public void Configure(IApplicationBuilder app, 
            IHostingEnvironment env, 
            ILoggerFactory loggerFactory, 
            IApplicationLifetime applicationLifetime
        )
...
applicationLifetime.ApplicationStopping.Register(Stop);
...

Also, in case of using some sort of load balancer, like docker swarm, it could be handy to redirect to the same url, so request will be processed by other running nodes:

                if (_stopRequest)
                {
                    var response = context.Response;
                    response.StatusCode = 308;
                    response.Headers.Add("Location", $"{context.Request.Scheme}://{context.Request.Host}{context.Request.Path}{context.Request.QueryString}");
                    return;
                }

[avtc]

Created middleware based on a proposed workaround.
https://github.com/avtc/GracefullShutdown

[davidfowl]

I think this will solve the problem:

https://github.com/aspnet/Hosting/issues/947

Lets track the issue over there

[cesarblum]

@cphillips83 Kestrel supports graceful shutdown, but by default it gives active connections a short period of time to complete their work (5 seconds). You can increase the shutdown timeout by setting KestrelServerOptions.ShutdownTimeout to a value suitable to your requirements.

[algirdask]

I cannot find KestrelServerOptions.ShutdownTimeout in 2.0 release. Was it removed/moved somewhere else? API documentation in web is still present

[Tratcher]

It was moved to IWebHostBuilder: https://github.com/aspnet/Hosting/blob/03bdb40f8a07427bbd83c89154c02da2ec92aea9/src/Microsoft.AspNetCore.Hosting.Abstractions/HostingAbstractionsWebHostBuilderExtensions.cs#L168

[FrederickBrier]

I downloaded and was reading the WebHostBuilderTests in Hosting to try and understand if there was a similar mechanism that avtc created with his GracefulShutdown extension. I need to save state once the Kestrel web server is shutdown and has no active connections. It seems that the Hosting is a generic IServer lifecycle management, and not doing active connection counting. Is there a similar hook or singleton in the DI that I can reference to determine if Kestrel is quiescent? Thank you.