Runtime: AppDomain.ProcessExit is not invoked on docker stop

[davidmatson]

Describe the bug

Running a Generic Host that uses ConsoleLifetime in a Windows Container (nanoserver/.NET Core or servercore/.NET Framework) doesn't do graceful shutdown when the container is stopped - it just terminates the process.

To Reproduce

Steps to reproduce the behavior:
I'm using VS 2019 with .NET Core 2.2(.6) though I believe this repros with .NET Core 3.0.0-preview6 as well.

Program.cs:
```c#
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using System;

class Program
{
static void Main()
{
using (IHost host = new HostBuilder().ConfigureLogging(l => l.AddConsole()).Build())
{
host.Start();
host.WaitForShutdown();
Console.WriteLine("Shutting down gracefully.");
}
}
}

App.csproj:
```xml
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <SelfContained>false</SelfContained>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.2.0" />
  </ItemGroup>
</Project>

docker run -v X:\path\to\source\bin\Release\netcoreapp2.2\win-x64\publish\:C:\app mcr.microsoft.com/dotnet/core/runtime:2.2.6-nanoserver-1809 C:\app\App.exe
(wait for container to print started message)
docker container stop <id/alias>

Expected behavior

Container output shows "Shutting down gracefully."

Actual behavior

Container terminates without showing that message.

Additional context

It looks like the AppDomain.ProcessExit event doesn't trigger during any Windows container shutdown (regardless of nanoserver vs servercore and .NET Core vs .NET Framework of any version), though I didn't test all combinations.

With 1809, I can get a graceful shutdown notification from the OS directly by P/Invoking SetConsoleCtrlHandler and looking for a callback for CTRL_SHUTDOWN_EVENT.

Note that this event does not get delivered if docker is run with the -t (tty) flag (that interferes with delivering the shutdown signal - see the moby issue for details).

[anurse]

docker stop should be sending SIGTERM which should trigger our graceful shutdown. We should investigate further and see if we can fix this up.

[davidmatson]

Yeah, on Windows, they used to send a CTRL_CLOSE_EVENT, which I think .NET handled, but they changed changed docker stop to do CTRL_SHUTDOWN_EVENT, which I think is not handled by .NET the same way as SIGTERM is. (The moby issue linked above has more details.)

[anurse]

Gotcha. I saw docker and assumed Linux ;)

[sondreb]

The issue that running .NET Core on Docker does not handle shutdown, makes it very advisable to avoid using .NET Core in relation with Docker.

I have searched for hours and looked through countless issues, not able to find any good solution to this problem. Have not found any way of doing a graceful shutdown of .NET Core 3.0 apps running on Docker.

The issue popped up on Linux when the code very often made a database corrupt due to non-graceful shutdown. After this I have done a lot of debugging and research with Docker Desktop on Windows to find a solution, but none so far.

Creating the most basic "Worker Service" project in Visual Studio 2019 and .NET Core 3.0, checking the "Docker" support, and then running on Docker - then running docker stop, should trigger at least some event ASAP in the running code - but there is none, have not found any way to listen, it just eventually is killed and dies horribly.

This basically stops anyone from doing any sensible and production grade deployment of .NET Core 3.0 on Docker. Unless your application is 100% stateless and involves no other local resources, it can be very problematic.

I hope someone comes up with a solution that is easy to implement, at the current state it is not good at all.

This has been an ongoing issue since 2016 from all the github issues I read all over, and looking at the issue referenced above here as well, it clearly says:

Windows containers cannot be gracefully shutdown, either there is no shutdown notification or they are forcefully terminated after a while.

That was posted in September 2017. That is more than 2 years ago.

Though that issue is related to Windows Containers, but the same issue applies to Linux containers.

Anyone else found a usable solution to the issue?

What is the guidance from Microsoft? Should we avoid using Docker until a solution is found? When will that happen, as this has been ongoing for many years in various forms already.

[davidmatson]

@sondreb - it's possible to do graceful shutdown with Docker on Windows today, it just isn't easy. On dotnet/runtime#35990, I posted some workaround code I'm currently using to make it happen.

[sondreb]

Visual Studio Preview has a beautiful new Containers feature that appears while debugging, even clicking the "Stop" button within this fancy new UI, gives the same horrible shutdown experience as manually calling docker stop "image".

@davidmatson I did see your code earlier and download it, I just couldn't believe it is needed. I'm gonna give it a try and see, thakns.

[sondreb]

@davidmatson Your workaround is only for Windows, it doesn't work with Linux containers, it uses native Windows functionality.

[davidmatson]

@sondreb - yes, the workaround I posted is for Windows containers only.

[sondreb]

Done some additional research. Hope this can help anyone else.

This is

• Visual Studio 2019 Preview
• .NET Core 3.1.100-preview3-014645
• Docker Desktop 2.1.0.5
• Windows 10 with Linux containers.

default STOPSIGNAL (not sure what it really is, looks same as SIGTERM):

3
StopAsync

STOPSIGNAL SIGINT:

StopAsync
DONE
2
3

STOPSIGNAL SIGQUIT:

StopAsync
DONE
2
3

STOPSIGNAL SIGTERM:

3
StopAsync

STOPSIGNAL SIGKILL:

Nothing is logged, process dies immediately.

Code

Here is the code:

        public static void Main(string[] args)
        {
            System.Console.CancelKeyPress += (s, e) =>
            {
                Console.WriteLine("1");
            };

            System.Runtime.Loader.AssemblyLoadContext.Default.Unloading += ctx =>
            {
                Console.WriteLine("2");
            };

            AppDomain.CurrentDomain.ProcessExit += (s, e) =>
            {
                Console.WriteLine("3");
            };

            System.AppDomain.CurrentDomain.DomainUnload += (s, e) =>
            {
                Console.WriteLine("4");
            };

            Console.WriteLine("START");

            CreateHostBuilder(args).Build().Run();

            Console.WriteLine("DONE");
        }

And the BackgroundService:

        public override Task StopAsync(CancellationToken cancellationToken)
        {
            Console.WriteLine("Worker: StopAsync");

            return base.StopAsync(cancellationToken);
        }

Observations

One thing to observe here, is that with the default STOPSIGNAL, any code that is added below the .Run() method is not executed. So you should be able to hook up to ProcessExit, or the StopAsync on your BackgroundService, if that is the type of app you are building.

I suggest specifying the "STOPSIGNAL SIGINT" in your Dockerfile, which will ensure that code after .Run() is executed. Obviously if your code takes too long to shut down, it will eventually be killed (default is 10 seconds).

Debugging does not work

It is clear that debugging from Visual Studio does not work properly when the container is being stopped, but depending on your STOPSIGNAL, there are some events that are triggered for .NET Core apps running on Docker, which is good. I hope that Microsoft can do something about this, making it possible to debug shutdown events and logic, and if you have important resources locked up in your code, you might need to restart the PC on every debug session, which is not good at all.

[sondreb]

Have done further investigation of my initial issue with ungraceful shutdowns. The issue was that shutdown was signaled, and the container instance started logging, but shutdown took more than 10 seconds, so it was killed before it was able to flush any log messages indicating that shutdown was being signaled.

So after debugging (tailing) the logs, and then comparing to logs on disk, it is clear that the container is never able to flush when shutdown only has 10 seconds to process.

Using the -t parameter to increase shutdown, made it work as expected.

(This is on Ubuntu 19.10, not Windows)

So primarily the main issue here is that debugging from Visual Studio, no signals are sent from the Docker runtime to your running container. I hope that will be added soon, as it can create issues with resource disposal during development and debugging.

[msftbot[bot]]

As part of the migration of components from dotnet/extensions to dotnet/runtime (https://github.com/aspnet/Announcements/issues/411) we will be bulk closing some of the older issues. If you are still interested in having this issue addressed, just comment and the issue will be automatically reactivated (even if you aren't the author). When you do that, I'll page the team to come take a look. If you've moved on or workaround the issue and no longer need this change, just ignore this and the issue will be closed in 7 days.

If you know that the issue affects a package that has moved to a different repo, please consider re-opening the issue in that repo. If you're unsure, that's OK, someone from the team can help!

[tsu1980]

FYI, This is my workaround to send signal when docker debugging by Visual Studio.

1. click 'open terminal window' in the container view to enter into the docker container.
   

2. In the terminal, you enter pidof dotnet to get pid of entry process.

3. then you enter kill -s SIGTERM <PID> with pid.

Note: Im using docker with ubuntu based image, and entrypoint like ENTRYPOINT ["dotnet", "myapp.dll"].

[msftbot[bot]]

Paging @dotnet/extensions-migration ! This issue has been revived from staleness. Please take a look and route to the appropriate repository.

[anurse]

There are a few crossed-paths going on here. This thread is specific to Windows containers, so let's keep it focused on that. I'll move this over to runtime to track future improvements (likely via hooking CTRL_SHUTDOWN_EVENT as suggested above). If there are issues with Hosting on Linux containers, I'd suggest filing a separate issue on dotnet/runtime for that.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[davidfowl]

cc @stephentoub @richlander thoughts?

[stephentoub]

We probably want to update this:
https://github.com/dotnet/runtime/blob/99aae90739c2ad5642a36873334c82a8b7fb2de9/src/coreclr/src/vm/ceemain.cpp#L400-L404
to include CTRL_SHUTDOWN_EVENT. @jkotas, any reasons not to do that (other than "compat")?

[jkotas]

Sounds fine to me. Maybe we should just do this for all events i.e. delete if (dwCtrlType == CTRL_CLOSE_EVENT).

Found interesting note in Windows docs - from https://docs.microsoft.com/en-us/windows/console/setconsolectrlhandler :

If a console application loads the gdi32.dll or user32.dll library, the HandlerRoutine function that you specify when you call SetConsoleCtrlHandler does not get called for the CTRL_LOGOFF_EVENT and CTRL_SHUTDOWN_EVENT events.

If it is really the case, this event will stop working without warning in more complex apps that happen to load gdi32.dll or user32.dll.

[whihathac]

We would really like to see this in .NET Core 3.1 as we are driving a large fleet of our services from .NET Framework hosted in Cloud Services to .NET Core.

[ericstj]

Retitled this issue. Ultimately applications need a way to gracefully shutdown on docker stop. Previously AppDomain.ProcessExit worked for this, but it no longer does. We should either fix that, or recommend a better way to gracefully stop.

[dekenless]

I find it hard to believe that this at revision 3.1 is still an issue (or is an issue again); this is pretty fundamental and crucial functionality.

[ericstj]

It seems like https://github.com/dotnet/coreclr/pull/17265 attempted to address this same thing but this must have regressed. @jeffschwMSFT any idea who can drive this forward.

[msftbot[bot]]

Tagging subscribers to this area: @vitek-karas
Notify danmosemsft if you want to be subscribed.

[ericstj]

@agocke @vitek-karas is this on your radar for 5.0? This and linked issues seem to have interest.

[agocke]

It was on my radar, but my understanding is that it only affects a small number of Windows container issues. Since this seems to be bigger than I thought I'll see if we can get to this issue soon.

[agocke]

@vitek-karas Could you take a look when you get a chance and see what we should do here?

[agocke]

@sbomer Could you take a look at this?

[danmosemsft]

@agocke it's also a partner ask as @preethikurup is tracking on https://github.com/orgs/dotnet/projects/7

[davidmatson]

@sbomer - will this change fully resolve the repro at the top of this bug? In other words, will the CTRL_SHUTDOWN_EVENT event handler block until Main returns, so that lines like:
c# Console.WriteLine("Shutting down gracefully.")

can run and do graceful shutdown? Or will it only block while running the ProcessExit event, meaning the code after WaitForShutdown may not be able to run?

[sbomer]

It should fix the above issue - I tested with a variety of scenarios including an ASP.NET Console lifetime. If you see different behavior, please let me know and I'd be happy to investigate.

[sbomer]

@davidmatson since I realized you may be asking about the scenario in https://github.com/dotnet/runtime/issues/35990#issue-614281995, let me clarify: the ProcessExit handler on its own won't block until Main returns for a simple console app, but it can be used to set up synchronization with Main (at least within the docker stop timeout). ConsoleLifetime keeps the handler blocked until its Dispose method runs:
https://github.com/dotnet/runtime/blob/551288fae07e3fb064f1fe18f612f87c559ca111/src/libraries/Microsoft.Extensions.Hosting/src/Internal/ConsoleLifetime.cs#L82
https://github.com/dotnet/runtime/blob/551288fae07e3fb064f1fe18f612f87c559ca111/src/libraries/Microsoft.Extensions.Hosting/src/Internal/ConsoleLifetime.cs#L106
So code after the using in your example still isn't guaranteed to run, but code within using should - again, subject to timeouts timeouts, and assuming there aren't other concurrency bugs.

[davidmatson]

Makes sense. Thanks for the clarification, @sbomer!