Runtime: CoreCLR should handle SIGTERM as equivalent to Environment.Exit()

[stephentoub]

SIGTERM is meant to be used for graceful shutdown of an app. The runtime should handle it by doing the equivalent of Environment.Exit(). Then with https://github.com/dotnet/corefx/issues/5205, apps will also be able to plug in their own cleanup that'll be triggered on any graceful shutdown, including those triggered by SIGTERM.

As an example, docker stop will initially send a SIGTERM to give the app a chance to gracefully shutdown before it eventually then sends SIGKILL if the process is still alive.
https://github.com/aspnet/Hosting/issues/516

[masonwheeler]

On Windows, a graceful shutdown request is a _request_, generally provided by the user hitting ALT-F4 or clicking the red X. In many cases it is inappropriate to call a shutdown routine at this point; rather you need to confirm with the user and give them the chance to save and/or cancel shutdown. (There's nothing graceful about having the application pulled out from under you; from the end-user's perspective that's a crash, not a graceful shutdown.)

We should do our best to maintain consistency across platforms. SIGTERM should be treated as a polite request to shut down, to be handled in an application-defined manner, not as a kill switch that automatically calls Environment.Exit.

[stephentoub]

not as a kill switch that automatically calls Environment.Exit.

Right now SIGTERM isn't handled at all, so it's a kill switch that's even less graceful than calling Exit, running any Unloading event handlers, etc.

[masonwheeler]

Right now SIGTERM isn't handled at all, so it's a kill switch that's even less graceful than calling Exit, running any Unloading event handlers, etc.

Fair enough. I'm just saying that if we want to handle this, it's important to get it right, no?

[stephentoub]

Fair enough. I'm just saying that if we want to handle this, it's important to get it right, no?

My thinking is this:

• The current state is the worst: it's supposed to be graceful, but because the signal isn't handled at all, the default handling kicks in and the app is simply terminated.
• We can change make a relatively small but impactful change that at least allows the system's normal shutdown to take place, including executing any cleanup handlers the app has registered. libcoreclr can register a signal handler if no other handler has been registered, making it a pure step forward. This doesn't require any additional APIs.
• At some point in the future if/when we provide specialized APIs for signal handling, they can be used to override the default handling.

I don't see us designing, implementing, and shipping that last bullet for an initial release. But since it can be done purely additively on top of the second bullet, I think we should do the second bullet, as in my mind it's significantly better than the first and for relatively little work.

[adityamandaleeka]

Done with dotnet/coreclr#4309

[stephentoub]

Thanks.

[tugberkugurlu]

@adityamandaleeka, @stephentoub might not be relevant here but how should I listen on SIGTERM on a .NET Core console application? My app level entry point accepts CancellationToken and I am planning to trigger a cancellation through a CancellationTokenSource when I receive SIGTERM in my static Main method.

[tugberkugurlu]

Maybe https://github.com/dotnet/coreclr/issues/2688#issuecomment-216841213 is related to https://github.com/dotnet/corefx/issues/3188?

[tugberkugurlu]

might not be relevant here but how should I listen on SIGTERM on a .NET Core console application?

A few other places like https://github.com/Microsoft/vsts-agent/issues/215 suggests that AssemblyLoadContext.Default.Unloading is where one would hook into SIGTERM but it doesn't seem to be working for me.

[nathan-alden-sr]

I am wondering the same thing, @tugberkugurlu. There was a lot of discussion on the implementation details but I can't find any guidance on how I should be implementing graceful shutdown in a .NET Core console application so it behaves the same on any OS.

[axelbodo]

With AssemblyLoadContext.Default.Unloading literally you are enabled to place a shutdown hook, so imo it is a question in what state the runtime is in that point. A related question which I not found here: When I call IApplicationLifeTime.StoppApplication in other place than the Unloading event (e.g. ina controller handler), after the unloading executes, the Kestler will throw an ObjectDisposedException and the ServiceProvider I've created won't call any Dispose on any singleton. there is a related stackoverflow question, but as I see StopApplication suffers from the same issue than the original requestor's problem.

[kouvel]

Finalizers don't run on shutdown in .NET Core, so if the finalizer calls Dispose and the object is held in a static field or is otherwise reachable, Dispose won't be called on shutdown. See dotnet/corefx#5205 for more details.

If you have unmanaged resources that need to be cleaned up on graceful shutdown, the Unloading event can be handled to do so. Since other code may run during or after this, this cleanup and consumers of the unmanaged resources may need to be hardened for this possibility.

[Mattacks]

@kouvel
I’m trying to understand how our .Net Asp.Net Core MVC system should gracefully handle SIGTERM and SIGKILL events when running in Kubernetes.

All the posts I read on SO and GitHub seem to contradict each other.

This post says use AssemblyLoadContext.Default.Unloading, then to use AppDomain.CurrentDomain.ProcessExit

But the official documentation is vague and ambiguous.
AppDomain.CurrentDomain.ProcessExit ‘Occurs when the default application domain's parent process exits.’
AssemblyLoadContext.Default.Unloading (no documentation at all)

In the former it says one has 2 seconds to do the work, but here it says in Kubernetes one has 30 seconds.
The discussion in coreclr git are a mess. Here, for example, it talks about having to look for a keypress event!

How do we do this bit: ‘The system [Kubernetes] waits terminationGracePeriodSeconds (default 30s), or until the Pod completes on it's own.’?

Please could we have some official advice?
Thanks
Matt

[kouvel]

gracefully handle SIGTERM and SIGKILL events

According to this, SIGKILL can't be caught or ignored.

SIGTERM raises the AssemblyLoadContext.Default.Unloading and AppDomain.CurrentDomain.ProcessExit events. The equivalent on Windows is CTRL_CLOSE_EVENT.

The events above are equivalent. The Unloading event was added when AppDomain was initially not going to be exposed in the .NET Core API. That has since changed, so now both events exist.

At the point when these events are raised, nothing significant has been torn down yet, so the event handlers should be able to do most things. Those events are currently raised on the finalizer thread, so finalizers can't be relied upon, even with GC.WaitForPendingFinalizers(). Probably would be good to raise those events on a different thread in the future.

Any exception thrown by an event handler will crash the process.

SIGINT and SIGQUIT are currently treated as abrupt termination and do not raise the events above. However, a Console.CancelKeyPress handler can prevent termination.

AssemblyLoadContext.Default.Unloading (no documentation at all)

Thanks for pointing that out, I'll file an issue.

In the former it says one has 2 seconds to do the work

That is for .NET Framework, the docs need to be updated, I'll file an issue. .NET Core currently does not time-out the event handlers above, though that may change in the future.

but here it says in Kubernetes one has 30 seconds.
How do we do this bit: ‘The system [Kubernetes] waits terminationGracePeriodSeconds (default 30s), or until the Pod completes on it's own.’?

Kubernetes may have its own timeout but that's separate and not in .NET Core's control. The parent of the .NET Core process may for instance issue a SIGTERM and if the process does not exit within a timeout the parent may issue a SIGKILL. A parent process may do something like that to ensure that a child process is torn down in a timely manner. It may be possible to configure the timeout, you'd have to consult the container's docs.

The discussion in coreclr git are a mess. Here, for example, it talks about having to look for a keypress event!

Yea there was a time when SIGTERM was not handled. There was some other bug related to handling the CancelKeyPress event, I don't remember the details. You can ignore these.

[stephentoub]

SIGINT and SIGQUIT are currently treated as abrupt termination and do not raise the events above.

But will also raise the Console.CancelKeyPress event, which can override that termination.

[kouvel]

I was just editing to add that above :)

[Mattacks]

@kouvel @stephentoub Perfect answer for me, thanks.