Micronaut-core: Support for graceful shutdown [Feature Request]

[sollecitom]

At the moment it's not obvious to me how to prevent a shutdown signal sent to the application from immediately stopping the Netty HTTP server.
I can intercept the events generated by the signals, but by then is too late.

Ideal behaviour:

There should be a way of specifying custom logic so that it's run when a soft process kill signal is received, before actually shutting down the HTTP server.
This could be used in a number of ways, including ensuring that all in-progress HTTP requests have a grace time window to drain before the connections are closed to the clients.

I also tried replacing the NettyHttpServer with a wrapper EmbeddedServer that would override function stop() before delegating to the underlying NettyHttpServer, but I couldn't find a way to replace the implementation.

Apologies if such a feature already exists, but I couldn't find any mention / example of it.

[graemerocher]

This may be possible to implement by listening for a ApplicationShutdownEvent and blocking for a configured period then resuming when it is ready to shutdown. Would that not work?

[ktholen]

@graemerocher I am running into the same issue. It looks like the requests are still in flight (specifically writing to redis using the micronaut-redis client). When I tried adding the following code (based on your suggestion), it appears to still be happening.

@Singleton
class ApplicationHooks : ApplicationEventListener<ShutdownEvent> {
    companion object {
        private val logger = LoggerFactory.getLogger(ApplicationHooks::class.java)
    }

    override fun onApplicationEvent(event: ShutdownEvent) {
        logger.info("ShutdownEvent before wait")
        Thread.sleep(10000)
        logger.info("ShutdownEvent after wait")
    }
}

Let me know if you have any other ideas and I will try them out.

[gokhanoner]

@graemerocher any update on this one? Even with Application Event listener, the first message printed is this one

23:59:42.630 [Thread-0] INFO  io.micronaut.runtime.Micronaut - Embedded Application shutting down.

so application is technically started shutting down & Micronaut prevent accessing any APIs. The use case here is Cloud deployment, especially AWS ECS/Fargate or similar serverless environments. Example: Fargate spot instances can send a SIGTERM to the instance 2 min prior to actual container shutdown so we need a way to prevent app shutdown but disable some APIs manually which tiggers load balancer to remove that node from the group & based on some other config, start a new deployment etc.. So it's important to handle SIGTERM before micronaut start shutting down or tell micronaut not to start shutdown.

[graemerocher]

Try ApplicationShutdownEvent instead of ShutdownEvent

[graemerocher]

Seems as well you could also throw an exception from your event listener and then use `

Micronaut.builder().mapError(MyException.class, (throwable) -> {
    // error handling logic here
});

[gokhanoner]

@graemerocher ApplicationShutdownEvent is working, thank you.

[x80486]

Shouldn't this be a built-in feature where one can activate or not? Something like shutdown: graceful | right_away.

Then Micrnoaut should take care automatically about the remaining resources, connections, etc. I mean, this could be tricky to implement but I think it's a must to have these days.

[graemerocher]

@x80486 PRs welcome

[jameskleeh]

@x80486 @graemerocher What would this feature do compared to what is already possible? Through the event listeners and the event loop group shutdown settings (quiet period, timeout), it seems the use case is covered.

[x80486]

@jameskleeh, I'm eventually not familiar with it. I can see that Micronaut waits for a while before shutting down completely the embedded server, but I don't know if, for instance, all related resources used by the application — messaging, HTTP traffic, etc. — are going to be managed appropriately while on that phase.

[jameskleeh]

@x80486 All related resources should be handled gracefully. We can't have a central feature in core that ensures your Kafka listeners are shut down correctly for example. As far as I know all things that we are responsible for are shutdown with sensible defaults. Are you aware of a feature or resource that is not managed appropriately for shutdown?

[x80486]

No, I can't really tell because I'm just starting (for real) with couple of services with Micronaut.

[jameskleeh]

@x80486 Thanks - then it seems like this issue is resolved