Presto: SIGTERM doesn't wait GracefulShutdownHandler completion

[xerial]

Steps to reproduce this problem:

• Set /v1/info/state to SHUTTING_DOWON for invoking GracefulShutdownHandler
• Run launcher.py stop sends TERM signal to JVM and it will start LifeCycleManager.stop process (Guice injected modules will start @PreDestroy processes)

Problem:

• GracefulShutdownHandler's thread executor is not a part of this life cycle:
  https://github.com/prestodb/presto/blob/92309cb907a11446bd4afc0d03f8826e61e5ab75/presto-main/src/main/java/com/facebook/presto/server/GracefulShutdownHandler.java#L48
  
  
  
  • So after receiving SIGTERM JVM process terminates quickly even if there are tasks whose states are not yet synchronized to the coordinator (2min. grace period by default).
  
  

Question: @nezihyigitbasi
GracefulShutdownHandler should be aware of SIGTERM? If so GracefulShutdownHandler should have @PreDestroy shutdown hook to wait task completion and its state synchronization to the coordinator.

This simplifies lifecycle management of Presto worker processes. For example, for autoscaling, we only need to set SHUTTING_DOWN state and run launcher.py stop for graceful scaling-in of presto worker nodes.

[electrum]

You shouldn't ever have to send SIGTERM (or run launcher stop). GracefulShutdownHandler should forcibly terminate the JVM after a 30s timeout using System.exit(0) in DefaultShutdownAction. Is it not existing on it's own?

Handling SIGTERM in this manner is very difficult. When the JVM receives SIGTERM, it starts all the shutdown hooks which will run concurrently in an arbitrary order. Airlift LifeCycleManager.stop() is triggered via a shutdown hook, which will run all of the @PreDestroy methods in reverse creation order.

In order to reliably block in shutdown, we'd want to add a handler to LifeCycleManager that gets run at the start of the stopping phase. The order in which @PreDestroy methods get called is too arbitrary for this.

However, there's still the problem that all the other shutdown hooks will run, and there's no telling what effect that might have on various libraries. For example, I think the HDFS code does cleanup using them, so it's likely anything Hadoop file system related would break.

[xerial]

@electrum Thanks for the comment. Actually, we don't need to call launcher stop, but we sometimes see the following error in the coordinator log if we terminate EC2 instances after the graceful shutdown:

Dec  1 03:06:22 ip-172-18-153-237 presto[15387]: 2017-12-01T03:06:22.339Z#011WARN#011TaskInfoFetcher-20171201_025133_00501_rdjpk.2.1-996#011com.facebook.presto.server.remotetask.RequestErrorTracker#011Error getting info for task 20171201_025133_00501_rdjpk.2.1: java.net.NoRouteToHostException: No route to host: http://172.18.148.95:8081/v1/task/20171201_025133_00501_rdjpk.2.1

And this causes the query failure (usually PAGE_TRANSPORT_TIMEOUT, etc.). We will try adding more grace period before shutting down the instance.

I understand adding shutdown hook for GracefulShutdownHandler could be tricky because we usually need to start the termination process of the other modules as well such as Hadoop related components as you mentioned.

[xerial]

No further action is necessary for this. I'll close this ticket. Thanks.