Ray: Handling KeyboardInterrupts

[smsxgz]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• Ray installed from (source or binary): pip install
• Ray version: 0.5.0
• Python version: 3.6.4
• Exact command to reproduce: ray.shutdown()

Describe the problem

1. Module 'ray' has no attribute 'shutdown'.
2. And how can I deal with KeyboardInterrupt by myself correctly with ray.

[richardliaw]

shutdown is in the current master but not in the last release. You can install the latest master via pip wheels here: http://ray.readthedocs.io/en/latest/installation.html

Can you provide more information about what you want to do with KeyboardInterrupt?

[smsxgz]

Thank you!

I want to close my workers by myself after receiving the signal of KeyboardInterrupt. But now ray kill workers before I handle with the signal.

[richardliaw]

Great - I've renamed the issue to highlight the interrupt problem.

cc @robertnishihara

[robertnishihara]

Hm this sounds undesirable to me. If you start Ray with ray.init() and then you kill the application with a keyboard interrupt, we don't want to leave a bunch of workers/processes running. #1005 is the exact opposite issue, where we fail to properly kill all of the started processes.

Can you describe your use case and say precisely how you want to shutdown your workers? Can you just register a custom signal handler?

[smsxgz]

In my use case, the main process uses pyzmq to communicate with workers. After receiving the signal of KeyboardInterrupt, the main process send a message like 'close' to workers. Workers do something like saving some parameters to files and shut down after receiving the messages.

I tried the same idea with multiprocessing and got the same question.
So now I don't know if this idea is desirable.

[mitar]

I do think it would be reasonable to be able to intercept ctrl-c, send a bit more to Ray workers, and then initiate ray.shutdown yourself. One could even implement it so that if ctrl-c is pressed twice, it goes and directly calls ray.shutdown.

[robertnishihara]

@smsxgz @mitar are you saying that Ray is preventing you from handling keyboard interrupts? Shouldn't it be possible for your application to intercept the interrupt or is Ray preventing that somehow?

[mitar]

I think the issue is that Ray unconditionally intercepts keyboard interrupt.

Maybe one solution could be that one could register a hook which is run when Ray tries to be shutdown. And then @smsxgz could register its logic there, block until finish, and then let it return, which would then leave to Ray to finish shutting down. Or maybe even hook could be made that if it returns True or something, shutdown does not continue.

[guoyuhong]

I just provide some information.
In services.py, before the driver exits, it will kill all the child processes. It will first send the SIGTERM signal and send the SIGKILL after 1 second.
In raylet backend, it will kill all the worker processes using SIGKILL. Do you mean you want to use SIGTERM here? Since there is only 1 second for raylet to clean its worker child processes. If the workers take too long to handle the SIGTERM signal. The raylet process may exit ahead of workers.

[ericl]

Auto-closing stale issue.

[mitar]

In #3797 you mentioned this has been fixed? Has this been fixed? Can you point to a PR fixing this?