joblib leaves zombie processes on windows

[isaacgerg]

I am using sklearn, and thus in conjunction joblib, on Windows. I am using the Wing IDE to debug my program. However, everytime I hit stop while running it, joblib leaves several python processes open and does not kill them. Is there a way around this? My fix is to manually delete all of them but this can be quite error prone (if i delete another python program i am running) and time consuming (after a few runs, i easily have 50 python.exe's sitting there doing nothing).

[ogrisel]

Thank you for the report. Indeed workers should self-cleanup.

When fixing this issue we should also check whether the concurrent.futures module in the standard library of recent Python versions (e.g. Python 3.8) has a similar problem or not.

[ogrisel]

@pierreglaser you might want to have a look at this one while working on other Windows-related issues :)

[ogrisel]

@isaacgerg I cannot reproduce this on my Windows VM: as soon as I terminate the process of the main Python program, all the auxialiary Python processes for the loky workers used by joblib are also terminated as expected.

I tried with joblib 0.15.1.

Maybe when you "stop" you Python program it is actually not terminated but still alive? If so this is expected that that the worker processes stay alive as well doing nothing for a couple of minutes. However as soon as you terminate the main program, the loky worker processes should also be terminated without delay.

[andrei-rusu]

Not sure if this is related to Windows or Jupyter, but I am experiencing the same issue in Notebook sessions on Windows. Forgive my intervention if this is to be expected, but I did not find any related thread yet. Processes get killed after a longer period, even when running the sample code _delayed(sqrt)_ etc. This makes testing things quite annoying. Is there any way to force kill the processes after they successfully returned?