Joblib: Loky backend doesn't cleanup worker processes

[rbedi]

Python 3.7.2, macOS 10.13.3 and Ubuntu 18.04

I notice when using the Loky backend, joblib doesn't clean up after itself even when explicitly calling _terminate_backend(). Here's a minimal example:

from joblib import Parallel, delayed
from multiprocessing import active_children

def f(x): return x**2
par_loky = Parallel(n_jobs=32, backend="loky")
par_loky(delayed(f)(i) for i in range(10))

print(len(active_children())) # Prints 32
par_loky._terminate_backend()
print(len(active_children()))  # Prints 32

Same effect if I use the context manager to construct the pool:

from joblib import Parallel, delayed
from multiprocessing import active_children

def f(x): return x**2

with Parallel(n_jobs=32, backend="loky") as par_loky:
    par_loky([delayed(f)(i) for i in range(10)])

print(len(active_children())) # Prints 32

However, with the multiprocessing backend, it works as expected:

from joblib import Parallel, delayed
from multiprocessing import active_children

def f(x): return x**2
par_mp = Parallel(n_jobs=32, backend="multiprocessing")
par_mp(delayed(f)(i) for i in range(10))
print(len(active_children())) # prints 0, as expected

[tomMoral]

Hi,

thanks for reporting. This is actually the expected behavior for loky. The loky backend rely on spawn to start the new processes for the pool of worker. As this method may take up to few tenth of seconds, this can be quite costly to start a new pool each time you need to call Parallel. To that end, loky manage a reusable pool of workers that is reused for multiple call to Parallel, hence not terminated even once the Parallel object is cleaned up with _terminate_backend.

If they are not reused, the processes will be cleaned up if they time out (default is 300s and apparently there is no way to modify that yet). If you need to force the clean-up of such process, you could call:

from joblib.externals.loky import get_reusable_executor
get_reusable_executor().shutdown(wait=True)

Let me know if this solves your problem. As for the API, would it be better if you controlled the timemout delay for the worker or would you need to directly clean up the processes with imperative instruction?

[ogrisel]

Maybe we should improve the joblib documentation.