Joblib: SIGTERM after all workers done in 0.11

[stroykova]

Hello!

This problem relates to v0.11. I can not reproduce it in master, so maybe this was fixed occasionally or not.
I did not find any issue about it, so maybe it will be interesting as information.

from joblib import Parallel, delayed 
import multiprocessing 
import signal 

def f(i): 
    return multiprocessing.current_process().pid 

def f3(): 
    print('f3') 
    def signal_handler(signum, frame): 
        raise ValueError("received SIGTERM signal") 
    signal.signal(signal.SIGTERM, signal_handler) 
    for r in Parallel(n_jobs=5, verbose=1000)(delayed(f)(i) for i in range(20)): 
        print(r) 

This code listens to SIGTERM.

If I call f3 function sometimes(!) I see a ValueError:

In [3]: f3()
f3
[Parallel(n_jobs=5)]: Done   1 tasks      | elapsed:    0.0s
[Parallel(n_jobs=5)]: Batch computation too fast (0.0011s.) Setting batch_size=374.
[Parallel(n_jobs=5)]: Done   2 tasks      | elapsed:    0.0s
[Parallel(n_jobs=5)]: Done   3 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   4 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   5 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   6 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   7 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   8 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done  18 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done  20 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done  20 out of  20 | elapsed:    0.0s finished
5260
5261
5263
5264
5262
5260
5261
5262
5261
5263
5260
5260
5260
5260
5260
5260
5260
5260
5260
5260

In [4]: f3()
f3
[Parallel(n_jobs=5)]: Done   1 tasks      | elapsed:    0.0s
[Parallel(n_jobs=5)]: Batch computation too fast (0.0013s.) Setting batch_size=304.
[Parallel(n_jobs=5)]: Done   2 tasks      | elapsed:    0.0s
[Parallel(n_jobs=5)]: Done   3 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   4 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   5 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   6 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   7 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   8 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done   9 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done  20 out of  20 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=5)]: Done  20 out of  20 | elapsed:    0.0s finished
Process ForkPoolWorker-15:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/opt/conda/lib/python3.6/site-packages/joblib/pool.py", line 364, in get
    rrelease()
  File "<ipython-input-1-f5429385facc>", line 13, in signal_handler
    raise ValueError("received SIGTERM signal")
ValueError: received SIGTERM signal

Maybe you will need ten or more tries to reproduce, but sometimes this happens.

I noticed this issue while using airflow. I launch my airflow dag with joblib code and it always is getting a SIGTERM signal because airflow tasks listen it to stop the job if necessary.

There is also an old task about that.

Close the issue please if it is known and fixed in new version.

P.S. When a new version is planned?

Thanks!

[ogrisel]

I would like to release tomorrow I think.

[GaelVaroquaux]

+1 for release. It seems that all major issues are addressed.

Gaël

⁣Sent from my phone. Please forgive typos and briefness.​

On Jun 21, 2018, 09:44, at 09:44, Olivier Grisel [e-mail] wrote:

I would like to release tomorrow I think.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/joblib/joblib/issues/702#issuecomment-399007854

[tomMoral]

This issue is probably solved in master because we do not use ForkProcess anymore. With fork, the processes will share the signal_handler, which will be triggered when we forcefully kill the workers at the end of the Parallel call.

With the new version, the signal_handler is not passed to the workers so you do not see the ValueError anymore.

OTOH, note that if you intent to use a signal handler in the child processes, you need to specifically set it at the beginning of your function and it would be best to remove it at the end to avoid such issue.

[ogrisel]

Let's close this now that @tomMoral has explained why it does not happen in master :)