Joblib: Parallel crashed on python 3.6 when running in vscode debugger

[fans3210]

My environment is python 3.6 and I ran a simple script file like below:

from math import sqrt
from joblib import Parallel, delayed


def main():
    print('simulate workers, testing parallel')
    Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))


if __name__ == '__main__':
    print('simulate workers, testing parallel')
    Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))

And facing a crash:

Below is the log details:
E00001.438: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/site-packages/joblib/externals/loky/backend/semaphore_tracker.py:74: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak.
warnings.warn('semaphore_tracker: process died unexpectedly, '
E00001.467: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

E00001.481: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

E00001.515: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

The only solution I can use is to set" "prefer = 'threads'" in the function call. Appreciated if someone could help!

[fans3210]

Note. This is tested by using vscode debugger. Tried to run this simple example from terminal and things works fine

[zwlanpishu]

the same problem as you, wonder someone have a solution？

[mridunarang]

Facing the same problem. Any update on the solution?

[machineCYC]

Facing the same problem when using MultiOutputRegressor in sklearn.multioutput.

Have a solution to fix this?

[prittjam]

I've also encountered this.

[ChienDuong]

I also faced this problem on Vscode! Anyone has the solution?
I tried to debug on Pycharm, it worked.

[Eralien]

Hey, I saw a solution by amittleider and it worked for me. Paste it here:

On Ubuntu, putting this at the top of the main entrypoint file worked for me:

import multiprocessing
multiprocessing.set_start_method('spawn', True)

[yongqianxiao]

I got the same problem with Vscode too, any solution?

[juescuder]

Thanks @Eralien it worked like a charm...or should I say like a pycharm?
import numpy as np
import pandas as pd
import multiprocessing
multiprocessing.set_start_method('spawn', True)
from sklearn import preprocessing

[shlomiamitai]

Works in Mac as well :)

[BernierCR]

Thanks, great advice.

[johannespitz]

Thanks, worked also with Python3.7.

[cognitiveRobot]

Doesn't work for pytorch.

[johannespitz]

Doesn't work for pytorch.

As far as I remember I had to import a second package, but for me the easy workaround was to set num_workers=0 in the Dataloader constructor.

[BrettLL]

Works in python3.6 as well :)

[anerisheth19]

Do the following to avoid the semaphore error:
1) Write the following piece of code in the beginning of the .py file you are debugging.
import multiprocessing multiprocessing.set_start_method('spawn', True)
2) In the debug configurations file - launch.json, add the following:
"subProcess": true,
and for Pytorch DataLoader function, set num_worker threads as '0'.

Hope this helps you! It worked for me.

[sergioamr]

@anerisheth19 That worked to debug my celery application, thanks.