Joblib: Misleading ImportError when using Parallel inside a "with Parallel(...) as" block (backend='multiprocessing')

[lesteve]

from math import sqrt
from joblib import Parallel, delayed

input_list = [x**2 for x in range(10)]


def main():
    with Parallel(n_jobs=3, backend='multiprocessing') as parallel:
        output = Parallel(n_jobs=2, backend='multiprocessing')(delayed(sqrt)(i) for i in input_list)
        return output

if __name__ == '__main__':
    print(main())

ImportError                               Traceback (most recent call last)
/tmp/test_joblib_reload.py in <module>()
     12 
     13 if __name__ == '__main__':
---> 14     print(main())

/tmp/test_joblib_reload.py in main()
      8     with Parallel(n_jobs=2) as parallel:
--->  9         output = Parallel(n_jobs=2)(delayed(sqrt)(i) for i in input_list)
     10         return output
     11 

/home/lesteve/miniconda3/lib/python3.5/site-packages/joblib/parallel.py in __call__(self, iterable)
    764         self._aborting = False
    765         if not self._managed_pool:
--> 766             n_jobs = self._initialize_pool()
    767         else:
    768             n_jobs = self._effective_n_jobs()

/home/lesteve/miniconda3/lib/python3.5/site-packages/joblib/parallel.py in _initialize_pool(self)
    513                 already_forked = int(os.environ.get(JOBLIB_SPAWNED_PROCESS, 0))
    514                 if already_forked:
--> 515                     raise ImportError('[joblib] Attempting to do parallel computing '
    516                             'without protecting your import on a system that does '
    517                             'not support forking. To use parallel-computing in a '

ImportError: [joblib] Attempting to do parallel computing without protecting your import on a system that does not support forking. To use parallel-computing in a script, you must protect your main loop using "if __name__ == '__main__'". Please see the joblib documentation on Parallel for more information

It is misleading because 1) I am on Linux so my system supports forking 2) I am using a ifmain guard. Not sure what we should do here and if there is an easy way to detect this.

For completeness, I originally saw the error in https://github.com/scikit-learn/scikit-learn/issues/6258 and only found time to trace it back recently.

[fasiha]

In your block starting with with Parallel(n_jobs=3) as parallel:, do you actually use parallel (lowercase p) anywhere in the block? Why do you use with ... as and then create a new Parallel object? Just curious, I’m running into the same error (but very different circumstances) on Mac.

[lesteve]

This is just a snippet that reproduces the problem and not meant to be a real use case at all.

About your problem, the best thing to do is to create a separate issue with a code snippet reproducing the problem. If you got here by using scikit-learn, please open an issue in the scikit-learn repo and not the joblib one.

[lesteve]

Edit: this snippet may not apply any more 2017-10-04
For the record, I bumped into the same error, when I forgot a delayed in this snippet, not a great error message:

from joblib import Parallel, delayed

import multiprocessing


def func():
    return multiprocessing.current_process().pid


def parallel_func():
    return Parallel(n_jobs=2)(delayed(func)() for _ in range(2))

if __name__ == '__main__':
    print(Parallel(n_jobs=2)(parallel_func() for _ in range(3)))  # forgot delayed around parallel_func here

[Axik]

couldn't reproduce it on 0.11.1.dev0. Is this still relevant issue?

[lesteve]

Thanks for trying it out. The default backend has changed so I can reproduce the problem with the first snippet by setting backend='multiprocessing'. I'll edit the snippet.

About the second one, I am not sure, it looks like we get a different error with backend='multiprocessing' and it looks like it is hanging with backend='loky' (which is the default).

[essandess]

I see this issue on macOS in a Jupyter notebook working with scikit-learn and multicore processing. This MWE tickles the issue:

import numpy as np, numpy.random as npr
from sklearn import cluster

data = npr.poisson(1,(100,10))
algorithm = cluster.KMeans
algorithm_kwargs = dict(n_clusters=4,n_jobs=-1)
estimator = algorithm(**algorithm_kwargs)
labels = estimator.fit_predict(data)

The code runs as expected when run as a Python script.

However, under a Jupyter notebook, it throws this error:

ImportError                               Traceback (most recent call last)
<ipython-input-64-5a0071cf17dc> in <module>()
      3 algorithm_kwargs = dict(n_clusters=4,n_jobs=-1)
      4 estimator = algorithm(**algorithm_kwargs)
----> 5 labels = estimator.fit_predict(data)

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/cluster/k_means_.py in fit_predict(self, X, y)
    915             Index of the cluster each sample belongs to.
    916         """
--> 917         return self.fit(X).labels_
    918 
    919     def fit_transform(self, X, y=None):

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/cluster/k_means_.py in fit(self, X, y)
    894                 tol=self.tol, random_state=random_state, copy_x=self.copy_x,
    895                 n_jobs=self.n_jobs, algorithm=self.algorithm,
--> 896                 return_n_iter=True)
    897         return self
    898 

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/cluster/k_means_.py in k_means(X, n_clusters, init, precompute_distances, n_init, max_iter, verbose, tol, random_state, copy_x, n_jobs, algorithm, return_n_iter)
    361                                    # Change seed to ensure variety
    362                                    random_state=seed)
--> 363             for seed in seeds)
    364         # Get results with the lowest inertia
    365         labels, inertia, centers, n_iters = zip(*results)

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable)
    747         self._aborting = False
    748         if not self._managed_backend:
--> 749             n_jobs = self._initialize_backend()
    750         else:
    751             n_jobs = self._effective_n_jobs()

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in _initialize_backend(self)
    545         try:
    546             n_jobs = self._backend.configure(n_jobs=self.n_jobs, parallel=self,
--> 547                                              **self._backend_args)
    548             if self.timeout is not None and not self._backend.supports_timeout:
    549                 warnings.warn(

/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in configure(self, n_jobs, parallel, **backend_args)
    303         if already_forked:
    304             raise ImportError(
--> 305                 '[joblib] Attempting to do parallel computing '
    306                 'without protecting your import on a system that does '
    307                 'not support forking. To use parallel-computing in a '

ImportError: [joblib] Attempting to do parallel computing without protecting your import on a system that does not support forking. To use parallel-computing in a script, you must protect your main loop using "if __name__ == '__main__'". Please see the joblib documentation on Parallel for more information

I'm not sure if this is an issue with joblib or some downstream issue with Jupyter, but it is a problem.

The very same issue arises in related libraries, such as HDBSCAN. The following alternate code tickles the issue with HDBSCAN:

import hdbscan

algorithm = hdbscan.HDBSCAN
algorithm_kwargs = dict(min_cluster_size=10,allow_single_cluster=True)

[lesteve]

@essandess I can not reproduce on an Ubuntu box. For later reference can you edit your message and add your scikit-learn and joblib version.

Just to be on the safe side can you restart your kernel and only execute the cell that produces the error?

[bkellerman]

I can reproduce this error with this line in Jupyter on MacOS

cross_val_score(clf, X, y, cv=3, scoring='average_precision', n_jobs=-1, pre_dispatch=1)

Changing n_jobs to 1 fixes it.

sklearn 0.19.0 with joblib 0.11
jupyter 5.1.0
python 3.5.1
MacOS 10.13.4

Edit:

Restarting the Jupyter notebook fixes it.

[lesteve]

I can not reproduce on OSX with the snippet from https://github.com/joblib/joblib/issues/334#issuecomment-378878260.

Here is the output from:

import platform; print(platform.platform())
import sys; print("Python", sys.version)
import sklearn; print("Scikit-Learn", sklearn.__version__)
from sklearn.externals import joblib; print('Joblib', joblib.__version__)

Darwin-13.4.0-x86_64-i386-64bit
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
Scikit-Learn 0.19.1
Joblib 0.11

@bkellerman if you can put together a stand-alone reproducible snippet that would be very helpful!
Please read https://stackoverflow.com/help/mcve for more details.

[simonhughes22]

I am having this issue with joblib 0.11.0 and 0.12.0 on MacOS 10.13.4 (High Sierra) running jupyter lab.

[lesteve]

@simonhughes22 if you can put together a stand-alone reproducible snippet that would be very helpful!
Please read https://stackoverflow.com/help/mcve for more details.