Joblib: Error:Could not pickle the task to send it to the workers

[EPinzuti]

Hi,

I get the error 'Could not pickle the task to send it to the workers' when I call:
reconstructed=Parallel(n_jobs=6,verbose=50)(delayed(self.spectral_surrogates)(data_slice) for perm in range(0, self.settings['n_perm_spec']))

spectral_surrogates has inside a wrapper to work with a C function. Is it possible to use joblib in this case?

[ogrisel]

The function and all the arguments need to be picklable to use joblib. If you want more help please provide a minimal reproduction case:

http://mrocklin.github.io/blog/work/2018/02/28/minimal-bug-reports

[ogrisel]

And include the full python traceback of the error.

[EPinzuti]

I tried a very simple example and I get the same error

from joblib import Parallel, delayed
def multiple(a, b):
        return a*b
Parallel(n_jobs=2)(delayed(multiple)(a=i, b=j) for i in range(1, 6) for j in range(11, 16)) 

Traceback (most recent call last):
  File "<input>", line 8, in <module>
  File "/data/home1/epinzuti/PycharmProjects/tesz/venv/lib/python3.4/site-packages/joblib/parallel.py", line 962, in __call__
    self.retrieve()
  File "/data/home1/epinzuti/PycharmProjects/tesz/venv/lib/python3.4/site-packages/joblib/parallel.py", line 865, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/data/home1/epinzuti/PycharmProjects/tesz/venv/lib/python3.4/site-packages/joblib/_parallel_backends.py", line 515, in wrap_future_result
    return future.result(timeout=timeout)
  File "/data/home1/epinzuti/PycharmProjects/tesz/venv/lib/python3.4/site-packages/joblib/externals/loky/_base.py", line 431, in result
    return self.__get_result()
  File "/data/home1/epinzuti/PycharmProjects/tesz/venv/lib/python3.4/site-packages/joblib/externals/loky/_base.py", line 382, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.

[ogrisel]

I cannot reproduce this error (using joblib 0.11, 0.12.0 and 0.12.2 in a conda environment created with python 3.4).

[EPinzuti]

I checked again and it 's actually working outside the interactive interpreter.
Thanks

[ogrisel]

I ran the previous test in an interactive IPython session. Closing. Feel free to open a new issue if you have another issue with a minimal reproduction case.

[floria618]

@ogrisel

I encountered the same problem when I ran

Node2Vec(G, dimensions = 64, q = 0.025, p = 4, workers = 4, num_walks = 1, walk_length = 5, weight_key = 'weight')**

Traceback (most recent call last):
File "FluPrediction.py", line 255, in
node2vec_weightedGraph()
File "FluPrediction.py", line 14, in node2vec_weightedGraph
model = Node2Vec(G, dimensions = 64, q = 0.025, p = 4, workers = 4, num_walks = 2, walk_length = 10, weight_key = 'weight').fit(window = 2, min_count = 1, batch_words = 4)
File "D:Python3.6libsite-packagesnode2vecnode2vec.py", line 67, in __init__
self.walks = self._generate_walks()
File "D:Python3.6libsite-packagesnode2vecnode2vec.py", line 154, in _generate_walks
in enumerate(num_walks_lists, 1))
File "D:Python3.6libsite-packagesjoblibparallel.py", line 934, in __call__
self.retrieve()
File "D:Python3.6libsite-packagesjoblibparallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "D:Python3.6libsite-packagesjoblib_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "D:Python3.6libconcurrentfutures_base.py", line 432, in result
return self.__get_result()
File "D:Python3.6libconcurrentfutures_base.py", line 384, in __get_result
raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.

[eljalalpour]

@ogrisel

I encountered the same problem when I ran

Node2Vec(G, dimensions = 64, q = 0.025, p = 4, workers = 4, num_walks = 1, walk_length = 5, weight_key = 'weight')**

Traceback (most recent call last):
File "FluPrediction.py", line 255, in
node2vec_weightedGraph()
File "FluPrediction.py", line 14, in node2vec_weightedGraph
model = Node2Vec(G, dimensions = 64, q = 0.025, p = 4, workers = 4, num_walks = 2, walk_length = 10, weight_key = 'weight').fit(window = 2, min_count = 1, batch_words = 4)
File "D:Python3.6libsite-packagesnode2vecnode2vec.py", line 67, in init
self.walks = self._generate_walks()
File "D:Python3.6libsite-packagesnode2vecnode2vec.py", line 154, in _generate_walks
in enumerate(num_walks_lists, 1))
File "D:Python3.6libsite-packagesjoblibparallel.py", line 934, in call
self.retrieve()
File "D:Python3.6libsite-packagesjoblibparallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "D:Python3.6libsite-packagesjoblib_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "D:Python3.6libconcurrentfutures_base.py", line 432, in result
return self.__get_result()
File "D:Python3.6libconcurrentfutures_base.py", line 384, in __get_result
raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.

I got the same error, any update on that?