Joblib: `delayed` doesn't work with cached functions

[samuela]

I'm trying to get up and running with a what I figure is a pretty standard joblib usecase. Unfortunately, I'm getting a pickling error:

import numpy as np
from joblib import Memory, Parallel, delayed

mem = Memory('/tmp/joblib/')

@mem.cache
def foo(x): return n * 2

Parallel(n_jobs=1)(delayed(foo)(i) for i in range(10))

gives the error

Traceback (most recent call last):
  File "joblib_test.py", line 9, in <module>
    Parallel(n_jobs=1)(delayed(foo)(i) for i in range(10))
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/joblib/parallel.py", line 793, in __call__
    while self.dispatch_one_batch(iterator):
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/joblib/parallel.py", line 646, in dispatch_one_batch
    tasks = BatchedCalls(itertools.islice(iterator, batch_size))
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/joblib/parallel.py", line 57, in __init__
    self.items = list(iterator_slice)
  File "joblib_test.py", line 9, in <genexpr>
    Parallel(n_jobs=1)(delayed(foo)(i) for i in range(10))
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/joblib/parallel.py", line 150, in delayed
    pickle.dumps(function)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

If I remove @mem.cache everything works just fine.

[lesteve]

One work-around is to have a different name for the cached function:

import numpy as np
from joblib import Memory, Parallel, delayed

mem = Memory('/tmp/joblib/')

def foo(x): return n * 2

cached_foo = mem.cache(foo)

Parallel(n_jobs=1)(delayed(cached_foo)(i) for i in range(10))

The underlying problem is due to a pickle limitation because both the raw and the decorated function have the same name. One possible way we could solve this problem is to use dill for serialization.

[samuela]

@lesteve Ah, I see. I'm glad it's possible at least. Is dill on the roadmap at all?

[lesteve]

Is dill on the roadmap at all?

It is definitely a possible evolution of joblib but it's kind of hard to make any promise on that.