Joblib: Using Memory and Parallel with cached function defined inside Jupyter notebook results in not using the cache

[jontis]

I'm running a Jupyter notebook on a machine with multiple cores.
I'm trying to use both caching of intermediate results and parallelism for this.
I get a lot of:

python3.7/site-packages/joblib/parallel.py:256: JobLibCollisionWarning: Cannot detect name collisions for function 'cached_two' for func, args, kwargs in self.items]

I believe these are not appearing when I just run single threade or not in a notebook.
Sample code below to run in jupyter notebook.
joblib 0.14.1
notebook 6.0.3

[jontis]

from joblib import Parallel, delayed, Memory

memory = Memory('/tmp/joblib_cache', verbose=1)

def cached_one():
    return []


cached_one_wrap = memory.cache(cached_one)


def cached_two(*args):
    return []


cached_two_wrap = memory.cache(cached_two)


list_loop = Parallel(n_jobs=2)(delayed(cached_two_wrap)(i, cached_one_wrap()) for i in range(5)) # leave two phys cores free

[ogrisel]

Thanks for the reproducer. I get:

WARNING:root:[MemorizedFunc(func=<function cached_two at 0x7fad04d878b0>, location=/tmp/joblib_cache/joblib)]: Clearing function cache identified by __main__--home-ogrisel-code-scikit-learn-__ipython-input__/cached_two

and then the JobLibCollisionWarning from times to time when n_jobs > 1

When n_jobs == 1 I don't get any warning.

Not sure what's happening but it's probably a bug.

[ogrisel]

Note: the warnings do not show up if the code of snippet is written to a file on the disk and run with python scritptname.py or with %run scriptname.py in a jupyter / ipython session.

But I can reproduce the warnings when the code snippets is defined in jupyter or ipython cell.

[lesteve]

As mentioned in #1069 this is a bit of an annoying bug since the cache ends up not being used. I edited the issue title to reflect this.

A very likely work-around is to move the cached function outside of the notebook to a external .py module.

I think this used to work at one point but maybe I am wrong and it never properly worked inside a Jupyter notebook (or IPython) ...

[lesteve]

A snippet to reproduce:

from joblib import Parallel, delayed, Memory

memory = Memory('/tmp/joblib')

def func(x):
    time.sleep(1)
    return x

cached_func = memory.cache(func)

t0 = time.time()
cached_func(0)
dt = time.time() - t0
print(f'First call took {dt:.2f} seconds')

t0 = time.time()
cached_func(0)
dt = time.time() - t0
print(f'Second call took {dt:.2f} seconds')

t0 = time.time()
Parallel(n_jobs=2)(delayed(cached_func)(0) for _ in range(10))
dt = time.time() - t0
print(f'Parallel call took {dt:.2f} seconds')

Output inside a IPython session (keeping only the timings and removing the warnings + Memory and Parallel output):

First call took 1.01 seconds
Second call took 0.00 seconds
Parallel call took 5.33 seconds

The fact that the Parallel call takes 5 seconds shows that the cache is not used.

Output running a python file:

First call took 1.01 seconds
Second call took 0.00 seconds
Parallel call took 0.05 seconds

It seems like the extraction of the function code is to blame:

❯ cat /tmp/joblib/joblib/__main__--home-$USER-dev-joblib-__ipython-input__/func/func_code.py
# first line: -1
6189136146609509772

It is weird because joblib.func_inspect.get_func_code(func) gives something reasonable ...

In [4]: get_func_code(func)                                                                                                                                    
Out[4]: 
('def func(x):\n    time.sleep(1)\n    return x\n',
 '<ipython-input-2-fd9d03aaef8c>',
 1)

but maybe because the file <ipython-...> does not exists (as a file) so some special logic kicks in ...

If you look closely at the warnings the func addresses is different which is probably why you alternate between two differents values for func_code.py and each cached functions stepping over each other toes and recomputing from scratch rewriting the cache, recomputing from scratch rewriting the cache etc ...

WARNING:root:[MemorizedFunc(func=<function func at 0x7fc2a6f50200>, location=/tmp/joblib)]: Clearing function cache identified by __main__--home-lesteve-dev-joblib-__ipython-input__/func
WARNING:root:[MemorizedFunc(func=<function func at 0x7f7ba4d08200>, location=/tmp/joblib)]: Clearing function cache identified by __main__--home-lesteve-dev-joblib-__ipython-input__/func

[dsevero]

I was having the same problem in #1069. Moving the functions to a python file as @lesteve suggested worked. Just wanted to report it.

[pierreglaser]

Thank you very much for the report @dsevero and @jontis. I'm going to work on a fix for this.

[dsevero]

Great. I'm still using this "hack" so if you need any extra testing I'd be glad to help @pierreglaser . I'll keep my eye on this thread