Joblib: Inconsistent joblib.Parallel behaviour between v0.12 and v0.14 when running from within an imported module?

[Alessi0X]

I have a joblib.Parallel statement which is run from within an imported module. From v0.12 to v0.12.5 it works nicely, but with v0.13+ it goes serial with no errors.

The schema of my code (which spans several files) follows. Input arguments are omitted for the sake of readability.

# main file
import f1
for _ in range(n):
    f1()

# file containing f1
import f2
def f1():
    # do something with f2
    return something

# file containing f2
def f2():
    def help():
        return something
    joblib.Parallel(delayed(help))    # parallel with loki backend
    return something

I tried to find some information online regarding joblib.Parallel and its use on nested functions, with no luck.

However, if I change my workflow as follows

# main file
import f1
joblib.Parallel(delayed(f1))     # parallel with loki backend

# file containing f1
import f2
def f1():
    # do something with f2
    return something

# file containing f2
def f2():
    def help():
        return something
    for _ in range():
        # run help() in serial with a plain for loop
    return something

everything starts working also with v0.14.

What am I missing? And why with joblib v0.12-0.12.5 it works fine but with joblib v0.13+ it goes serial?

[pierreglaser]

Hi, thanks for the report!

That would be really great if you could craft a Minimal reproducible example that I could just run (python your_reproducer.py) to debug your situation.

Also, don't hesitate to use Parallel(verbose=100), to know which backend joblib uses (you should see the SequentialBackend/n_jobs=1 being mentioned somewhere according to the behavior you describe.

[Alessi0X]

Hi @pierreglaser
I have updated my code snippet with info on joblib's backends.

I will try to craft a minimal working example, but this ain't going to be easy. Unfortunately, things are quite nested here. I'll try to work it out anyway

[ogrisel]

A difference between the two snippets is that in the failing snippet, the help function that is passed to joblib.Parallel is locally defined (cannot be imported).

Loky should still work in this case, but this is likely part of the cause of the bug. @Alessi0X if you cannot produce a minimal reproducible example your own, please at least report the full traceback so that we can see if we can guess how to reproduce the errors you triggered.

[Alessi0X]

@ogrisel Unfortunately I cannot reproduce this strange behaviour. The interesting thing is that I do not have any errors or tracebacks. Everything works, but with v0.13+ joblib.Parallel goes serial whereas if I use v0.12-v0.12.5 it goes parallel.

[ogrisel]

Could you run the Parallel calls with verbose=100 and see which backend is used in each case?

[ogrisel]

@ogrisel Unfortunately I cannot reproduce this strange behaviour.

You mean you do not have the bug anymore? Did you upgrade joblib?

[Alessi0X]

@ogrisel I've never had a bug per se. I've just noticed this inconsistent behaviour of going parallel vs. going serial by using different versions of joblib on the very same code.

Parallel uses the loki backend, as confirmed by setting verbose=100:
[Parallel(n_jobs=28)]: Using backend LokyBackend with 28 concurrent workers.
where 28 is the number of (virtual) cores on my machine, so this makes sense of course. However, if I open the task manager, only 1 core is running.

This high level of verbosity shows a huge amount of
Pickling array (shape=(1, 2), dtype=float64).
printed on screen

[ogrisel]

Going serial when it should not is what I called being a bug.

Pickling small arrays should not be a problem.

[ogrisel]

We would really appreciate a minimal reproduction example for this bug. But no rush ;)

[Alessi0X]

@ogrisel

Going serial when it should not is what I called being a bug.

Sorry, since you were mentioning errors or tracebacks I thought you meant bugs as errors/crashes.

I will try my best to provide a minimal working example.