Joblib: No simple way to pass initializer for process

[cjw296]

multiprocessing.Pool has a handy initializer parameter to pass a callable for setting up per-process resources (database connections, loggers, etc) but joblib doesn't expose a way to pass this.

I see in 0.10 I can pass a custom multiprocessing context, which I hope I can use to achieve this, but per-process setup is likely something that many users will likely need, so would be good if there was an easier way.

(I'd suggest an initializer parameter to Parallel that's picked up by MultiprocessingBackend)

[cjw296]

...and sadly, context is only available on Python 3.4+, the project I'm working on is Python 2.7 :-(

[cjw296]

Hmm, actually, not even sure passing a context will let you do per-process initialisation...

[astromancer]

Custom initializers for processes is a must have feature!

[gdlmx]

For who is searching for a dirty workaround, I wrote a simple function to inject additional initializer to Parallel instance with a Loky backend.

def with_initializer(self, f_init):
    hasattr(self._backend, '_workers') or self.__enter__()
    origin_init = self._backend._workers._initializer
    def new_init():
        origin_init()
        f_init()
    self._backend._workers._initializer = new_init if callable(origin_init) else f_init
    return self

Example usage:

import matlab
from joblib import Parallel, delayed

x = matlab.double([[0.0]]) # this object can only be loaded after importing matlab

def f(i):
    print(i, x)

def _init_matlab():
    import matlab

with Parallel(4) as para:
    for _ in with_initializer(para, _init_matlab)(delayed(f)(i) for i in range(10)):
        pass

Data objects of some complex libraries such as matlab can only be loaded after importing the python module. An initializer seems to be the only way to guarantee to load a third party module before the child processes try to unpickle those global data objects.

[cwindolf]

(See https://stackoverflow.com/questions/55424095/error-pickling-a-matlab-object-in-joblib-parallel-context for context on the above)