Joblib: Loky BrokenProcessPool

[twiecki]

As of recent PyMC3 is seeing travis failures, I think this coincides with joblib 0.13. The errors look like this:

pymc3/sampling.py:440: in sample
    trace = _mp_sample(**sample_args)
pymc3/sampling.py:1033: in _mp_sample
    traces = Parallel(n_jobs=cores, mmap_mode=None)(jobs)
../../../miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/parallel.py:930: in __call__
    self.retrieve()
../../../miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/parallel.py:833: in retrieve
    self._output.extend(job.get(timeout=self.timeout))
../../../miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/_parallel_backends.py:521: in wrap_future_result
    return future.result(timeout=timeout)
../../../miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/externals/loky/_base.py:433: in result
    return self.__get_result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Future at 0x7f24374622d0 state=finished raised BrokenProcessPool>
    def __get_result(self):
        if self._exception:
>           raise self._exception
E           BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
E           
E           This was caused directly by 
E           '''
E           Traceback (most recent call last):
E             File "/home/travis/miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/externals/loky/process_executor.py", line 391, in _process_worker
E               call_item = call_queue.get(block=True, timeout=timeout)
E             File "/home/travis/miniconda2/envs/testenv/lib/python2.7/multiprocessing/queues.py", line 135, in get
E               res = self._recv()
E             File "/home/travis/build/pymc-devs/pymc3/pymc3/step_methods/arraystep.py", line 39, in __new__
E               model = modelcontext(kwargs.get('model'))
E             File "/home/travis/build/pymc-devs/pymc3/pymc3/model.py", line 191, in modelcontext
E               return Model.get_context()
E             File "/home/travis/build/pymc-devs/pymc3/pymc3/model.py", line 183, in get_context
E               raise TypeError("No context on context stack")
E           TypeError: No context on context stack
E           '''
../../../miniconda2/envs/testenv/lib/python2.7/site-packages/joblib/externals/loky/_base.py:381: BrokenProcessPool

https://travis-ci.org/pymc-devs/pymc3/jobs/455064428

Any ideas? Was the pickling behavior changed somehow?

[twiecki]

I confirmed that tests pass with joblib<0.13: https://github.com/pymc-devs/pymc3/pull/3254

[ogrisel]

Hi @twiecki . Is this a Python 2.7 only failure? You confirm that the same tests pass under Python 3?

[twiecki]

hi @ogrisel. unfortunately I can't confirm that because our tests use joblib for python 2 only (python 3 uses futures).

[ogrisel]

Alright, thanks for the feedback.

[pierreglaser]

Hi @twiecki,
I have been looking at the error in the build, and here is what I think is going on.

Recent versions of joblib currenlty use loky as a backend by default, where python objects used in the child processes must be serialized to be sent to the workers.

The problem with your current test is that the sgfs instance being serialized relies at creation time on a class attribute (the Context.contexts) that gets mutated during the execution of parent process (when Model.__enter__ is called). Unfortunatly, the mutation will not happen in the child processes, hence the error.

A quick fix option is to use "multiprocessing" instead of "loky" as a backend for joblib (that does not require any input serialization): Parallel(backend='multiprocessing'). Doing this for me made the test pass on my local laptop.

For the record, tests are passing on python3 because concurrent.futures also uses multiprocessing as their default backend.

One tricky case would be to run this test under windows, because the only available method to create new processes is spawn, so object must be serialized at some point. However, I did not find any CI scripts running under windows on pymc3. Could you confirm that?

[twiecki]

Thanks for digging into this. I will try the multiprocessing trick if I get around to it. It might not be important for pymc3 as we are sunsetting python 2 soon in which case the joblib dependency drops.

[ogrisel]

Note that the fork start method of multiprocessing used by default in concurrent.futures is still problematic (crash or freeze) when using libraries that use OpenMP thread pools (e.g. lightgbm, xgboost, and soon scikit-learn).

So I would advise you to use the spawn or forkserver start method of concurrent.futures in which case the mutated attributed will not be visible in the children process.

When doing parallel calls with concurrent futures you, you should probably pass the model context explicitly/