This issue continues even after re-installing Python 3 from Anaconda. I can no longer sample with multiprocessing.

fonnesbeck on 24 Jul 2016

@fonnesbeck is this still a problem?

twiecki on 28 Sep 2016

Havent seen it in awhile.

fonnesbeck on 29 Sep 2016

This has started occurring again. I currently cannot run any models using njobs>1. The process hangs, with almost no CPU use by Python:

with pritts_update:
    trace = pm.sample(2000, random_seed=rseeds[0], njobs=2)#, random_seed=[20140425, 19700903])

Assigned NUTS to ν_log_
Assigned NUTS to μ_age
Assigned NUTS to age_max_missing
Assigned NUTS to μ
Assigned NUTS to σ_interval_
Assigned NUTS to θ
Assigned NUTS to β
Assigned NUTS to α
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:

Reopening.

fonnesbeck on 19 Oct 2016

Even a fresh virtualenv does not fix the issue.

fonnesbeck on 20 Oct 2016

I've completely removed and reinstalled Anaconda, but this persists. I am completely unable to run parallel chains. Am I really the only person that gets this behavior??

fonnesbeck on 29 Oct 2016

Can you post an example to run? We added some sampling with njobs>1 to the
test suite for the random seed issue last week, but maybe those are not
sampling enough to replicate?

On Sat, Oct 29, 2016, 3:48 PM Chris Fonnesbeck [email protected]
wrote:

I've completely removed and reinstalled Anaconda, but this persists. I am
completely unable to run parallel chains. Am I really the only person that
gets this behavior??

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1247#issuecomment-257112043,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACMHEAMAxXds4jFotxj1EpaEUuMI1uRfks5q46LvgaJpZM4JPQk_
.

ColCarroll on 29 Oct 2016

The first model in this notebook is one example. Feel free to clone the whole project if you want to try it. I can do multiprocessing for the test suite and simple examples, but not for more complex model. I will check to see if I can trigger the issue on one of our notebooks.

fonnesbeck on 31 Oct 2016

The issue was never really resolved was it? For example I also still get segmentation errors on my models with the current implementation- it doesnt even start.

hvasbath on 1 Nov 2016

@hvasbath Only with multiprocessing or generally?

twiecki on 1 Nov 2016

Only with joblib ergo multiprocessing yes ... @twiecki

This is where we last stopped discussing, it was in connection with the recursion errors ...
https://github.com/pymc-devs/pymc3/issues/879

hvasbath on 1 Nov 2016

This works fine for me, with njobs \in {1,2,3,4}.

with pm.Model() as disaster_model:
    switchpoint = pm.DiscreteUniform('switchpoint', lower=0, upper=110)
    early_mean = pm.Exponential('early_mean', lam=1)
    late_mean = pm.Exponential('late_mean', lam=1)
    rate = pm.Deterministic('rate', tt.switch(switchpoint >= np.arange(112), early_mean, late_mean))
    trace = pm.sample(100000, step=pm.Metropolis(), njobs=4)

ColCarroll on 1 Nov 2016

@ColCarroll yes, I can get simple models to work, but I haven't been able to pinpoint what causes the failure, other than "more complex models". When they fail, they always fail for me, both on macOS and Linux.

fonnesbeck on 2 Nov 2016

Yep, I also encounter this on occasion - unfortunately I've had little luck replicating it in a test Notebook (and simply not time to follow up).

I've just seen this reported issue with joblib - possibly related?

jonsedar on 2 Dec 2016

I wonder if the recent thread fix maybe helped with this. @jonsedar are you on master?

twiecki on 2 Dec 2016

Afraid I haven't had time to create a new env and test it - but hopefully will get some time over Christmas.

jonsedar on 2 Dec 2016

This is what I was referring to: https://github.com/pymc-devs/pymc3/pull/1555 Might be totally unrelated but who knows.

twiecki on 2 Dec 2016

👍1

1555 does not fix this behavior for me, sadly.

fonnesbeck on 6 Dec 2016

This still occurs in the current master, on new hardware (2016 MB Pro). Can't multiprocess at all for many models.

Maybe time to explore ipyparallel or concurrent.futures for multiprocessing.

fonnesbeck on 29 Jan 2017

You can also try to use the parimap function in the current ATMCMC2 PR. I run that all the time and it works well also on very complex models.
I could help with implementing it, although I am not sure I will have much time for that. But you can see how to use it in the PR.

hvasbath on 30 Jan 2017

EDIT: I bodged a fix: uninstalled joblib 0.10.3 and pip installed 0.9.4 instead.

Now, njobs >1 works! Though I'm still none the wiser. Left the below text for posterity:

I _had_ a working env, and then clean-installed Sierra. Should have known better. No jobs >1 for me....

FWIW it seems to be something in joblib parallel, I don't know enough about the underlying to go digging. This is a clean install, clean env etc, installed from pip.

CPython 3.6.0
IPython 5.1.0

numpy 1.11.3
scipy 0.18.1
pymc3 3.0
theano 0.8.2
joblib 0.10.3

compiler   : GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)
system     : Darwin
release    : 16.4.0
machine    : x86_64
processor  : i386
CPU cores  : 4
interpreter: 64bit
Git hash   : 3bc212d20dae57234cedc59f3d89f77dad0521ed

RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
NotImplementedError: expected an aligned array of type 12 (NPY_FLOAT64), got non-aligned array of type 12 with 2 dimensions, with 3 last dims -1, 49832, 13 and 3 last strides -1 104, 8.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 344, in __call__
    return self.func(*args, **kwargs)
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/joblib/parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/pymc3/sampling.py", line 185, in _sample
    for strace in sampling:
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/pymc3/sampling.py", line 267, in _iter_sample
    point = step.step(point)
  File "/Users/jon/anaconda/envs/appliedai_pricing/lib/python3.6/site-packages/pymc3/step_methods/arraystep.py", line 142, in step
    apoint = self.astep(bij.map(point))

.... etc etc

jonsedar on 6 Feb 2017

Make sure to use Theano dev version or Theano 0.9b1. There was fix for that since the last release.

nouiz on 13 Feb 2017

I am running into this problem too. I'm working with a dataset where I can fit simple models (multiple regression, no random effects) using njobs>1, but as soon as I add any random effects, only njobs=1 will work. Beyond njobs=1 I get the behavior that @fonnesbeck described.

When I first ran into this I was running some older (but not that old) version of pymc3 that I didn't write down, theano==0.8.2, and joblib==0.10.2. I upgraded to the dev version of pymc3, and then following suggestions in this thread upgraded to theano==0.9.0rc1 and rolled back to joblib==0.9.4. But none of that solved the issue.

jake-westfall on 22 Feb 2017

I have not seen this since the 0.9 beta, but then it always appeared in unpredictable situations.

fonnesbeck on 22 Feb 2017

Okay, well, this is now unexpectedly working for me. All I did was (totally unrelatedly) install pystan, which in the process also updated:

numpy:  1.10.2-py35_p0 [mkl] --> 1.12.0-py35_0  
python: 3.5.2-0        --> 3.5.3-0

And I guess one of these two upgrades must have done the trick?

jake-westfall on 27 Feb 2017

Could be a change in Python 3.5.3. Let's hope so.

fonnesbeck on 27 Feb 2017

I'm also running into njobs >1 hanging as well. Noticing the same behavior of spawned python processes sitting still as shown in first post on this issue. I'm on Python 3.5.3. Am using metropolis step on a "complicated" model, and have turned to merge_traces as a workaround with chains run in sequence.

d-diaz on 15 Mar 2017

From memory, some BLAS implementation could cause that. Not sure if it was
fixed or not, but changing your blas implementation could help.

On Wed, Mar 15, 2017 at 5:03 PM David Diaz notifications@github.com wrote:

I'm also running into njobs >1 hanging as well. Noticing the same behavior
of spawned python processes sitting still as shown in first post on this
issue. I'm on Python 3.5.3. Am using metropolis step on a "complicated"
model, and have turned to merge_traces as a workaround with chains run in
sequence.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1247#issuecomment-286878415,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALC-3Ans8Mo1ObvdoHAYgv0esxDhG7kks5rmFIigaJpZM4JPQk_
.

nouiz on 15 Mar 2017

It may or may not help anyone, but I just created a new env with the latest master and latest joblib==0.11.0 and Theano==0.9.0, and re-encountered my aligned array issue from earlier when njobs>1

NotImplementedError: expected an aligned array of type 12 (NPY_FLOAT64), got non-aligned array of type 12 with 2 dimensions, with 3 last dims -1, 49832, 13 and 3 last strides -1 104, 8.

So, I uninstalled joblib==0.11.0 and installed the older joblib==0.9.4 and that fixed the issue, I can now set njobs>1. The core packages in my env (macOS Sierra 10.12.4) are now:

numpy==1.12.1
scipy==0.19.0
mkl=2017.0.1
mkl-service=1.1.2
matplotlib==2.0.0
Theano==0.9.0
pandas==0.19.2
patsy==0.4.1
joblib==0.9.4

jonsedar on 2 May 2017

About Alignement. Theano only handle aligned ndarray. So if there was a
record where element where float32,bool, then you make a view of just the
float32 element, you will get a non aligned float32 ndarray. This is
probably the caues of this. Probably something in the newer version is more
optimized and do less copy and generate such non aligned arrays. You can
just copy it when passing it to Theano and that will align the data.

There is probably an numpy fct that will copy the data only if not aligned
and return the input if aligned.

I'm not sure Theano should do that copy automatically in that case. What do
you think?

On Tue, May 2, 2017 at 5:59 AM Jonathan Sedar notifications@github.com
wrote:

It may or may not help anyone, but I just created a new env with the latest
master
https://github.com/pymc-devs/pymc3/commit/7d3fccb90b992daa460cbb5c1260357e2abf153f
and latest joblib==0.11.0 and Theano==0.9.0, and re-encountered my aligned
array issue from earlier when njobs>1

NotImplementedError: expected an aligned array of type 12 (NPY_FLOAT64), got non-aligned array of type 12 with 2 dimensions, with 3 last dims -1, 49832, 13 and 3 last strides -1 104, 8.

So, I uninstalled joblib==0.11.0 and installed the older joblib==0.9.4 and
that fixed the issue, I can now set njobs>1. The core packages in my env
(macOS Sierra 10.12.4) are now:

numpy==1.12.1
scipy==0.19.0
matplotlib==2.0.0
Theano==0.9.0
pandas==0.19.2
patsy==0.4.1
joblib==0.9.4

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1247#issuecomment-298586331,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALC-1PUpePxOjJH2Ioeg73GYDbhV262ks5r1v6BgaJpZM4JPQk_
.

nouiz on 2 May 2017

Okay, that's interesting @nouiz .... AFAIK the hand-off to joblib is inside pymc3.sampling._mp_sample() line 439

traces = Parallel(n_jobs=njobs)(delayed(_sample)(chain=chains[i],
                                                     progressbar=pbars[i],
                                                     random_seed=rseed[i],
                                                     start=start_vals[i],
                                                     **kwargs) for i in range(njobs))

@fonnesbeck @twiecki are you aware of any better ways to hand-off to Parallel()? Some way to data-align the arrays?

jonsedar on 2 May 2017

Something like require?

np.require(values, dtype=np.float32, requirements=['ALIGNED'])

fonnesbeck on 2 May 2017

Yeah, was pondering that... might it work to bodge that into _sample?

jonsedar on 2 May 2017

@jonsedar worth a try!

twiecki on 3 May 2017

Last night I did a fresh install of Anaconda, and I ran this on a Jupyter notebook:

import numpy as np
from pymc3 import Model, Exponential, DiscreteUniform, Poisson, sample
from pymc3.math import switch
import multiprocessing

disaster_data = np.ma.masked_values([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                            3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                            2, 2, 3, 4, 2, 1, 3, -999, 2, 1, 1, 1, 1, 3, 0, 0,
                            1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                            0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                            3, 3, 1, -999, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                            0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1], value=-999)
year = np.arange(1851, 1962)

with Model() as disaster_model:

    switchpoint = DiscreteUniform('switchpoint', lower=year.min(), upper=year.max(), testval=1900)

    # Priors for pre- and post-switch rates number of disasters
    early_rate = Exponential('early_rate', 1)
    late_rate = Exponential('late_rate', 1)

    # Allocate appropriate Poisson rates to years before and after current
    rate = switch(switchpoint >= year, early_rate, late_rate)

    disasters = Poisson('disasters', rate, observed=disaster_data)

    trace = sample(10000, njobs=multiprocessing.cpu_count())

I get these warnings:

/Users/jonathanng/anaconda/lib/python3.6/site-packages/theano/tensor/var.py:507: DeprecationWarning: elementwise == comparison failed; this will raise an error in the future.
if(inp == [] or inp == ()) else inp for inp in args])

/Users/jonathanng/anaconda/lib/python3.6/site-packages/joblib/parallel.py:547: UserWarning: Multiprocessing-backed parallel loops cannot be nested below threads, setting n_jobs=1
**self._backend_args)

It's a simple model, and I don't get a hanging process. It just runs the different chains in a serial fashion.

Is this the same issue that everyone is experiencing?

jonathanng on 28 Jun 2017

Actually I always have the same issue sampling using Metropolis. Sampler hangs after assigning step at

Process ForkPoolWorker-1:
Process ForkPoolWorker-2:

junpenglao on 28 Jun 2017

I've been experiencing an issue running the sample_ppc example here whereby the notebook starts spewing out a never-ending stack trace when I run with njobs > 1. I'm using pymc3==3.0, python27, on a Mac. I think it might be related to using masked values, since I added some to the data array to see what happens (as in the mining example). Without the masked values the example works fine. With masked values I get:

Auto-assigning NUTS sampler...
Initializing NUTS using advi...
Average ELBO = -138.87: 100%|██████████| 200000/200000 [00:17<00:00, 11293.59it/s]
Finished [100%]: Average ELBO = -138.87
Process PoolWorker-3:
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
Traceback (most recent call last):
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(self._args, *self._kwargs)
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/joblib/pool.py", line 362, in get
return recv()
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
File "/Users/wf/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__

(snip)

File "/Users/williamferreira/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
File "/Users/williamferreira/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
File "/Users/williamferreira/anaconda/envs/pymc3/lib/python2.7/site-packages/pymc3/distributions/distribution.py", line 98, in __getattr__
return getattr(self.parent_dist, name)
RuntimeError: maximum recursion depth exceeded

Here is the example code:

import numpy as np
import pymc3 as pm
data = np.random.randn(100)
data_with_missing = np.ma.masked_values(np.hstack((data, np.array([-999, -999]))), value=-999)

with pm.Model() as model:
    mu = pm.Normal('mu', mu=0, sd=1, testval=0)
    sd = pm.HalfNormal('sd', sd=1)
    n = pm.Normal('n', mu=mu, sd=sd, observed=data_with_missing)

    trace = pm.sample(5000, njobs=2)

Swapping data_with_missing for data, or removing the njobs parameter, fixes the issue.

willferreira on 8 Sep 2017

Same with the coal mining example here. The following code works fine with masked values removed in data (I just substituted some made up values):

import numpy as np
from pymc3 import Model, Exponential, DiscreteUniform, Poisson, sample
from pymc3.math import switch
import multiprocessing

data = np.array([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                            3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                            2, 2, 3, 4, 2, 1, 3, 2, 2, 1, 1, 1, 1, 3, 0, 0,
                            1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                            0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                            3, 3, 1, 1, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                            0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])

disaster_data = np.ma.masked_values([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                            3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                            2, 2, 3, 4, 2, 1, 3, -999, 2, 1, 1, 1, 1, 3, 0, 0,
                            1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                            0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                            3, 3, 1, -999, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                            0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1], value=-999)
year = np.arange(1851, 1962)

with Model() as disaster_model:
    switchpoint = DiscreteUniform('switchpoint', lower=year.min(), upper=year.max(), testval=1900)

    # Priors for pre- and post-switch rates number of disasters
    early_rate = Exponential('early_rate', 1)
    late_rate = Exponential('late_rate', 1)

    # Allocate appropriate Poisson rates to years before and after current
    rate = switch(switchpoint >= year, early_rate, late_rate)

    disasters = Poisson('disasters', rate, observed=data)

    trace = sample(10000, njobs=2)

But setting observed=disaster_data without setting njobs=1 (or missing it out) causes the same problem as in my previous post.

willferreira on 8 Sep 2017

@willferreira I think I fixed this in #2551. That you for reporting this.
In the future, you can just open a new issue for things like this, this isn't really related to the original issue.

aseyboldt on 9 Sep 2017

I have not seen this in awhile. Closing.

fonnesbeck on 3 Oct 2017

I'm using Anaconda 5.0.1 on Windows and pymc3 3.2 from Anaconda's default repo. I'm getting this problem with both pymc3 and pystan. So I'm thinking it's a problem on joblib rather than pymc3, but thought I share this problem came back. Yes I did wrap my code with if __name__ == "__main__". I'm just running the sample Tank Problem code from a blog post. If I keep njobs=1 there is no problem. It's when njobs > 1 that's when the problem starts. I have no issue with running with njobs > 1 with python.exe. The trouble is coming from IPython, specifically runfile() in IPython. Again just sharing. No action is requested. I'm opening an Anaconda issue: Spyder + IPython runfile(). Anaconda has confirm they can reproduce this issue. https://github.com/ContinuumIO/anaconda-issues/issues/7633
tankProblem.zip

wjcgh on 5 Dec 2017

👍1

Anaconda has confirm they can reproduce this issue.

Where did we do that? I'm not disputing that someone from Anaconda, Inc verified this but I do need a solid reference to this to be able to follow it up.

mingwandroid on 5 Dec 2017

For me also njobs>1 not working. When I run the code that @johnmcdonnell tried above, I see following message and no more results:
`
runfile('C:/xxxxxxxxxxxxxxxxx.py', wdir='C:/xxxxxxxxxxxxxxxxx')
Multiprocess sampling (8 chains in 8 jobs)
CompoundStep

CompoundStep

Metropolis: [disasters_missing]
Metropolis: [switchpoint]
NUTS: [late_rate_log__, early_rate_log__]`

After this nothing happens in spyder. The run never finish.

madarshahian on 4 May 2018

Still having this issue;

Exception: ('The following error happened while compiling the node', InplaceDimShuffle{}(inarray), 'n', 'Compilation failed (return status=3): ', '[InplaceDimShuffle{}(inarray)]')

forrtl: error (200): program aborting due to control-C event

Image PC Routine Line Source
libifcoremd.dll 00007FFF11EE94C4 Unknown Unknown Unknown
KERNELBASE.dll 00007FFF488D7C23 Unknown Unknown Unknown
KERNEL32.DLL 00007FFF4B7A8364 Unknown Unknown Unknown
ntdll.dll 00007FFF4C057091 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFF11EE94C4 Unknown Unknown Unknown
KERNELBASE.dll 00007FFF488D7C23 Unknown Unknown Unknown
KERNEL32.DLL 00007FFF4B7A8364 Unknown Unknown Unknown
ntdll.dll 00007FFF4C057091 Unknown Unknown Unknown
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFF11EE94C4 Unknown Unknown Unknown
KERNELBASE.dll 00007FFF488D7C23 Unknown Unknown Unknown
KERNEL32.DLL 00007FFF4B7A8364 Unknown Unknown Unknown
ntdll.dll 00007FFF4C057091 Unknown Unknown Unknown

After rolling back joblib from 0.12.1 to 0.9.4, njobs > 1 works.

VoroninDA on 9 Aug 2018

👍1

I still have this problem on a 2013 MBP even after 1) uninstalling theano and pymc3, and then pip3 install'ing theano and pymc3 from master, and 2) downgrading joblib to 0.9.4.

OS: El Capitan 10.11.6
Python 3.6.5
PyMC3==3.6

michael-ziedalski on 22 Jan 2019

@michael-ziedalski thanks for pinging in.

Until we have PyMC4 out, or until one of my fellow devs who is knowledgeable about the internals of PyMC3 finds a spare cycle to handle this (keeping in mind, nobody on the team develops PyMC3 full-time), from practical experience, I'd advise using njobs=1 (i.e. avoid multiprocessing).

ericmjl on 22 Jan 2019

@michael-ziedalski, what blas library is installed/being used by theano? It seems that there could be deadlocks with multiprocessing forks that come up due to blas and openmp.

lucianopaz on 22 Jan 2019

numpy.__config__.show() indicates that OpenBLAS and OpenBLAS Lapack are used by it. Though will update the versions and report if that changes anything.

michael-ziedalski on 22 Jan 2019

The stack trace I saw could be related to Theano and not numpy.
So from the link, I would suggest to try python 3.4 and change the function
subprocess_Popen in the file ./theano/misc/windows.py to use spawn or
forkserver start.

Just in case Popen have the same problem as multiprocess.
(Note I do not have time to do the code). Just in case someone here want to
try to fix it.
Also if MKL is available, it could be used instead of OpenBLAS.
Conda provide it freely.

If someone have a working work around, tell it (except njobs=1). This
affect many people.

On Tue, Jan 22, 2019 at 1:29 AM michael-ziedalski notifications@github.com
wrote:

numpy.__config__.show() indicates that OpenBLAS and OpenBLAS Lapack are
used by it. Though will update the versions and report if that changes
anything.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1247#issuecomment-456285779,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALC-yEBf04DeDiN6bkbp2adUG3dbosvks5vFq-8gaJpZM4JPQk_
.

nouiz on 22 Jan 2019

Pymc3: Multiprocessing failure

All 49 comments

1555 does not fix this behavior for me, sadly.

Related issues