Spyder: [Errno 32] 'Broken pipe' when trying to multiprocess

[TrentBrick]

I am running PyMC3 and it tries to sample my distribution with 4 cores. Every time this results in the above error. When I run this in Jupyter Notebooks instead there is no problem at all. Is there some setting for Spyder that I need to turn on to enable multiprocessing?

I have updated to the most recent versions of Spyder and PyMC3.

Thank you

[ccordoba12]

I guess you're on Windows, right?

@jnsebgosselin, what's the trick you used to make multiprocessing work there?

[TrentBrick]

Yes. Windows 10

[jnsebgosselin]

I @TrentBrick, could you share a minimal example that reproduce the issue please? That would be much appreciated.

[TrentBrick]

The code I run is below with all the error messages pasted below that. It loads in a datafile from https://xcelab.net/rm/statistical-rethinking/

Like I said runs with no problems in Jupyter Notebooks. It dies when I try to get the trace (last line of the with statement)

Code:

`# -- coding: utf-8 --
"""
Created on Tue Sep 4 20:58:42 2018

@author: fmsft
"""

import numpy as np
import matplotlib.pyplot as plt

'''samples = 10000

for i in range(100):

res = np.random.normal( np.random.normal(0, 10, samples), np.random.uniform(0,10,samples), samples )
plt.hist(res)
plt.show()'''

import os
import pandas as pd
df = pd.read_csv('rethinking/data/Howell1.csv', sep=';')
display(df.head())
display(df.info())
display(df.describe())

df2 = df[df.age >=18]

import pymc3 as pm

print('Running on PyMC3 v{}'.format(pm.__version__))

basic_model = pm.Model()

X1 = df2.weight
Y = df2.height

with basic_model:

# Priors for unknown model parameters
alpha = pm.Normal('alpha', mu=156, sd=100)
beta = pm.Normal('beta', mu=0, sd=10)
sigma = pm.Uniform('sigma', lower=0, upper=50)

# Expected value of outcome
mu = alpha + beta*X1

# Likelihood (sampling distribution) of observations
Y_obs = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
#step = pm.Slice()

map_estimate = pm.find_MAP(model=basic_model)
print(map_estimate)

start = pm.find_MAP()
step = pm.NUTS()
trace = pm.sample(100, step, start, random_seed=123, progressbar=True)

pm.traceplot(trace)

pm.summary(trace).round(2)

np.random.multivariate_normal(np.array(alpha, beta), covariance)`

`Traceback (most recent call last):

File "", line 1, in
runfile('C:/Users/fmsft/Rethinking Stats/HW4.py', wdir='C:/Users/fmsft/Rethinking Stats')

File "D:\Anaconda2\envs\py36\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 678, in runfile
execfile(filename, namespace)

File "D:\Anaconda2\envs\py36\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 106, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/fmsft/Rethinking Stats/HW4.py", line 56, in
trace = pm.sample(100, step, start, random_seed=123, progressbar=True)

File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\sampling.py", line 449, in sample
trace = _mp_sample(**sample_args)

File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\sampling.py", line 996, in _mp_sample
chain, progressbar)

File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 275, in __init__
for chain, seed, start in zip(range(chains), seeds, start_points)

File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 275, in
for chain, seed, start in zip(range(chains), seeds, start_points)

File "D:\Anaconda2\envs\py36\lib\site-packages\pymc3\parallel_sampling.py", line 182, in __init__
self._process.start()

File "D:\Anaconda2\envs\py36\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)

File "D:\Anaconda2\envs\py36\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)

File "D:\Anaconda2\envs\py36\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)

File "D:\Anaconda2\envs\py36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)

File "D:\Anaconda2\envs\py36\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe`

[TrentBrick]

The above was formatted weirdly but hope it is sufficient.

[jnsebgosselin]

Thank you very much, I'll try to take a look at it soon.

[reboss]

I enclosed the entire script in: if __name__ == '__main__': and that worked for me. Reference: https://discourse.pymc.io/t/multiprocessing-windows-10-brokenpipeerror-errno-32-broken-pipe/2259

[ccordoba12]

@reboss, thanks for letting us know about it! :+1:

[Christonikos]

Restarting the Kernel works for me.
@: Ubuntu 18.04 LTS.
Multithreading using the MNE-python parralel_func. (Note: Using all of the available threads).