Pymc3: DEMetropolis fails to start

[madanh]

Upon attempting to run the example in pymc3/pymc3/examples/samplers_mvnormal.py
I get this

Traceback (most recent call last):
  File "/mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/demetropolisexample.py", line 81, in <module>
    trace, rate, runtime = run(methods, p)
  File "/mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/demetropolisexample.py", line 49, in run
    start=start
  File "/home/madanh/.virtualenvs/pymcdev/lib/python3.4/site-packages/pymc3-3.2-py3.4.egg/pymc3/sampling.py", line 426, in sample
  File "/home/madanh/.virtualenvs/pymcdev/lib/python3.4/site-packages/pymc3-3.2-py3.4.egg/pymc3/sampling.py", line 487, in _sample_population
  File "/home/madanh/.virtualenvs/pymcdev/lib/python3.4/site-packages/pymc3-3.2-py3.4.egg/pymc3/sampling.py", line 820, in _prepare_iter_population
  File "/home/madanh/.virtualenvs/pymcdev/lib/python3.4/copy.py", line 97, in copy
    rv = reductor(2)
ValueError: must use protocol 4 or greater to copy this object; since __getnewargs_ex__ returned keyword arguments.

Versions and main components

• PyMC3 Version: 3.2 from master (commit eeef6c8a6603d866c7976c8824abde62a6eac566)
• Theano Version: 0.9.0
• Python Version: 3.4
• Operating system: linux mint 17
• How did you install PyMC3: with pip, into a virtualenv

[junpenglao]

I cannot reproduce it locally. I am using Theano 1.0 with PyMC3 from master
cc @michaelosthege

[madanh]

@junpenglao Upgraded theano -- same result. Just guessing here, but it might have something to do with python version, which one do you have?

[junpenglao]

python 3.5.2

[michaelosthege]

The error happens because the model (computation graph) can (for some unknown reason) not be pickled unless pickle.dump(..., protocol=4) is used.

But it is confusing because that sampling.py does use pickle.dumps(stepper, protocol=4).

[madanh]

@michaelosthege It's thrown by __reduce_ex__ of DEMEtropolis, which is probably inherited from some built-in, so it looks like a python bug. I will report when I will manage to upgrade python in that virtualenv.

[michaelosthege]

I'm running on Python 3.5.4.

@madanh Can you run the code below? And or modify it to reproduce the issue?

import pymc3
import copy

with pymc3.Model() as pmodel:
    n = pymc3.Normal('n')
    step = pymc3.DEMetropolis()
    copy.copy(step)
    trace = pymc3.sample(step=step, chains=4)

[madanh]

@michaelosthege Ran as is :

/home/madanh/.virtualenvs/pymcdev/bin/python /mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/minimal_example.py
/home/madanh/.virtualenvs/pymcdev/lib/python3.4/site-packages/pymc3-3.2-py3.4.egg/pymc3/step_methods/metropolis.py:519: UserWarning: Population based sampling methods such as DEMetropolis are experimental. Use carefully and be extra critical about their results!
Traceback (most recent call last):
  File "/mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/minimal_example.py", line 7, in <module>
    copy.copy(step)
  File "/home/madanh/.virtualenvs/pymcdev/lib/python3.4/copy.py", line 97, in copy
    rv = reductor(2)
ValueError: must use protocol 4 or greater to copy this object; since __getnewargs_ex__ returned keyword arguments.

Process finished with exit code 1

[madanh]

OK, reporting. I found no better way to test this in a recent python on my linux than to compile python. So I just did that for python3.6 with stable optimizations. Plopped it into a virtual env, ran the above example and now the theano won't start >:(((.

/mnt/emptyplaceholder/virtualenvs/py36pymcgit/bin/python3.6 /mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/minimal_example.py
Traceback (most recent call last):
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 75, in <module>

    raise ImportError()
ImportError
You can find the C code in this temporary file: /tmp/theano_compilation_error_l6t_sdme

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 92, in <module>
    raise ImportError()
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/emptyplaceholder/mcmc/Experiments/20180104_0_killer_tll/minimal_example.py", line 1, in <module>
    import pymc3
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/pymc3/__init__.py", line 5, in <module>
    from .distributions import *
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/pymc3/distributions/__init__.py", line 1, in <module>
    from . import timeseries
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 1, in <module>
    import theano.tensor as tt
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/__init__.py", line 110, in <module>
    from theano.compile import (
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/compile/__init__.py", line 12, in <module>
    from theano.compile.mode import *
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/compile/mode.py", line 11, in <module>
    import theano.gof.vm
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/gof/vm.py", line 673, in <module>
    from . import lazylinker_c
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 127, in <module>
    preargs=args)
  File "/mnt/emptyplaceholder/virtualenvs/py36pymcgit/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2359, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: Compilation failed (return status=1): /usr/bin/ld: /usr/local/lib/libpython3.6m.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC. /usr/local/lib/libpython3.6m.a: error adding symbols: Bad value. collect2: error: ld returned 1 exit status. 

Process finished with exit code 1

This happens with both theano 0.9.0 and 1.0.

[madanh]

Related: when running smc with njobs=4 it fails with a similar error, but all goes well (kind of) when njobs=1.

[madanh]

Guys could one of you test that minimal example with python downgraded to 3.4, please? If it passes then the problem is in my setup and we can close it.

[ColCarroll]

Fails on 3.4 with the same message for me. That's a pretty old version! In particular, pickle protocol 4 was added there. I wonder if there were some bugs that needed to be worked out...

Python 3.4.5 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:47:57) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymc3
>>> import copy
>>> 
>>> with pymc3.Model() as pmodel:
...     n = pymc3.Normal('n')
...     step = pymc3.DEMetropolis()
...     copy.copy(step)
...     trace = pymc3.sample(step=step, chains=4)
... 
/Users/colin/projects/pymc3/pymc3/step_methods/metropolis.py:519: UserWarning: Population based sampling methods such as DEMetropolis are experimental. Use carefully and be extra critical about their results!
  warnings.warn('Population based sampling methods such as DEMetropolis are experimental.' \
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/colin/anaconda3/envs/pymc33.4/lib/python3.4/copy.py", line 97, in copy
    rv = reductor(2)
ValueError: must use protocol 4 or greater to copy this object; since __getnewargs_ex__ returned keyword arguments.

[twiecki]

I don't think we should support 3.4. @madanh can't you upgrade?

[madanh]

@twiecki Apparently I will have to find a way:) I just can't reinstall the system now while in the middle of thesis writing, but this just for anecdotes, I agree that there's no good reason for you to support 3.4. Will try to make a new user and use conda there (it's interfering with my current virtualenvwrapper setup if I add it to PATH and bugs out if I don't :)

I suppose the "fix" for this is to have > 3.5 as requirement.

[ColCarroll]

The readme does say that 3.6 and 2.7 are the versions it is tested on (https://github.com/pymc-devs/pymc3#dependencies), though it might be helpful to document that it "probably works on 3.5 and no longer works on 3.4".

Closing for now, but happy to reopen (or continue discussion) if you'd like!

[madanh]

I've managed to get a working python 3.6 and confirm DEMetropolis starts without errors there.

[SOUMAJYOTI]

I was testing NUTS sampler and it raised the same error with Python 3.5 and Pymc3 3.5 - when the njobs is set to default 4. It only works when njobs is set to 1 (but then the operations are too slow). However, surprisingly the DEMEtropolis sampler works fine in my case.