Pymc3: Vectors of multivariate variables

[fonnesbeck]

It would be useful if we could model multiple independent multivariate variables in the same statement. For example, if I wanted four multivariate normal vectors with the same prior, I should be able to specify:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), shape=(4,3))

but it currently returns a ValueError complaining of non-aligned matrices.

[jsalvatier]

will it be obvious what dimension is the multivariate dimension?

On Fri, May 2, 2014 at 10:16 AM, Chris Fonnesbeck
[email protected]:

It would be useful if we could model multiple independent multivariate
variables in the same statement. For example, if I wanted four multivariate
normal vectors with the same prior, I should be able to specify:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), shape=(4,3))

but it currently returns a ValueError complaining of non-aligned matrices.

—
Reply to this email directly or view it on GitHubhttps://github.com/pymc-devs/pymc/issues/535
.

[fonnesbeck]

It should be intuitive, if not obvious. I think most people would expect a vector of variables, which implies that the first dimension is the number of variable elements and the remaining dimension(s) the size of each variable.

[fonnesbeck]

We at least need to be able to do the analog of this:

m = [pm.MvNormal('m_{}'.format(i), mu, Tau, value=[0]*3) for i in range(len(unique_studies))]

This has been a show-stopper for me trying to use PyMC 3 for new work, so I'm going to try to set aside some time to work on this.

Thinking about it some more, however, I think that shape is not the appropriate way to specify the dimension of a multivariate variable -- that should be reserved for the size of the vector of variables. Might be best to have:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), dim=3)

for a single variable and:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), shape=4, dim=3)

for a vector containing 4 MvNormals of dimension 3. Better yet, we ought to be able to infer the dimension of the MvNormal from its arguments.

[jsalvatier]

That makes some sense. The words shape and dim seem very close, so it seems
confusing to have both

On Thu, May 29, 2014 at 1:30 PM, Chris Fonnesbeck
[email protected]:

We at least need to be able to do the analog of this:

m = [pm.MvNormal('m_{}'.format(i), mu, Tau, value=[0]*3) for i in range(len(unique_studies))]

This has been a show-stopper for me trying to use PyMC 3 for new work, so
I'm going to try to set aside some time to work on this.

Thinking about it some more, however, I think that shape is not the
appropriate way to specify the dimension of a multivariate variable -- that
should be reserved for the size of the vector of variables. Might be best
to have:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), dim=3)

for a single variable and:

f = pm.MvNormal('f', np.zeros(3), np.eye(3), shape=4, dim=3)

for a vector containing 4 MvNormals of dimension 3. Better yet, we ought
to be able to infer the dimension of the MvNormal from its arguments.

—
Reply to this email directly or view it on GitHubhttps://github.com/pymc-devs/pymc/issues/535#issuecomment-44581060
.

[fonnesbeck]

Perhaps using plates here would be clearer, since this is common terminology in graphical models. The we could generalize the business of generating vectors of variables.

[fonnesbeck]

Just bumping this one. I come up against it frequently in epidemiological analyses.

[aflaxman]

I like the originally proposed notation, shape=(4,3), since that will be the shape of f.value. Am I stuck in a PyMC2 way of thinking?

[fonnesbeck]

I'd be happy with that. We would just have to adopt the convention that the last dimension is always the size of the individual multivariate node, and not the size of the array containing the nodes. The tricky part comes when you have, say, a vector of Wisharts that is itself multidimensional, so the total shape could be (4,4,3,3) for a 4x4 array of 3x3 variables.

[twiecki]

I would imagine it's a rare case but can't hurt to consider it and come up with a sane way to handle. In the end, complex things will be complex in code but defaulting to the last dimensions is an easy rule to keep in mind.

[aflaxman]

+1 for shape=(4,4,3,3) to get a 4x4 array of 3x3 wisharts. So

C = pm.WishartCov('C', C=np.eye(3), n=5)
C.value.shape == (3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=(4,3,3))
C.value.shape == (4,3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=(4,4,3,3))
C.value.shape == (4,4,3,3)

[fonnesbeck]

Multivariate classes could have the appropriate dimension specified in the class to know how to deal with the shape argument. Wisharts will always be 2-dimensional, for example, so any remaining dimensions will always be how many wisharts are in the set. Multinomials will always be a 1-d vector, etc.

[jsalvatier]

Okay, are we agreed that when we do this the multivariate dimensions start at the back?

We could start them at the front, but the way numpy.dot works suggests at the back.

[twiecki]

Agree on defaulting to last dimension.

I wonder, is the shape argument not redundant? Seems like we can always infer it from the inputs. shape could then only add the dimensions. Personally I would find this less confusing:

C = pm.WishartCov('C', C=np.eye(3), n=5)
C.value.shape == (3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=4)
C.value.shape == (4,3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=(4,4)))
C.value.shape == (4,4,3,3)

The 3,3 is already encoded in np.eye(3), no?

[jsalvatier]

Shape is not redundant when you want to have the same prior arguments for a
bunch of variables.

On Mon, Jul 27, 2015 at 2:14 PM Thomas Wiecki [email protected]
wrote:

Agree on defaulting to last dimension.

I wonder, is the shape argument not redundant? Seems like we can always
infer it from the inputs. shape could then only add the dimensions.
Personally I would find this less confusing:

C = pm.WishartCov('C', C=np.eye(3), n=5)
C.value.shape == (3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=4)
C.value.shape == (4,3,3)

C = pm.WishartCov('C', C=np.eye(3), n=5, shape=(4,4)))
C.value.shape == (4,4,3,3)

The 3,3 is already encoded in np.eye(3), no?

—
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-125345104.

[twiecki]

right, I'm only talking about the case where the input to the RV (e.g. C above) is multi-dimensional already. Then you can use shape to repeat that input arbitrarily.

pm.Normal('x', mu=[1, 2, 3], shape=2) would give a 2x3 in my proposal.

[jsalvatier]

Yeah, we could do that. I'm slightly worried that its going to make
implementation more complex. And perhaps be confusing to users. But maybe
either way is going to be confusing.

On Mon, Jul 27, 2015 at 2:23 PM Thomas Wiecki [email protected]
wrote:

right, I'm only talking about the case where the input to the RV (e.g. C
above) is multi-dimensional already. Then you can use shape to repeat
that input arbitrarily.

pm.Normal('x', mu=[1, 2, 3], shape=2) would give a 2x3 in my proposal.

—
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-125346789.

[twiecki]

I recently ran into the confusion where I wanted 2 Dirichlets of len 3, should I do:
pm.Dirichlet(np.ones((2, 3)), or should I do pm.Dirichlet(np.ones((2, 3)), shape=(2, 3)) or maybe pm.Dirichlet(np.ones((2, 3)), shape=2) or pm.Dirichlet(np.ones(3), shape=2)? Are some equivalent? I actually still don't know.

So with my proposal there's a clear rule and I don't have to remember which dimensions of the shape kwarg match to which dimensions of my input.

Why do you think it would be harder to implement?

[jsalvatier]

That does seem attractive from an API point of view.

Let me check how that plays with broadcasting rules.

[jsalvatier]

That does seem to play nicely with things.

I see two issues. Maybe we can resolve them.

First, this change will break previously working models.

Second, shape is a common argument for all distributions and this means the shape argument won't match the actual shape of the variable. I think that might not actually break anything right now, but seems like a bug waiting to happen.

Perhaps we should have a different argument, not shape for multivariate distributions, but count or dimensions or something else that is used to compute the shape. That would make it more obvious that the behavior is different.

[twiecki]

Or maybe repeat? pm.Dirichlet(np.ones(3), repeat=2) would give a 2x3.

What I also like about this is that it makes the translation from pymc2 style [pm.Dirichlet(np.ones(3)) for i in range(2)] more direct.

So if we were to change this, do we still need the shape kwarg? Do we deprecate it?

[twiecki]

And maybe we could even use theano.tensor.extra_ops.repeat(x, repeats, axis=None) for this.

[twiecki]

Theoretically we could even teach users to use repeat directly and not be concerned with all this in the API. E.g.:

from pymc3 import repeat # alias to theano.tensor.extra_ops.repeat
pm.Dirichlet(repeat(my_prior, 2)) # gives 2x3 if my_prior is shape 3

[fonnesbeck]

I don't think we should worry about breaking changes too much in a beta for such an important design decision.

I like the idea of a dim (dimension) argument that represents the shape of the variable, rather than how many of them there are:

Sigma = Wishart('Sigma', 4, np.eye(3), dim=3)
mu = pm.Normal('mu', 0, 0.001, shape=3)
x = pm.MvNormal('x', mu, Sigma, dim=3, shape=5)

which results in an x that consists of 5 multivariate normals, each of dimension 3.

[jsalvatier]

Shape currently means the actual shape of the resulting variable, and I kind of want to keep that unless there's a good reason.

[fonnesbeck]

So, it would be x = pm.MvNormal('x', mu, Sigma, dim=3, shape=(5,3))? I wonder if it can just be inferred, then, from the shapes of mu and Sigma?

Or, why not have the variable compose the variable shape from dim and shape, just to be explicit ?

[jsalvatier]

Dim seems like it should always be inferrable and we should compute shape. Then we could have count or something which gives the non-dim shape of the variable?

[twiecki]

I think that with count (or repeat) we can infer dim _and_ shape. IMO it's really bad user experience if there's 3 ways to specify the same redundant thing.

[fonnesbeck]

This is something we need to solve before a 1.0 release. Brought up again in #824

[twiecki]

@fonnesbeck What's your take on the proposal to remove shape in favor of count or repeat?

[fonnesbeck]

I like the idea of having a count argument that defaults to 1, and the shape of the variable is inferred. So, for example:

X = Normal('X', 0, 1) results in a scalar normal variable
X = Normal('X', [0,0], [1,2]) results in a vector-valued normal, as inferred from the arguments
X = Normal('X', 0, 1, count=2) results in an array of 2 scalar normals
X = MvNormal('X', mu, Sigma) results in a single multvariate normal whose shape is inferred from mu and Sigma
X = MvNormal('X', mu, Sigma, count=2) results in a vector of multvariate normals, the size of each element inferred from mu and Sigma

Will this work, and does it provide a clear separation between the shape of a variable and the size of a vector of variables?

[twiecki]

@fonnesbeck sounds great then. I think we can use theano.tile and deprecate the shape argument (which will be inferred). And should the resulting shape be the shape of the final RV (i.e. including count)?

For example Normal('X', 0, 1, count=2).shape == (1, 2)?

[twiecki]

@jsalvatier I can't find where the shape argument takes effect, could you point me to that?

[fonnesbeck]

@twiecki I guess it would have to be, since we can't easily iterate over things in Theano. In PyMC 2, this would have been a Container object.

BTW, I think the shape there would be (2,1). The first dimension should be the count, and all subsequent dimensions the shape, I would think.

[twiecki]

Right, (2, 1)

[jsalvatier]

@twiecki Do you mean here? https://github.com/pymc-devs/pymc3/blob/master/pymc3/model.py#L438

You're looking to change the name of 'shape'?

[twiecki]

@jsalvatier Yes. I just went through the code but it seems that inferring the shape from the input arguments (e.g. mu) might be difficult because they don't get forwarded to the super class (i.e. Distribution which sets the shape parameter). So for example, Normal() gets a float or an array for mu but only stores it and doesn't let the parent class know about it where would want to infer its shape.

Unfortunately, I don't see an easy way to change that without a medium refactoring. Ideas?

[fonnesbeck]

There will still be a shape attribute, but it will not be an argument for creating variables.

[twiecki]

Of course, we could add specific logic to each Distribution to extract the shape from the input parameters. That doesn't strike me as very elegant though.

[fonnesbeck]

That could be done as a decorator, making it less inelegant.

[twiecki]

Good idea, that would work. Then we infer the shape in the child class, pass it to the base class along with count where the broadcast is done. Easy peasy.

Just not sure about the kwarg name. Don't really like count too much. How about repeat or tile?

[fonnesbeck]

It should be something that clearly denotes a number (I.e. variable count) of variables, as separate from the size or dimension of the variable. Not sure tile or repeat make this clear, but I could be convinced. I suppose tile to be consistent with Theano? Perhaps nvars or similar?

[twiecki]

well it could be multdim, no? E.g. Normal(0, 1, repeat=(3, 3)) would give a 3x3, same as Normal(np.zeros((3, 3)), np.ones((3, 3))). Not ideal to have two ways to specify the same thing but probably not unreasonable. Both tile and repeat have connections to theano.

[fonnesbeck]

I would argue that those are two different parameterizations; the first is a single multidimensional normal with shape (3,3) and the second a 3x3 matrix of univariate normals. The difference is trivial in this example, but when you pop a "true" multvariate distribution in place of the normal the distinction is real.

MvNormal('X', np.zeros(2), np.eye(2), tile=2)

This cannot be expressed without the tile argument.

One possible utility of the difference in parameterizations may be in whether a variable gets block-updated or not, say, in a Metropolis algorithm. In the first parameterization perhaps it would, and the second it would not.

[twiecki]

OK, we're on the same page. I like the idea of Metropolis but that's probably better left for later.

I don't think this would be hard to implement at all with the decorator. The key is that we just pass the correct shape to the base class where the broadcasting is already implemented. The logic would be something like:
shape = np.concatenate(mu.shape, tile)

The only thing I noticed, which I don't think is a big deal, is that numpy.tile works slight different as it requires specification of the full shape:

But it'd be fine for our tile to work differently. It feels pretty intuitive to me (unlike our earlier shape spec).

[twiecki]

Turns out a decorator is difficult because parameters can either be args or kwargs. Instead, here's a function that infers the shape given one parameter and kwargs passed to Distribution:

import warnings

def infer_shape(param, kwargs):
    shape = kwargs.get('shape', None)
    if shape is not None:
        warnings.warn('Explicitly specifying the shape argument will be deprecated. Shape will be inferred from the input variables as well as the new tile keyword argument.',
                      PendingDeprecationWarning)
        shape = np.atleast_1d(shape)

    tile = kwargs.get('tile', ())

    if shape is None:
        if isinstance(param, (float, int)):
            shape = ()
        else:
            shape = np.asarray(param).shape

    inferred_shape = np.concatenate([np.atleast_1d(tile), shape])
    if len(inferred_shape) == 0:
        inferred_shape = np.array([1])

    return np.int8(inferred_shape)


print(infer_shape(0, {}))
print(infer_shape([0, 0], {}))
print(infer_shape([0, 0], {'tile': 3}))
print(infer_shape(0, {'tile': 3}))
print(infer_shape(np.zeros((3, 3)), {'tile': 2}))
print(infer_shape(np.zeros((3, 3)), {'shape': 2, 'tile': 3}))

Output:

[1]
[2]
[3 2]
[3]
[2 3 3]
[3 2]

Btw. I saw above you suggested plates that might also be a good name for tile.

[twiecki]

[twiecki]

The remaining issue is related to theano not allowing linear algebra on higher-dimensional variables. I started a PR here to support it https://github.com/Theano/Theano/pull/3836

[fonnesbeck]

We could require that tile be a keyword argument. I don't think that would be a huge inconvenience.

[twiecki]

tile is already a keyword argument. The problem is that mu can be a kwarg or an arg.

[twiecki]

@fonnesbeck supporing multidimensional linear algebra in theano is tricky. However, I was able to do this with list comprehension:

%matplotlib inline
import pymc3 as pm
import numpy as np
import scipy.stats as stats
cov = np.asarray([[1, 0, 0],
                  [0, 1, 0],
                  [0, 0, 1]])

rand = np.random.multivariate_normal([0, 1, 2], cov, size=(2, 100))

with pm.Model() as model:
    mu = pm.Normal('mu', mu=1, sd=2, tile=(2, 3))
    tau = np.stack([np.eye(3), np.eye(3)])
    like = [pm.MvNormal('like%i', mu=mu[i, :], tau=tau[i, :, :], observed=rand[i, :, :]) for i in range(2)]

    trace = pm.sample(500)

That should give us what we need, no?

[fonnesbeck]

That is one solution, I suppose. I do like the idea of having a PyMC object representing the vector of variables, rather than a list, but perhaps I'm just being picky. In other words, treating a multivariate distribution the same way we treat a univariate distribution.

[fonnesbeck]

I'm trying to use the list comprehension approach above, for example:

        p_uae = [pm.Dirichlet('p_uae_%i' % i, θ, shape=n_outcomes) for i,θ in enumerate(θ_uae)]

however, I get errors suggesting that this is not cool with Theano:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py in __setitem__(self, item, value)
    501                     raise TypeError("Ambiguous name: %s - please check the "
    502                                     "names of the inputs of your function "
--> 503                                     "for duplicates." % str(item))
    504                 if isinstance(s, gof.Container):
    505                     s.value = value

TypeError: Ambiguous name: p_uae_30_stickbreaking - please check the names of the inputs of your function for duplicates.

Prior to the error, I am also flooded with Theano warnings that look like this:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/gradient.py:545: UserWarning: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: p_6_stickbreaking
  handle_disconnected(elem)

As far as I can tell, there should be nothing in the model that is ambiguously named. Here is the full model:

import theano.tensor as T

SumTo1 = pm.transforms.SumTo1()
inverse_logit = pm.transforms.inverse_logit

def specify_model(model):

    with model:

        # Impute followup times
        followup_time = pm.Uniform('followup_time', followup_min, followup_max, 
                                   shape=len(followup_min), 
                                   observed=followup_masked)

        # Mean probabilities (on logit scale)
        μ = pm.Normal('μ', 0, 0.01, shape=n_outcomes)
        # Followup time covariates 
        β_fup = pm.Normal('β_fup', 0, 0.01, shape=n_outcomes)
        # Age covariate
        β_age = pm.Normal('β_age', 0, 0.01, shape=n_outcomes)

        # Study random effect
        τ = pm.Exponential('τ', 0.1, testval=1)
        ϵ = pm.Normal('ϵ', 0, τ, shape=n_studies)

        # Expected value (on logit scale)
        θ_uae = [T.exp(μ + β_fup*followup_time[i] + β_age*age_centered[i] + ϵ[study_index[i]]) 
                         for i in range(arms)]

        # Inverse-logit transformation to convert to probabilities
        p_uae = [pm.Dirichlet('p_uae_%i' % i, t, shape=n_outcomes) for i,t in enumerate(θ_uae)]

        # Multinomial data likelihood
        likelihood = [pm.Multinomial('likelihood_%i' % i, followup_n[i], p_uae[i], 
                                     observed=outcomes[i]) for i in range(arms)]

        p_6 = pm.Dirichlet('p_6', T.exp(μ + β_fup*6), shape=n_outcomes)
        p_12 = pm.Dirichlet('p_12', T.exp(μ + β_fup*12), shape=n_outcomes)
        p_6_50 = pm.Dirichlet('p_6_50', T.exp(μ + β_fup*6 + β_age*10), shape=n_outcomes)

    return model

There is no naming ambiguity that I can see.

[twiecki]

Hm, yeah, that looks pretty good to me. Have you tried T.concatenate to convert them to vectors? I'd also be curious if @nouiz thinks that a list-comprehension of theano expressions isn't supposed to work.

[nouiz]

I didn't found what line was tried for list-comprehension that didn't worked. Can you give me that line?

List-comprehension work when Theano is able to infer the shape of the variable. But we didn't implemented all cases. Also, in the dev version of Theano, there is more case implemented then in the last release 0.7. This work for example:

import theano
x=theano.tensor.matrix()
[i+j for i,j in enumerate(x.shape)]

[fonnesbeck]

It happens in the p_uae list comprehension. Says there is an ambiguous name despite being uniquely named via enumerate indices.

[fonnesbeck]

Using concatenate or stacklists does not seem to help either:

ValueError                                Traceback (most recent call last)
<ipython-input-92-882bbf1c3e98> in <module>()
----> 1 uae_model = specify_model(uae_model, 'uae')
      2 
      3 with uae_model:
      4 
      5     if True:

<ipython-input-91-98403b254ade> in specify_model(model, intervention)
     46         # Inverse-logit transformation to convert to probabilities
     47         π = T.concatenate([pm.Dirichlet('π_%i' % i, t, shape=n_outcomes) 
---> 48                           for i,t in enumerate(θ_uae)])
     49 
     50         # Multinomial data likelihood

<ipython-input-91-98403b254ade> in <listcomp>(.0)
     45 
     46         # Inverse-logit transformation to convert to probabilities
---> 47         π = T.concatenate([pm.Dirichlet('π_%i' % i, t, shape=n_outcomes) 
     48                           for i,t in enumerate(θ_uae)])
     49 

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/tensor/var.py in __iter__(self)
    546     def __iter__(self):
    547         try:
--> 548             for i in xrange(theano.tensor.basic.get_vector_length(self)):
    549                 yield self[i]
    550         except TypeError:

/Users/fonnescj/anaconda3/lib/python3.5/site-packages/theano/tensor/basic.py in get_vector_length(v)
   4282     else:
   4283         msg = str(v)
-> 4284     raise ValueError("length not known: %s" % msg)
   4285 
   4286 

[fonnesbeck]

Even without concatenating, and also dropping back to Metropolis sampling, it does not go well.

[Miles-Garnsey]

How do these proposals play with constructs like the randomwalk structures which have a natural shape?

I might be running across some of the issues described here in a new issue I raised #960.

[nouiz]

You can not always loop on a Theano variable. Doing it via a for loop or list comprehensions is the same.

Theano need to infer the shape when the loop is executed and we don't do it for many cases. If you know the shape, you can do it manually, but looping for the number of iteration you need and do the indexing.

replace:
[... for i, v in enumerate(o_uae)]

by

[... for i in range(N)]

and replace v by o_uae[i] in the ...

[junpenglao]

My personal experience:
I applied the code similar @fonnesbeck and it works really well under Metropolis

kobs = [pm.Multinomial('kobs_%i' % i, p = theta[:,i], n=Nitem, 
                                     observed=k[i]) for i in np.arange(Nsubj)]

However, using NUTS it returns the following Theano error, which I am not able to track down.

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
    clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

But the sampling result is correct as well.
If you guys find it helpful I can post the full model I used.

[nouiz]

Can you update Theano to the dev version?

http://www.deeplearning.net/software/theano/install.html#bleeding-edge-install-instructions

If that don't fix it, I need a way to reproduce it. You can use the
function_dump method in Theano:

http://deeplearning.net/software/theano/library/compile/function.html#theano.compile.function.function_dump

On Fri, Feb 26, 2016 at 8:49 AM, Junpeng Lao [email protected]
wrote:

My personal experience:
I applied the code similar @fonnesbeck https://github.com/fonnesbeck
and it works really well under Metropolis

kobs = [pm.Multinomial('kobs_%i' % i, p = theta[:,i], n=Nitem,
observed=k[i]) for i in np.arange(Nsubj)]

However, using NUTS it returns the following Theano error, which I am not
able to track down.

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR (theano.gof.opt): SeqOptimizer apply MergeOptimizer
ERROR:theano.gof.opt:SeqOptimizer apply MergeOptimizer
ERROR (theano.gof.opt): Traceback:
ERROR:theano.gof.opt:Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 230, in apply
sub_prof = optimizer.optimize(fgraph)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 89, in optimize
ret = self.apply(fgraph, _args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/theano/gof/opt.py", line 809, in apply
clients = pairs[0][0].clients + pairs[0][1].clients
AttributeError: 'TensorVariable' object has no attribute 'clients'

But the sampling result is correct as well.
If you guys find it helpful I can post the full model I used.

—
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-189284686.

[junpenglao]

Thanks @nouiz, I am already using the dev branch. I will try to use the function_dump method.

[psinger]

I am wondering whether there is already a solution for this? How can I simply specify multiple Dirichlet distributions at once (e.g., for a mixture)?

[PietJones]

I am also trying to get see how to implement a dirichlet process that can be applied to a matrix/dataframe, trying to extend the existing example of the Dirichlet Process.

N = df.shape[0] #df has a shape of 400,1000
K = 30

with pm.Model() as model:    
    alpha = pm.Gamma('alpha', 1., 1.)
    beta = pm.Beta('beta', 1., alpha, shape=K)
    w = pm.Deterministic('w', beta * T.concatenate([[1], T.extra_ops.cumprod(1 - beta)[:-1]]))
    component = pm.Categorical('component', w, shape=N)

    alpha_w = pm.Gamma('alpha_w',1.,1.,shape=K)
    target_beta  = pm.Beta('targets', alpha_w,1., shape=K)
    likelihood = pm.Beta('like', alpha_w[component], 1.,observed=df.values)
    #likelihood = pm.Beta('like', T.concatenate([alpha_w[component] for ii in range(df.shape[1])]), 1.,observed=df.values)
    #likelihood = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)]

    step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
    step2 = pm.ElemwiseCategoricalStep([component], np.arange(K))

    trace_ = pm.sample(20000, [step1, step2])

I have tried the three definitions of the likelihood, but they all fail. The first one gives an inconsistent shape error. The second produces a similar error to what Chris got in his gist. The third gives me a theano error: Attribute list has not attribute owner, when trying to assign the step method.

Sorry if I this is not the correct topic, but the information in this thread seems to be related. Any help would be much appreciated.

[nouiz]

Can you give me the full Theano error for the 3rd case?

On Wed, May 4, 2016 at 10:55 PM, PietJones [email protected] wrote:

I am also trying to get see how to implement a dirichlet process that can
be applied to a matrix/dataframe, trying to extend the existing example of
the Dirichlet Process.

N = df.shape[0] #df has a shape of 400,1000
K = 30

with pm.Model() as model:
alpha = pm.Gamma('alpha', 1., 1.)
beta = pm.Beta('beta', 1., alpha, shape=K)
w = pm.Deterministic('w', beta * T.concatenate([[1], T.extra_ops.cumprod(1 - beta)[:-1]]))
component = pm.Categorical('component', w, shape=N)

alpha_w = pm.Gamma('alpha_w',1.,1.,shape=K)
target_beta  = pm.Beta('targets', alpha_w,1., shape=K)
likelihood = pm.Beta('like', alpha_w[component], 1.,observed=df.values)
#likelihood = pm.Beta('like', T.concatenate([alpha_w[component] for ii in range(df.shape[1])]), 1.,observed=df.values)
#likelihood = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1., observed=df.values[ii,:]) for ii in range(N)]

step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
step2 = pm.ElemwiseCategoricalStep([component], np.arange(K))

trace_ = pm.sample(20000, [step1, step2])

I have tried the three definitions of the likelihood, but they all fail.
The first one gives an inconsistent shape error. The second produces a
similar error to what Chris got in his gist. The third gives me a theano
error: Attribute list has not attribute owner, when trying to assign the
step method.

Sorry if I this is not the correct topic, but the information in this
thread seems to be related. Any help would be much appreciated.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217064072

[nouiz]

The first Theano function called is:

/Users/jq2/git/Theano/theano/gof/graph.py in inputs(variable_list, blockers)

I think the problem is that variable_list or blockers aren't a list of
Theano variable. One of the element is probably a list instead of a Theano
variable.

Can you verify that with pdb? If so, the fix will be in pymc3

On Thu, May 5, 2016 at 9:34 AM, PietJones [email protected] wrote:

AttributeError Traceback (most recent call last)
in ()
15 obs = [pm.Beta('m_%i' %ii, alpha_w[component[ii]],1.,
observed=df.values[ii,:]) for ii in range(N)]
16
---> 17 step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w,
target_beta,obs])
18 step2 = pm.ElemwiseCategoricalStep([component], np.arange(K))
19

/Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/step_methods/arraystep.pyc
in _new_(cls, _args, *_kwargs)
46
47 #get the actual inputs from the vars
---> 48 vars = inputvars(vars)
49
50 if not blocked and len(vars) > 1:

/Users/jq2/miniconda/envs/pymc3/lib/python2.7/site-packages/pymc3-3.0-py2.7.egg/pymc3/theanof.pyc
in inputvars(a)
19 r : list of tensor variables that are inputs
20 """
---> 21 return [v for v in inputs(makeiter(a)) if isinstance(v,
t.TensorVariable)]
22
23 def cont_inputs(f):

/Users/jq2/git/Theano/theano/gof/graph.py in inputs(variable_list,
blockers)
695
696 """
--> 697 vlist = ancestors(variable_list, blockers)
698 rval = [r for r in vlist if r.owner is None]
699 return rval

/Users/jq2/git/Theano/theano/gof/graph.py in ancestors(variable_list,
blockers)
674 if r.owner and (not blockers or r not in blockers):
675 return reversed(r.owner.inputs)
--> 676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs')
677 return dfs_variables
678

/Users/jq2/git/Theano/theano/gof/graph.py in stack_search(start, expand,
mode, build_inv)
642 rval_list.append(l)
643 rval_set.add(id(l))
--> 644 expand_l = expand(l)
645 if expand_l:
646 if build_inv:

/Users/jq2/git/Theano/theano/gof/graph.py in expand(r)
672 """
673 def expand(r):
--> 674 if r.owner and (not blockers or r not in blockers):
675 return reversed(r.owner.inputs)
676 dfs_variables = stack_search(deque(variable_list), expand, 'dfs')

AttributeError: 'list' object has no attribute 'owner'

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217155671

[PietJones]

Tried pdb, not sure I am doing it right, but it gave the following, after inserting:

pdb.set_trace()

Before the step1 assignment

Applied log-transform to alpha and added transformed alpha_log to model.
Applied logodds-transform to beta and added transformed beta_logodds to model.
Applied log-transform to alpha_w and added transformed alpha_w_log to model.
Applied logodds-transform to targets and added transformed targets_logodds to model.
> /Users/jq2/Downloads/TestPYMC3.py(45)<module>()
 -> step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])
(Pdb) n
 AttributeError: "'list' object has no attribute 'owner'"
> /Users/jq2/Downloads/TestPYMC3.py(45)<module>()
-> step1 = pm.Metropolis(vars=[alpha, beta, w, alpha_w, target_beta,likelihood])

(Pdb) type(likelihood)
<type 'list'>
(Pdb) type(target_beta)
<class 'pymc3.model.TransformedRV'>
(Pdb) type(likelihood[0])
<class 'pymc3.model.ObservedRV'>

[twiecki]

@PietJones You shouldn't include observed variables to be sampled.

[nouiz]

Can PyMC3 give a better user error for that case?

On Thu, May 5, 2016 at 10:21 AM, Thomas Wiecki [email protected]
wrote:

@PietJones https://github.com/PietJones You shouldn't include observed
variables to be sampled.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217167357

[PietJones]

Cool, I took that from:

http://austinrochford.com/posts/2016-02-25-density-estimation-dpm.html

After changing, now I get the following error:

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpYXDK_O/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256. 

Is there some size limit that I am not aware of? The data frame is not that large: (450, 1051)

[nouiz]

You can use this PR for a work around:

https://github.com/Theano/Theano/pull/4289

On Thu, May 5, 2016 at 10:30 AM, PietJones [email protected] wrote:

Cool, I took that from:

http://austinrochford.com/posts/2016-02-25-density-estimation-dpm.html
http://url

After changing, now I get the following error:

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpYXDK_O/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

Is there some size limit that I am not aware of? The data frame is not
that large: (450, 1051)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217169460

[PietJones]

@nouiz Thnx for the advice, again not sure if this was what you meant that I should do, but I tried the following, and I still get the same error:

cd ~/git
git clone https://github.com/Theano/Theano
git fetch origin pull/4289/head:pr-4289
git checkout pr-4289
python setup.py develop

I then restarted my ipython/jupyter kernel and reran my code.

[nouiz]

Delete your Theano cache. If that don't fix it, you probably using the old
Theano. Uninstall Theano many times to be sure it is not installed and
reinstall as you just did.

On Thu, May 5, 2016 at 11:05 AM, PietJones [email protected] wrote:

@nouiz https://github.com/nouiz Thnx for the advice, again not sure if
this was what you meant that I should do, but I tried the following, and I
still get the same error:

cd ~/git
git clone https://github.com/Theano/Theano
git fetch origin pull/4289/head:pr-4289
git checkout pr-4289
python setup.py develop

I then restarted my ipython/jupyter kernel and reran my code.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217178862

[PietJones]

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.__version__"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

[nouiz]

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me
the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones [email protected] wrote:

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1): /Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32: fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using
jupyter (did restart the kernel), (don't have cuda). Sorry for the trouble.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

[PietJones]

Hi,

Find attached the mod.cpp file which failed to compile.

https://gist.github.com/PietJones/8e53946b2738008095ced8fb9ab4db44

I dont think the entire file uploaded, just in case here is google drive link:

https://drive.google.com/file/d/0B2e7WGnBljbJZnJ1T1NDU1FjS1k/view?usp=sharing

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien [email protected]
wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me
the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones [email protected]
wrote:

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):
/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:
fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv, using
jupyter (did restart the kernel), (don't have cuda). Sorry for the
trouble.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

[nouiz]

Update Theano to 0.8.2. I have the impression that you use an older version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones [email protected] wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien <[email protected]

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send me
the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones [email protected]
wrote:

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:
fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv,
using
jupyter (did restart the kernel), (don't have cuda). Sorry for the
trouble.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

[PietJones]

I originally had that version of Theano, which gave the same error. The
older version I posted about above was using a specific Pull Request to see
if that would help.

P

On Fri, May 6, 2016 at 9:03 AM, Frédéric Bastien [email protected]
wrote:

Update Theano to 0.8.2. I have the impression that you use an older
version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones [email protected]
wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien <
[email protected]

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error send
me
the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones [email protected]
wrote:

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:

fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv,
using
jupyter (did restart the kernel), (don't have cuda). Sorry for the
trouble.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
<
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605>

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217433674

[nouiz]

Can you confirm it was the pull request about the GpuJoin proble on windows
(
https://github.com/Theano/Theano/pull/4289)?
https://github.com/Theano/Theano/pull/4289

I check that code and it would have tested what I wanted to test.

Can you manually apply this diff and test again?

"""
diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
index cd74c1e..e9b44b5 100644
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -6761,7 +6761,7 @@ def elemwise_max_input_fct(node):
# inputs.
if not theano.config.cxx:
return 31

• return 1024
• return 512

local_elemwise_fusion = local_elemwise_fusion_op(T.Elemwise,
"""

If it still fail, instead of a max of 512, try 256, 128, ...

Tell me the biggest number that work.

On Fri, May 6, 2016 at 9:47 AM, PietJones [email protected] wrote:

I originally had that version of Theano, which gave the same error. The
older version I posted about above was using a specific Pull Request to see
if that would help.

P

On Fri, May 6, 2016 at 9:03 AM, Frédéric Bastien <[email protected]

wrote:

Update Theano to 0.8.2. I have the impression that you use an older
version.

Fred

On Thu, May 5, 2016 at 1:25 PM, PietJones [email protected]
wrote:

Hi,

Find attached the mod.cpp file which failed to compile.

On Thu, May 5, 2016 at 1:00 PM, Frédéric Bastien <
[email protected]

wrote:

I taught that you where on windows with a GPU.

Then you have a new case.

Can you use this Theano flag: nocleanup=True then after the error
send
me
the file that failed compilation.

On Thu, May 5, 2016 at 12:44 PM, PietJones <[email protected]

wrote:

I tried the following.

rm -r ~/.theano*
pip uninstall theano #did this several times until there was error
cd ~/git/theano #then fetched the PR, did git checkout etc
python setup.py develop #also tried python setup.py install

python -c "import theano; print theano.version"
0.8.0.dev-410eacd379ac0101d95968d69c9ccb20ceaa88ca

Still the same problem.

Exception: ('Compilation failed (return status=1):

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpJ01xYP/mod.cpp:27543:32:

fatal error: bracket nesting level exceeded maximum of 256.

If it helps, I am running this on a MacOSX, in a conda virtualenv,
using
jupyter (did restart the kernel), (don't have cuda). Sorry for the
trouble.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
<
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217206605>

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
<
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217210834>

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217216992

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217433674

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-217444869

[PietJones]

Thnx for the advice, I tried all of the above, editing the file manually, removing the .theano directory, then restarting the jupyter kernel and running the code again, still get the same error. This is also further down before the actual traceback:

/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpEMrDR6  /mod.cpp:35943:32: fatal error: bracket nesting level exceeded maximum of 256
    if (!PyErr_Occurred()) {
                           ^
/Users/jq2/.theano/compiledir_Darwin-14.5.0-x86_64-i386-64bit-i386-2.7.11-64/tmpEMrDR6/mod.cpp:35943:32: note: use -fbracket-depth=N to increase maximum nesting level
1 error generated.

[nouiz]

Which new value did you try? Only 512? Can you try something like 31? If it still fait with 31, then try this diff:

diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
index a08e900..ef0821d 100644
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -6724,6 +6724,8 @@ def local_add_mul_fusion(node):
                 isinstance(inp.owner.op.scalar_op, s_op)):
             l = list(node.inputs)
             l.remove(inp)
+            if len(l) + len(inp.owner.inputs) > 31:
+                return
             output_node = node.op(*(l + inp.owner.inputs))

             copy_stack_trace(node.outputs[0], output_node)

This opt could also cause this extra big Elemwise.

[PietJones]

I have tried 1024, 512, 256 and 31, they all result in the same problem.
Not sure what correction you want me to implement, as the formatting of
what you sent has been corrupted. But the changes that I tried was :

5549 def local_add_mul_fusion(node):
5550 """Fuse consecutive add or mul in one such node with more inputs.
5551
5552 It is better to fuse add/mul that way then in a Composite node as
5553 this make the inner graph of the Compiste smaller. This allow to
5554 put more computation in a Composite before hitting the max
5555 recusion limit when pickling Composite.
5556
5557 """
5558 if (not isinstance(node.op, Elemwise) or
5559 not isinstance(node.op.scalar_op, (scalar.Add, scalar.Mul))):
5560 return False
5561
5562 s_op = node.op.scalar_op.class
5563 for inp in node.inputs:
5564 if (inp.owner and
5565 isinstance(inp.owner.op, Elemwise) and
5566 isinstance(inp.owner.op.scalar_op, s_op)):
5567 l = list(node.inputs)
5568 l.remove(inp)
5569 if len(l) + len(inp.owner.inputs) > 31:
5570 return
5571 #return [node.op(_(l + inp.owner.inputs))]
5572 output_node = node.op(_(l + inp.owner.inputs))
5573 copy_stack_trace(node.ouput[0],output_node)
5574

which still gave an error:
https://gist.github.com/PietJones/26339593d2e7862ef60881ea09a817cb

On Tue, May 10, 2016 at 10:16 AM, Frédéric Bastien <[email protected]

wrote:

Which new value did you try? Only 512? Can you try something like 31? If
it still fait with 31, then try this diff:

diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
index a08e900..ef0821d 100644
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -6724,6 +6724,8 @@ def local_add_mul_fusion(node):
isinstance(inp.owner.op.scalar_op, s_op)):
l = list(node.inputs)
l.remove(inp)

• ## if len(l) + len(inp.owner.inputs) > 31:

```
return
output_node = node.op(*(l + inp.owner.inputs))

copy_stack_trace(node.outputs[0], output_node)
```

This opt could also cause this extra big Elemwise.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/pymc-devs/pymc3/issues/535#issuecomment-218170906

[twiecki]

@fonnesbeck I think this works for Multivariate now, right?

[fonnesbeck]

A list comprehension seems to work now, yes. Ultimately I'd like to be able to specify a vector of multivariates using the shape argument, as in the original issue, but that will be for post-3.0.

[twiecki]

I think that should also work, no? At least for 3D multivariates.

[fonnesbeck]

It runs, but does not "work":

from pymc3 import *

with Model():

    p = Dirichlet('p', np.ones(3), shape=(4,3))
    x = Multinomial('x', np.array([20, 16, 10, 5]), p, shape=(4,3))
    print('p initial:', p.tag.test_value)
    print('x initial:', x.tag.test_value)
    tr = sample(50)
    print('x final:', x.tag.test_value)

yields:

p initial: [[ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]
 [ 0.33333333  0.33333333  0.33333333]]
x initial: [[7 7 7]
 [5 5 5]
 [3 3 3]
 [2 2 2]]
Assigned NUTS to p_stickbreaking_
Assigned Metropolis to x
100%|██████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 287.60it/s]
x final: [[7 7 7]
 [5 5 5]
 [3 3 3]
 [2 2 2]]

So, the x's don't sum to n, yet it does not fail!

[fonnesbeck]

This is tied up in the shape refactoring. Closing.