Pymc3: Multinomial Mixture in PyMC3

[vsmolyakov]

I'm trying to create a mixture of multinomials using PyMC using pm.Mixture construct as described in documentation. However, I'm running into dimension mismatch error:

T = 4
V = 10
with pm.Model() as model:
    H = pm.Dirichlet("H", a=np.ones(V), shape=V)
    phi = [pm.Multinomial('phi_%s' %t, n=10, p=H, shape=V) for t in range(T)]
    phi_dists = [comp.distribution for comp in phi]
    G0 = pm.Mixture('G0', w = np.ones(T)/T, comp_dists = phi_dists)

This code results in the following error:
ValueError: Input dimension mis-match. (input[0].shape[1] = 4, input[1].shape[1] = 10)
I do have a mixture with 4 components each of size 10.

I also tried constructing the mixture manually:

with model:
    phi_mix = [w[t]*phi_top[t] for t in range(T)]
    G0 = pm.Deterministic('G0', sum(phi_mix))

However, the above code lead to a different downstream error. So I was wondering how I can construct a mixture of multinomials using PyMC3? This should be a common task for discrete data models.
This question is also posted on stack overflow.

[junpenglao]

This seems to be a bug related to Multinomial.

[springcoil]

Aha thanks @junpenglao I completely missed this :)

[bsmith89]

Sorry if this isn't helpful (I'm not entirely clear on what you're trying to do), but I have a Dirichlet-multinomial implementation that might be related to what you're looking for.

import pymc3 as pm
import theano.tensor as tt

class DirichletMultinomial(pm.Discrete):
    """Dirichlet Multinomial Distribution

    Parameters
    ----------
    alpha :  dirichlet parameter

    """
    def __init__(self, alpha, n, *args, **kwargs):
        super(DirichletMultinomial, self).__init__(*args, **kwargs)
        self.alpha = alpha
        self.n = n

    def logp(self, x):
        gammaln = pm.distributions.special.gammaln
        sum = tt.sum

        alpha = self.alpha

        if not n:
            n = sum(x, axis=-1)
        sum_alpha = sum(alpha, axis=-1)
        const = (gammaln(n + 1) + gammaln(sum_alpha)) - gammaln(n + sum_alpha)
        series = gammaln(x + alpha) - (gammaln(x + 1) + gammaln(alpha))
        result = const + sum(series, axis=-1)
        return result

[vsmolyakov]

Thanks. I'm trying to implement a Hierarchical Dirichlet Process (HDP) mixture model for discrete data, e.g. an HDP topic model where each document is a mixture of topics, i.e. a mixture of multinomials. So I need to be able to implement a mixture of discrete multivariate distributions in PyMC3.

[bsmith89]

I think the Mixture class takes hard-coded (float) weights (rather than a tensor of weights), so it may not be appropriate for this application.

[junpenglao]

@bsmith89 the weights in Mixture takes tensor as well - the doc string need to be updated. Thanks for noticing ;-)

I am still trying to identify the Input dimension mis-match bug related to @vsmolyakov's original post...

[AustinRochford]

The Mixture class should definitely be able to take a tensor of weights. If not, it's definitely a bug.

[aloctavodia]

I am getting the same error for a mixture of MVNormal.

The following toy-model runs only when K = D.

K = 2
D = 2
data = np.random.normal(size=(D, 10)).T  # nonsensical data
with pm.Model() as model:
    μ = pm.Normal('μ', 0 , 100, shape=(K, D))
    Σ = np.eye(D)
    y = pm.Mixture('y', w=np.array([1/K] * K),
                   comp_dists=[pm.MvNormal.dist(μ[i], Σ) for i in range(K)],
                   observed=data)
    trace = pm.sample()

It's not clear to me why this is failing, @AustinRochford @junpenglao?

[junpenglao]

@aloctavodia I think the reason is https://github.com/pymc-devs/pymc3/issues/2614.

[ColCarroll]

Will hopefully get to that PR again this weekend, but feel free to steal it
if you have time! I like the strategy of relying on np.broadcast, since it
is "least surprising".

On Wed, Oct 11, 2017, 11:22 AM Junpeng Lao [e-mail] wrote:

@aloctavodia https://github.com/aloctavodia I think the reason is #2614
https://github.com/pymc-devs/pymc3/issues/2614.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/2478#issuecomment-335847521,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACMHEDd1q6zWonUw8CeseM1qGbw7-syIks5srN0ogaJpZM4OuM37
.

[AustinRochford]

I'll take a look this evening, but agree that the semantics for mixtures of vector-values expressions are not great.

[shkr]

@junpenglao How were you able to run this notebook https://gist.github.com/junpenglao/f5b3749393c92f516230a1ea389d1380 ? The shape mismatch error is still present. I am not able to use the Mixture RV unusable because only when mixture components is equal to dimension, then only it builds the model.

[junpenglao]

@shkr I got a more up to date notebook you can run from the current master: https://github.com/junpenglao/Planet_Sakaar_Data_Science/blob/master/WIP/%5BWIP%5D%20Bayesian%20GMM.ipynb