Flux.jl: Bayesian inference

[datnamer]

Hi Mike,

Any plans to include the ability to create models with priors and distributions?

Then inference would have to include facilities to sample from the posterior like MCMC, Variational methods etc

[MikeInnes]

I would love to start exploring this direction and I think that Flux can support it well. My main reason for prioritising ANNs right now is that focusing just makes the design problem more tractable; we can make that really good and then start branching out.

That's not to say I'm against starting to think about this, but I may need some help from people like yourself to start fleshing out the design. I think a good starting point is: What's the "hello world" for bayesian inference? e.g. the moral equivalent of logreg over MNIST for ANNs. We can start to work out how to express that minimal example in Flux's notation, and consider what extra features more advanced models introduce.

[datnamer]

Simplified target at first makes sense, but I'm thinking that Flux may not be right for bayesian modeling.

After perusing some papers and APIs, I think doing bayesian stuff right requires stochastic control flow and loops (ie a full turing complete prob programming lang). I don't think Flux's functional API would work best here.

[MikeInnes]

I imagine you're right that a functional interface isn't a good fit, but that's not all Flux provides. Essentially Flux does provide an embedded turing-complete programming language that happens to be amenable to various kinds of analysis (in particular differentiation, for now). Adding support for more control flow and loops in that language and doing different kinds of analysis shouldn't be an issue.

Of course, one of the benefits of sketching out a simple "hello world" is that it's a great test of feasibility.

[kskyten]

Edward is a python library that does this. Using Julia would make the syntax a lot nicer. BayesFlow in the tensorflow contributions is similar. From what I understood it is possible to use different backends for the ANN stuff in Flux? I'm not really familiar with it but Mocha.jl is a deep learning framework for Julia. What would be really awesome would be to combine Mocha.jl, Flux and something like Mamba.

[kskyten]

It would also be nice to simplify the analytically tractable portions of the graph with ConjugatePriors.jl and Sympy.jl or Mathematica.jl.

[kskyten]

Here's an example model http://mambajl.readthedocs.io/en/latest/tutorial.html. I think converting the Flux graph to the corresponding Mamba graph should work. I'm thinking something like this.

@dag
  s2 = InverseGamma(0.001, 0.001)
  beta = MvNormal(2, sqrt(1000))
  mu = xmat*beta
  sigma = sqrt(s2)
  y = MvNormal(mu, sigma)
end

The graph generated by this macro would act as a datastructure to encode to model structure. It could then be converted into for example the corresponding Mamba model

Mamba.Model(

  y = Stochastic(1,
    (mu, s2) ->  MvNormal(mu, sqrt(s2)),
    false
  ),

  mu = Logical(1,
    (xmat, beta) -> xmat * beta,
    false
  ),

  beta = Stochastic(1,
    () -> MvNormal(2, sqrt(1000))
  ),

  s2 = Stochastic(
    () -> InverseGamma(0.001, 0.001)
  )

)

It would also be great to be able to import these graphs from other MCMC software (e.g Stan).

[currymj]

Turing https://github.com/yebai/Turing.jl seems to be the best Julia package for probabilistic programming right now. They've got a lot of different MCMC samplers and a very intuitive DSL for defining the models.

Edward is definitely a good reference point for what a Bayesian framework that integrates with Flux's computation graph might look like. It can do both MCMC and variational inference. It provides a lot of useful abstractions but doesn't hide the Tensorflow backend.

Edward's examples directory provides a lot of good "hello, world!" examples. The classic ones are probably either Bayesian linear regression or the beta-bernoulli coin flip model (both in there).

https://github.com/blei-lab/edward/tree/master/examples

[MikeInnes]

Closing this for now as I don't think there's an immediate action item for us here. Most likely, it'd make sense to do probabilistic programming directly on top of CuArrays, rather than going through Flux.

[yebai]

@kskyten @currymj @MikeInnes I came across these comments when looking at Flux.jl.

I'm a developer for the Turing package. Just FYI, we are considering integrating dynamic graphs from Flux.jl into Turing.