Turing.jl: TuringLang Roadmap

[yebai]

Roadmap

• Support for missing values #786 #544 #792
• Support for Gaussian processes (https://github.com/TuringLang/Turing.jl/issues/777)
• Distributed sampling (see e.g. posterior server)
• GPU-based Parallel Computation Support (see e.g. Stan https://arxiv.org/abs/1907.01063)

New Sampling Algorithms

• Piecewise-Deterministic MCMC sampling methods (see e.g. PDSampler.jl)
• Adaptive MH #287
• Approximate Bayesian computation (ABC) #347
• The scalable MH sampler described in https://arxiv.org/abs/1901.09881 might be interesting to integrate into Turing.

Deterministic Inference Algorithms

• Message passing algorithms for factor graphs (see e.g. ForneyLab.jl)
• Variational Inference for TuringLang #775
• Structured Stochastic Variational Inference #794

Tutorials

• Maybe translate some models from Infer.NET - https://dotnet.github.io/infer/userguide/Infer.NET%20tutorials%20and%20examples.html

Feature Requests

These are assembled in no particular order, and represent things various Turing users have requested or mentioned as something it would be nice to have.

• Formula notation support for Bayesian regression, as in @model Y ~ x1 + x2. This would create a Turing model on the backend, and support functions like predict(model) or something. See R's brms for more info.
• There's been a request for better causal inference support. See Omega.jl and this blog post.

Misc

• Generally improved plotting and visualization.
• Support for parallel chains.
• Running parallel chains with a single sample call.

[cpfiffer]

One thing I was thinking about is how cool it would be for us to provide a couple of "canned" models, like Bayesian linear regression to interface with StatsModels.jl. I could see someone calling

fit = bayeslm(@formula(y ~ 1 + x1 + x2), data = my_data_frame, nchains=4)

We could also do a logistic regression model, binomial, you name it. Plus, we could pretty heavily optimize the inference on the back end. It'd be a relatively quick way to get us more involved with the JuliaStats people and provide a rapid way of using Turing's awesome features without too much stress. It might get more people involved in MCMCChains as well, so we can figure out how to make it work better for everyone.

[yebai]

fit = bayeslm(@formula(y ~ 1 + x1 + x2), data = my_data_frame, nchains=4)

It would be cool to have this and might be a killer app for Julia. In fact, this should be a low-hanging fruit given Julia's meta-programming support and Turing's inference engines.

[cpfiffer]

I had actually start typing this up a little while ago and it was pretty straightforward. I can put it on my plate if you'd like, whenever I'm not working on the API overhaul.

[lamhm]

Hi guys,

What is the current status of the MH sampler. Does it auto-optimise the variances of the proposal distributions?

I am using the MH sampler for my current work (since I am facing a few troubles with the autodiff, and hence cannot use HMC). But the results returned from the MH sampler looked really strange, i.e. the chain barely moved. I have tried setting initial values for my parameter in the hope of improving the sampling. But the chain only moved a bit at the beginning, and the got stuck completely (this happened to all the parameters).

Anyway, I have not tried to run a long chain. The result above was from a few quick tests with only 1000 iterations. And I am not sure if this problem comes from the way I implement my model at the moment or from the MH sampler. I hope to have some advices from you guys before running more tests.

Thanks a lot.

Lam

[cpfiffer]

Our Metropolis-Hastings implementation is really bare-bones. Currently it's the most basic possible implementation of MH -- no adaptation of any kind.

If your model is a complex one you should probably try running a lot more samples (by the way, we can't see whatever result you are mentioning). You should open a new issue with your model specification -- it might also be worth fixing whatever your autodiff problem if possible so you can use a better sampler than MH.

[nickeubank]

The other example of "canned" Bayesian models is rstanarm: https://mc-stan.org/users/interfaces/rstanarm.html