Pymc3: What needs to be done for a release?

[springcoil]

At PyData London @twiecki and I chatted about what needed to be done for a release, and @jonsedar presented a nice advanced tutorial on it.
As I understand it there's just a few things left before a release candidate can be released.

• [x] API docs in Sphinx
• [x] ADVI to be better documented

Am I missing something major?

[springcoil]

@fonnesbeck @jsalvatier @taku-y you all may have opinions too...

[twiecki]

The other big missing piece is consistent shape handling (e.g. https://github.com/pymc-devs/pymc3/issues/1079). I talked a bunch with @brandonwillard who has an idea for a refactoring that might break some things. So we'd rather do such a change before 3.0. @brandonwillard anything to add?

[fonnesbeck]

A few things come to mind:

• [x] fix categorical variable sampling
• [ ] address issues with stalling of NUTS sampler
• [ ] vectors of multivariate variables (shape vs size argument) -- related to the comment above, I suppose
• [x] add deterministics to (and remove transformed stochastics from) ADVI fit output

[twiecki]

@fonnesbeck do we have a stalling NUTS example? I wonder if the ADVI init PR https://github.com/pymc-devs/pymc3/pull/1088 helps with that.

[taku-y]

@fonnesbeck Does deterministics mean the variational parameter in the original (not transformed) space? I think it is difficult to transform the standard deviation of the variational posterior in the transformed space back to the original space, because in the original space the variational distribution is no longer than Gaussian. Sorry if I misunderstand what you said.

And should we need a function to draw samples from variational posterior of ADVI? I wrote such a function and applied to the Bayesian neural network example. Unfortunately, in this example, variational posterior failed to fit data (https://gist.github.com/taku-y/6fabb9b905a6639019369ef418960bd2), possibly due to that the mean field approximation is too simple.

[fonnesbeck]

@taku-y the reported estimates from fitting with ADVI currently report only the stochastics, some of which are transformed to the real line, and hence not meaningful to users (we have actually hidden these variables from the trace in the MCMC samplers), whereas the deterministics are often of interest, and it would be helpful if their means and variances were reported automatically.

[twiecki]

@taku-y I definitely think such a sampling function would be useful. Ideally it returns a familiar Trace object. In your example, it's interesting that it doesn't work too well. I think the @stan-dev team uses some more tweaks as well, like a grid-search to optimize the eps parameter and something else to tune the learning rate. Have you looked at these?

[brandonwillard]

@twiecki I can try to get a branch with those changes up this week.

[twiecki]

@brandonwillard :+1:

[twiecki]

@taku-y The other thing that would be great for ADVI is a short tutorial in the form of an IPython NB with some basic info / references.

I agree that the retransformations will be difficult.

[taku-y]

@fonnesbeck Thanks for your comment. I understood what you said.

Instead of getting means/stds of deterministic variables, I'm trying to make a sampling function supporting deterministic variables. But I could not put drawn samples into a trace object. The notebook including the sampling function (sample_vp) is here (https://gist.github.com/taku-y/d768a34f9d3f985bab818669a0106bb7). Could you tell me how to solve the problem?

[taku-y]

@twiecki I will try to refine my Gaussian mixture example; add info of ADVI and references.

[taku-y]

@twiecki Thanks for telling me grid search of hyperparameters in STAN. Parameter tuning may solve the problem. Another option is taking more complex form of variational posterior as done by normalizing flows (http://arxiv.org/abs/1505.05770) and Hamiltonian variational inference (http://jmlr.org/proceedings/papers/v37/salimans15.pdf). Although it is out of scope of this issue, these techniques are interesting and may be useful.

[twiecki]

@taku-y here is the note from @akucukelbir on grid-search: https://github.com/pymc-devs/pymc3/pull/775#issuecomment-204059736

I see you already started work on the normalizing flows, does it work?

[taku-y]

@twiecki Thanks for pointing the code for grid-search. It's helpful for implementation in PyMC3.

I have implemented the normalizing flow but it has only single transform and did not seem show any effect so not confirmed it works. I will stack normalizing flows, say 40, as described in the paper.

[taku-y]

I fixed the problem on recording samples drawn (https://gist.github.com/taku-y/d768a34f9d3f985bab818669a0106bb7). After checking the sampling function on some other models (simple Gaussian and Bayesian NN), I will send a PR.

[twiecki]

@taku-y that looks very promising!

[taku-y]

I fixed a bug in the sampling function and the bayesian NN result looks great (https://gist.github.com/taku-y/6fabb9b905a6639019369ef418960bd2). I think it's almost ready to merge though should add a test.

[twiecki]

@taku-y Wow, that looks great. Very encouraging that mini-batch ADVI works so well in this case. Unit-test would certainly be great but looking forward to the PR.

[springcoil]

How are we doing on all of this, I've not had time to do much lately other than merging other PR's. Am I correct in appreciating that the ADVI stuff is pretty much fixed up and so we're nearly there?

[Eh2406]

HI, as a random person from the internet. Let me second the "ADVI to be better documented" Just heard about ADVI, and I am no expert. So I don't know what "this is a mean-field approximation" means in practise? More generally what can I do with ADVI? What can I not do with ADVI? Are there ways to use ADVI in conjunction with other methods?

[taku-y]

@springcoil Sorry for late to reply. In addition to documents and including deterministics in sampling function for ADVI, I want to add autoencoding variational Bayes for mini-batch ADVI. It allows us to include local latent variables to models so I believe it would enhance the applicability of ADVI. I will send a proposal as an issue.

[taku-y]

@Eh2406 Thanks for your comments. I think your questions are important especially for practitioners. I will answer these questions in the document.

[springcoil]

Hi @taku-y that's really cool I look forward to it. I'll look into the ADVI documentation too - maybe that's a contribution I could make.

[taku-y]

@springcoil I'm working on autoencoding VB on #1236 and #1241. After these PR accepted, I will start to write some documents.

[springcoil]

What needs to be done here? @twiecki and @fonnesbeck what do you think needs to be done for a release? I think the documentation is coming along.

[twiecki]

I think we can release anytime (and should soon), there's always something.

[twiecki]

I think we should just pull the trigger.

On Sep 3, 2016 7:43 PM, "Peadar Coyle" [email protected] wrote:

What needs to be done here? @twiecki https://github.com/twiecki and
@fonnesbeck https://github.com/fonnesbeck what do you think needs to be
done for a release?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244559981,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AApJmHfnnqtm_fyM1a0KNTK9tiJ8YQGcks5qmbHTgaJpZM4Ie77t
.

[springcoil]

Agreed. Pull the trigger

On 4 Sep 2016 7:45 AM, "Thomas Wiecki" [email protected] wrote:

I think we should just pull the trigger.

On Sep 3, 2016 7:43 PM, "Peadar Coyle" [email protected] wrote:

What needs to be done here? @twiecki https://github.com/twiecki and
@fonnesbeck https://github.com/fonnesbeck what do you think needs to
be
done for a release?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244559981,
or mute the thread
fyM1a0KNTK9tiJ8YQGcks5qmbHTgaJpZM4Ie77t>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244587207,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8DiP2C4w4li7Z5U07eXmqNID7uRXAUks5qmmkjgaJpZM4Ie77t
.

[springcoil]

Ok let's put a date down - we can fight over this later :) I propose 6th of September. Is that too early? What needs to be changed for a 'release' do we just need to update something on PyPI?

[fonnesbeck]

Well, I was hoping to have the multivariate shape issue and the stalling behavior for NUTS sampling to be solved before a release, but I am happy to set those aside if everyone else is keen.

I will go over the outstanding issues today and see if anything jumps out. We should definitely give master a good PEP8 compliance check and general linting before tagging a release.

[springcoil]

Hi Chris,
I think while those are great - pushing for a release now with all the
momentum we've had would be great.

I don't think multivariate shape issue is a deal breaker.

Definitely +1 on the PEP8 stuff

On Mon, Sep 5, 2016 at 5:29 PM, Chris Fonnesbeck [email protected]
wrote:

Well, I was hoping to have the multivariate shape issue and the stalling
behavior for NUTS sampling to be solved before a release, but I am happy to
set those aside if everyone else is keen.

I will go over the outstanding issues today and see if anything jumps out.
We should definitely give master a good PEP8 compliance check and general
linting before tagging a release.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244782232,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8DiCekLswK8uYbSUzkMWspRMQqUaKoks5qnEN7gaJpZM4Ie77t
.

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

[springcoil]

I think assuming @twiecki nails the PEP8 stuff today, which I think he will, all the other things like test fixes, and other fixes by myself last night mean I think we're ready for a release candidate. I'd love to announce this today at PyData London Meetup.

[jsalvatier]

I just want to cheer you guys on! Great job!

[springcoil]

Hey John - thanks man - you of course can't be ignored in this effort :)

On Tue, Sep 6, 2016 at 4:30 PM, John Salvatier [email protected]
wrote:

I just want to cheer you guys on! Great job!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244989413,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8DiCjBOduBoP0EJRcludU9kehIaNRcks5qnYcSgaJpZM4Ie77t
.

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

[springcoil]

Now we have a release candidate done. I'll close this.