Turing.jl: [BNP] Bayesian Nonparametrics project

[emilemathieu]

This is a macro issue to follow the development of Turing being able to handle nonparametric models.

• [x] Defining the class of models we aim to handle: Q-class cf https://github.com/yebai/Turing.jl/issues/374

  • [x] Implement DP, PYP via stick breaking: Here
  • [x] Q-class total mass: rand(ExpTiltedSigma(sigma, tau)): Here
  • [ ] Q-class total mass: logpdf(ExpTiltedSigma(sigma, tau))
  • [x] Q-class new weight with sigma=0.5: rand : Here
  • [ ] Q-class new weight with sigma=0.5: logpdf
  • [ ] Q-class new weight with sigma<0.5: rand and logpdf

• [x] Choose & implement model / metrics for the tests cf https://github.com/yebai/Turing.jl/issues/371

• [ ] Infinite array data structure for weights and locations of the RPM

  • [x] Can use tzeros(Float64, N) with N being the number of datapoints for both locations and weights.
  • [ ] Can use Dict{Int, Any}() for locations ?
  • [ ] Need a specific data structure InfiniteArray ?

• [x] SMC working on those models which implies handling array of random length cf https://github.com/yebai/Turing.jl/issues/367#issue-271525324

  • [ ] Chain wrapper issue https://github.com/yebai/Turing.jl/issues/386
  • [ ] [cache issue for getidcs](https://github.com/TuringLang/Turing.jl/blob/project-bnp/src/core/varinfo.jl#L260)

• [x] CSMC working

  • [x] Bug: missing prior of conditioned random variables in particles-weight https://github.com/yebai/Turing.jl/issues/383
  • [x] Handling random variables sampled in random order cf https://github.com/yebai/Turing.jl/issues/367#issue-271525324. Solutions proposed https://github.com/yebai/Turing.jl/issues/367#issuecomment-343270076
  • [x] Unnecessary operation in forkc cf https://github.com/yebai/Turing.jl/issues/358#issuecomment-343319106 ?
  • [x] Unknown random variables that do not belongs to CSMC's space cf https://github.com/yebai/Turing.jl/issues/373

• [x] HMC working on those models.

  • [x] Computing gradient cf https://github.com/yebai/Turing.jl/issues/375
  • [x] Initialisation of random variables in HMC cf https://github.com/yebai/Turing.jl/issues/368 => override init for SizeBiasedSamplingDistribution with rand(dist) https://github.com/yebai/Turing.jl/commit/1fb6ab880660186856defcc07ca6ef64d79683ee
  • [x] Do not update unused clusters in HMC ? cf https://github.com/yebai/Turing.jl/issues/372

• [ ] Combining samplers:

  • [ ] Particle Gibbs: Gibbs(N, CSMC(:z), HMC(:a, :p), HMC(...) )

  • [ ] Pseudo marginal: PMMH(N, SMC(...), MH(...))

  • [ ] New syntax: Avoid having to write the all stick-breaking / stick-biased process (by using a macro for instance)

[emilemathieu]

I created a branch with a proof of concept: https://github.com/yebai/Turing.jl/blob/project-bnp/test/rpm.jl/imm.jl.

[trappmartin]

Hi,
As discussed with @yebai I'll be looking into approaches to store parameters of RPMs in the Mamba.Chain struct within the next weeks. I'll keep you informed about the progress.

Cheers,
Martin

[yebai]

@emilemathieu @ae-foster @mlomeli1 Just to add, @trappmartin is a new member on the Turing team. He has previously implemented https://github.com/trappmartin/BNP.jl and now helping us to get BNP integrated into Turing:

• resolve https://github.com/TuringLang/Turing.jl/issues/413 first
• create a working example of DPmix

[emilemathieu]

Hi @trappmartin ! Great news !
I can't exactly remember where we were in https://github.com/TuringLang/Turing.jl/tree/project-bnp, but I believe I had a SMC and Gibbs sampler working on a PYP and NIGP mixture model.

We could have a skype session to talk about your plans, and if you have questions on what we did on that matter.

Cheers,
Emile

PS: @yebai we had the above mentioned examples working but not in the master branch indeed.
PS2: @trappmartin @yebai Also at the time there was still a cache issue with getidcs in VarInfo, that I hard fixed by commenting some lines which is obviously damaging performances. Something was wrong also I believe with checkgid in VarInfo.

[mlomeli1]

Hi @trappmartin , that's great news! If you have any questions I am happy to talk too.

[trappmartin]

Hi,
Thanks a lot for the warm welcome.

I’m happy to talk via Skype once I have a better overview of the codes.

[yebai]

I’m happy to talk via Skype once I have a better overview of the codes.

@trappmartin the current code is not super well documented, so pls feel free to ask any questions related to the code.

[emilemathieu]

Hi guys !
Is there some progress concerning the handling of BNP models ?

It'd be cool for Turing to be the first PPL to handle a non ridiculous sub-class of BNP models in an efficient manner. I feel it's not that far from being readily usable :)

Cheers,
Emile

[trappmartin]

Hi, after finishing the refactoring of the compiler we will work on changing the indexing scheme which will allow us to handle BNP models.

I agree that this will be super cool and hopefully possible very soon.

[emilemathieu]

Cool ! :)
Don't hesitate at that point if you have questions on the work I did.

[mlomeli1]

On my side, I am not working on how to improve the efficiency but will resume my poster work from next week and contribute with a notebook as @trappmartin suggested :)

[trappmartin]

Thanks, Maria!

@cpfiffer is currently working on improving the tutorials and documentation of Turing. It would probably be the best if @cpfiffer takes over the finished notebook once we can run BNP models using the master branch of Turing.

[emilemathieu]

I'd be willing to write a blog post about that once it can be ran using the master branch of Turing :)

[emilemathieu]

Hi all!

The 12th Bayesian nonparametrics (BNP) meeting will be held in Oxford from the 24th to the 28th of June 2019. The deadline for submitting a title/abstract is 15th December 2018.

Would you think it'd be worth submitting something related to generic inference in Turing for random probability measure based models ?

[trappmartin]

I think this is a great idea and would increase the visibility of Turing outside of the ML community.

[emilemathieu]

Great ! I have no idea whether I'll be in Oxford at that time though, but you may ;)

I guess the minimal goal is to be able to perform SMC inference on a simple model (e.g. mixture model) relying on another random probability measure than the DP or PYP (e.g. NIGP) using "first class object" (not having to write the stick breaking / size biased sampling process). It'd be even more impressive if we could perform Gibbs sampling since it should improve a lot mixing.

@trappmartin what is specifically missing towards such a goal ?

[trappmartin]

Sounds interesting. I already had different discussions with Hong. Let’s discuss this offline and after NIPS.

My time is currently rather limited so any help would appreciated.

[mlomeli1]

Hi @trappmartin , I am happy to help out but we would need to discuss which specific points I can contribute with. Let me know if Hong and you want to skype when you are back from NIPS.

[emilemathieu]

I'll also be happy to help and/or have a chat !

[trappmartin]

Thanks, I'm currently working on getting a iMM test example running in the BNP PR. Feel free to have a look and fix things. Most of the things should be there and should be working. I'll try to find some time tomorrow evening so that the PR is soon merged in the master.

Feel free to contribute if you like.

I would have time for a chat on Tuesday, Monday is already super packed with talks and meetings.

[emilemathieu]

Sure, can have a chat on Tuesday, what time ?