Arviz: request for Bayes Factor plot

[drbenvincent]

I wonder whether it would be useful to have a Bayes Factor type plot, like this?

[ColCarroll]

It sounds like there is at least one person who would find it useful, and my impression is that they are standard (if somewhat controversial?)

We can leave this issue open for discussion, or for potential contributors to mention that they are working on it?

As an aside, this is the first time I've seen a sketch as a PR, and I love it.

[drbenvincent]

It's not impossible that I might attempt it. But only after I've actually submitted a bunch of papers, late in the summer.

There's a bit of debate between estimation and CI's vs Bayesian hypothesis testing. I've not really dug into it, but good examples of the pro-estimation approach are:

• Kruschke, J. K. (2011). Bayesian Assessment of Null Values Via Parameter Estimation and Model Comparison. Perspectives on Psychological Science, 6(3), 299–312. http://doi.org/10.1177/1745691611406925
• Kruschke, J. K., & Liddell, T. M. (2015). The Bayesian New Statistics: Two historical trends converge, 1–21.

And some examples on the pro hypothesis testing side...

• Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E.-J. (2014). Why hypothesis tests are essential for psychological science: a comment on Cumming (2014). Psychological Science, 25(6), 1289–1290. http://doi.org/10.1177/0956797614525969
• Wagenmakers, E.-J., Verhagen, J., Ly, A., Matzke, D., Steringoever, H., Rouder, J. N., & Morey, R. D. (2015). The Need for Bayesian Hypothesis Testing in Psychological Science, 1–29.

I'm not sure, but I think the Bayes Factor camp might have the slight edge, certainly in my field because of the legacy of frequentist hypothesis testing and because of it's use in tools like JASP which makes great plots like this...

Personally I always report CI's or HDI's and optionally BF's when practical for the modelling context.

[treszkai]

We should keep in mind that this ArviZ not only allows people to plot the things they want to, but it also shapes how people think about statistical plotting (or it will in the future). So the question is if we think Bayesian hypothesis testing against a point parameter value is useful. Some arguments against this method are:

• BF is quite sensitive to the prior on the parameter.
• Testing against a single parameter value should be replaced with parameter estimation, so suggested statistics are the HDI, posterior median/mean/mode, and the posterior probability of a ROPE'd value are more meaningful statistics. (See e.g. Kruschke: The Bayesian New Statistics.)

Furthermore, the suggested graph looks like the prior and posterior plotted on the same Axes, with a particular value singled out (which is incompatible with Bayesian thinking), and the BF plotted on it. This could be done without new plotting functions, as described in #749.

(Nonetheless, +1 for the sketch!)

[drbenvincent]

Well I guess this just highlights that there is a debate between parameter estimation vs treating specific values as special. I was previously very much in the parameter estimation camp, but then got convinced that there is some usefulness in evaluating point hypotheses with Bayes Factors sometimes. I am not a statistician, so I can't really contribute to that debate as such. But Bayes Factors do seem to have some influence in my field - may that's a hang up from people who've not fully gotten away from NHST yet. Although I don't think that's really it... EJ Wagenmakers is a strong advocate of using Bayes Factors for example.

In terms of sensitivity to priors... yes! Although there are some classes of model (eg linear modelling, but not just that) where it is possible to define sensible priors which you can defend and justify.

In terms of people missing BF's and not knowing what they are doing, all I can say is that that is a danger in many ways when doing Bayesian modelling. I wasn't advocating for a pre-defined workflow, pushing people into BF's, more like just requesting the feature for those who want it and look for it in documentation etc.