The-turing-way: A new chapter for Power Analysis

[sparkler0323]

Summary

@KirstieJane, @minnieho1115 and I are working on a registered report in neuroscience. When preparing a registered report, the power analysis is necessary and is also very important for the reproducibility. However, it is often overlooked in studies in the neurosciences (median statistical power is between ∼8% and ∼31%) based on a previous study. Thus, here I aim to add some documents to generally & vividly introduce power analysis as part of the Turing Way.

What needs to be done?

• [ ] The basic concept of effect size and power analysis
• [ ] Drawing an original figure to help readers to understand the concept
• [ ] Why power analysis is important in registered reports/replication studies.
• [ ] A simple and clear example in python (downloading available data from the internet)
• [ ] the low statistical power and low reproducibility in current neuroscience studies
• [ ] Some useful tools or links for further reading

My preliminary thoughts are shown above, and I'm looking forward to your suggestions!

Who can help?

• Hope that @KirstieJane can help.
• Anyone who is familiar with power analysis or replication study

---

Updates

Lorem ipsum dolor sit amet, consectetur adipiscing.

[johalgermissen]

Sounds like a great initiative!

(Mostly) @jsheunis and (a bit also) me have recently worked on a similar project as a spin-off the this year's SIPS meeting in Rotterdam: https://brainpower.readthedocs.io/
Unfortunately the project is still in the early stage and a lot of documentation still must be done. Most promisingly, there are some recent simulation tools that we wanted to try out and document in a tutorial-style. See under "simulating data". We have an overview of useful tools, but still need to adequately review and document them. The goal was some kind of "TripAdvisor" for power-analysis tools in neuroimaging.

The problem with neuroimaging data is that inferences are typically not based on a single test, but usually on mass-univariate tests with some sort of correction for multiple comparisons (e.g. cluster-based inference). In those cases, defining and reporting an appropriate effect size is challenging in itself---just the maximum t- or z-value won't do, because people never do a single test on an isolated voxel. Taking the maximum only will overestimate any true underlying effect and its hardly useful to inform power for future studies. Rather, since tests in neuroimaging are based on summary statistics like cluster extend or cluster mass, one would first have to come up with proper metrics to define and report those, and then proper tools to incorporate them in simulations.
I've been thinking about/ working on this for a while (also related to EEG), but nothing shareable yet.

That's why in my opinion, the Button paper should be taken with care--it computes post-hoc power (which is just a transformation of the p-value with hardly any added value), they rely on dubious definitions of effect sizes extracted from a bunch of meta-analyses, and they report extremely low power (i.e. high chance for a Type II error) although many studies (in neuroscience in general and also in their sample) find significant effects (and thus by definition cannot commit a Type II error).

[KirstieJane]

This all sounds awesome @sparkler0323 & @johalgermissen!

Its important that _The Turing Way_ doesn't reinvent the wheel, nor that it gets too detailed on a specific domain. Having said that, I do think that a chapter on "powering your analyses" will be generally very important.

What I'd propose is that we build up a nice fun interactive jupyter notebook that can run in binder to demonstrate something quite simple and then _link_ to the Brain Power project. So the Turing Way becomes the "welcome mat" so that folks know that they should care about power, and if they're brain imagers they can link to https://brainpower.readthedocs.io (and other domain specfic projects!)

@sparkler0323 - if there's content that you can add to Brain Power that sounds like a really valuable contribution to me 💖

[sparkler0323]

Many thanks @jamespjh and @KirstieJane!

I read the paper you recommended and found it exactly can be added in this chapter. This paper addressed a very simple but important concept: it is half-chance to significantly replicate a work whose p-value is near a significant level by exactly the same paradigm and same size. As Kirstie mentioned, I expect this chapter could help someone who is not familiar with this topic and can understand this concept clearly, then he can also find some useful tools (like the BrainPower). Thus, I hope to add some simple concepts like this rather than some obscure mathematical equations, and also attach some demo examples to play with. It will be great if @johalgermissen can help or review this chapter later.

BrainPower looks a promising and useful tool to the neuroimaging community. I will love it more if it had some tutorial-style guidelines as you plan. I think BrainPower will be particularly useful for researchers in task-based fMRI (although I have no experience) who need to precisely determine their sample sizes (In other fields such as structural MRI or resting-state functional MRI, it seems researchers may expect a larger sample size if possible). This could be complicated and it will need some pilot data (maybe can use some existing datasets). I'm not an expert in power analysis but I will be very glad to provide some help in documentation if you need it.