Power Contours: Optimising Sample Size and Precision in Experimental Psychology and Human Neuroscience

Abstract

When designing experimental studies with human participants, experimenters must decide how many trials each participant will complete, as well as how many participants to test. Most discussion of statistical power (the ability of a study design to detect an effect) has focused on sample size, and assumed sufficient trials. Here we explore the influence of both factors on statistical power, represented as a 2-dimensional plot on which iso-power contours can be visualized. We demonstrate the conditions under which the number of trials is particularly important, that is, when the within-participant variance is large relative to the between-participants variance. We then derive power contour plots using existing data sets for 8 experimental paradigms and methodologies (including reaction times, sensory thresholds, fMRI, MEG, and EEG), and provide example code to calculate estimates of the within- and between-participants variance for each method. In all cases, the within-participant variance was larger than the between-participants variance, meaning that the number of trials has a meaningful influence on statistical power in commonly used paradigms. An online tool is provided ( https://shiny.york.ac.uk/powercontours/) for generating power contours, from which the optimal combination of trials and participants can be calculated when designing future studies.

Translational Abstract

Many studies in neuroscience and experimental psychology involve testing human participants multiple times in a given condition, and averaging across these repetitions to get a more accurate estimate of the true response. Yet most researchers do not have a principled way to decide how many trials they should conduct, and decisions are often made using arbitrary criteria. This is an important issue because the number of trials has a direct effect on the statistical power of a study—the likelihood that it is able to detect a real effect. In the context of the recent “replication crisis” in psychology, researchers need tools to optimize the quality of their research designs to increase power. Here we propose a way to visualize the combined effect of sample size (the number of participants tested) and number of trials per participant on statistical power, using a two-dimensional contour plot. We show by subsampling eight existing data sets from a range of widely used methods (including reaction times, EEG, MEG, and fMRI) that these contours are curved, and permit estimation of an optimal number of participants and trials at the study design stage. All of the analysis scripts, as well as an online tool, are provided to permit others to tailor our methods to their own experimental paradigms. We anticipate that this approach will facilitate the design of experimental studies that are more efficient, and more likely to report real effects.