Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

The trim and fill method allows estimation of an adjusted meta-analysis estimate in the presence of publication bias. To date, the performance of the trim and fill method has had little assessment. In this paper, we provide a more comprehensive examination of different versions of the trim and fill method in a number of simulated meta-analysis scenarios, comparing results with those from usual unadjusted meta-analysis models and two simple alternatives, namely use of the estimate from: (i) the largest; or (ii) the most precise study in the meta-analysis. Findings suggest a great deal of variability in the performance of the different approaches. When there is large between-study heterogeneity the trim and fill method can underestimate the true positive effect when there is no publication bias. However, when publication bias is present the trim and fill method can give estimates that are less biased than the usual meta-analysis models. Although results suggest that the use of the estimate from the largest or most precise study seems a reasonable approach in the presence of publication bias, when between-study heterogeneity exists our simulations show that these estimates are quite biased. We conclude that in the presence of publication bias use of the trim and fill method can help to reduce the bias in pooled estimates, even though the performance of this method is not ideal. However, because we do not know whether funnel plot asymmetry is truly caused by publication bias, and because there is great variability in the performance of different trim and fill estimators and models in various meta-analysis scenarios, we recommend use of the trim and fill method as a form of sensitivity analysis as intended by the authors of the method. Copyright 2007 John Wiley & Sons, Ltd.

Publication selection bias is a serious challenge to the integrity of all empirical sciences. We derive meta-regression approximations to reduce this bias. Our approach employs Taylor polynomial approximations to the conditional mean of a truncated distribution. A quadratic approximation without a linear term, precision-effect estimate with standard error (PEESE), is shown to have the smallest bias and mean squared error in most cases and to outperform conventional meta-analysis estimators, often by a great deal. Monte Carlo simulations also demonstrate how a new hybrid estimator that conditionally combines PEESE and the Egger regression intercept can provide a practical solution to publication selection bias. PEESE is easily expanded to accommodate systematic heterogeneity along with complex and differential publication selection bias that is related to moderator variables. By providing an intuitive reason for these approximations, we can also explain why the Egger regression works so well and when it does not. These meta-regression methods are applied to several policy-relevant areas of research including antidepressant effectiveness, the value of a statistical life, the minimum wage, and nicotine replacement therapy.