Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results.

Abstract

Today most conclusions about cumulative knowledge in psychology are based on meta-analysis. We first present an examination of the important statistical differences between fixed-effects (FE) and random-effects (RE) models in meta-analysis and between two different RE procedures, due to Hedges and Vevea, and to Hunter and Schmidt. The implications of these differences for the appropriate interpretation of published meta-analyses are explored by applying the two RE procedures to 68 meta-analyses from five large meta-analytic studies previously published in Psychological Bulletin. Under the assumption that the goal of research is generalizable knowledge, results indicated that the published FE confidence intervals (CIs) around mean effect sizes were on average 52% narrower than their actual width, with similar results being produced by the two RE procedures. These nominal 95% FE CIs were found to be on average 56% CIs. Because most meta-analyses in the literature use FE models, these findings suggest that the precision of meta-analysis findings in the literature has often been substantially overstated, with important consequences for research and practice.