“Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology

Abstract

Although the use of manipulation checks is widespread in social psychology, several researchers have raised methodological concerns about their use and interpretations. However, knowledge of how they are actually being used has been lacking. Extracting data from published reports of 207 recent experiments, we provide an empirical review of current practices concerning manipulation checks in social psychology. Our review suggests that there are serious deficiencies in the manner in which manipulation checks are used and interpreted. For example, published reports tend to contain highly limited quantitative reasoning about the effectiveness of manipulations, and researchers report little or nothing to address the possibility that manipulation checks might cause undesirable reactivity among participants. However, we argue that manipulation checks can be highly beneficial components of experiments when used properly, and they have untapped potential for the quantitative assessment of the strength of manipulations relative to the effect on the dependent variable (i.e., causal efficacy). To assist with such assessments, we provide empirical benchmarks for causal efficacy in social psychology. Additionally, we provide several recommendations for researchers and reviewers for improving the use and reporting of manipulation checks.