A Word of Caution about Many Labs 4: If You Fail to Follow Your Preregistered Plan, You May Fail to Find a Real Effect

Abstract

A team of 37 researchers (Many Labs 4; Klein et al., 2019) has recently reported that it has failed to replicate the effect of mortality salience on worldview defense – a classic finding from terror management theory (TMT). This collaborative project (21 labs, N = 2220) has the potential to provide useful information regarding the robustness of an often-replicated and influential empirical finding. However, Klein et al. (2019) deviated from their preregistered plan by including smaller samples than specified in their pre-registration. This undisclosed deviation from their plan is problematic because it undermines the potential benefits of pre-registration; indeed, the negative results of their meta-analyses appear to be predominantly driven by small studies. We reanalyzed the Many Labs 4 data after excluding studies that did not meet the preregistered minimum sample size (40 participants per cell). Results showed that the data actually do replicate the original study. This successful replication emerged only in the expert advice variation of these studies, which addresses one of the purposes of this study by showing that replication is more likely to occur when researchers follow the advice of researchers with considerable experience in this domain. We discuss the importance of following preregistered plans to avoid misleading conclusions and potential issues involved with literal replications of effects that may depend on societal zeitgeist at the time data are collected.