On the reliability of individual economic rationality measurements

Significance

Identifying potential determinants of rationality—interpreted as a characteristic of decision makers—is of great relevance from an applied science perspective: both policy makers and industry have a pronounced interest in understanding which individuals make rational decisions, be it to design effective policies, enhance equity, or fine-tune talent selection processes. However, especially for research at the frontier of foundation to application, we must ensure that our measurements are precise and reliable. Here, we show that established empirical measurements of rationality are not reliable enough, implicating the urgent need for advances in measurement of rationality.

Abstract

A contemporary research agenda in behavioral economics and neuroeconomics aims to identify individual differences and (neuro)psychological correlates of rationality. This research has been widely received in important interdisciplinary and field outlets. However, the psychometric reliability of such measurements of rationality has been presumed without enough methodological scrutiny. Drawing from multiple original and published datasets (in total over 1,600 participants), we unequivocally show that contemporary measurements of rationality have moderate to poor reliability according to common standards. Further analyses of the variance components, as well as a allowing participants to revise previous choices, suggest that this is driven by low between-subject variance rather than high measurement error. As has been argued previously for other behavioral measurements, this poses a challenge to the predominant correlational research designs and the search for sociodemographic or neural predictors. While our results draw a sobering picture of the prospects of contemporary measurements of rationality, they are not necessarily surprising from a theoretical perspective, which we outline in our discussion.