Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Related collections

Most cited references 55

Record: found
Abstract: found
Article: not found

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Siri Isaksson, Colin F Camerer, Anna Dreber … (2018)

Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

0 comments Cited 381 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: not found
Book: not found

Principles of topological psychology.

Kurt Lewin, Fritz Heider, Grace M. Heider (1936)

0 comments Cited 333 times – based on 0 reviews

Bookmark

Record: found
Abstract: found
Article: not found

An Agenda for Purely Confirmatory Research.

Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom … (2012)

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

0 comments Cited 313 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Journal

Title: Advances in Methods and Practices in Psychological Science

Abbreviated Title: Advances in Methods and Practices in Psychological Science

Publisher: SAGE Publications

ISSN (Print): 2515-2459

ISSN (Electronic): 2515-2467

Publication date Created: November 08 2018

Publication date Created: December 2018

Publication date (Electronic): December 24 2018

Publication date (Print): December 2018

Volume: 1

Issue: 4

Pages: 443-490

Affiliations

[1 ]Laboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble Alpes

[2 ]Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua

[3 ]Behavioural Science Institute, Radboud University Nijmegen

[4 ]School of Pedagogical and Educational Sciences, Radboud University Nijmegen

[5 ]Department of Social Psychology, Tilburg University

[6 ]Department of Industrial Psychology and People Management, University of Johannesburg

[7 ]Department of Psychology, The Pennsylvania State University

[8 ]Department of Psychology, Yasar University

[9 ]Department of International Studies, American University of Sharjah

[10 ]Center for Advanced Hindsight, Duke University

[11 ]College of Business and Economics, United Arab Emirates University

[12 ]Department of Management, Faculty of Business Administration, University of Economics, Prague

[13 ]Erivan K. Haub School of Business, Saint Joseph’s University

[14 ]Institute of Psychology, ELTE Eötvös Loránd University

[15 ]Psychological and Social Sciences Program, Pennsylvania State University Abington

[16 ]Department of Psychology, California State University San Marcos

[17 ]Warsaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

[18 ]Department of General and Applied Psychology, University of Jos

[19 ]Sopot Faculty of Psychology, SWPS University of Social Sciences and Humanities

[20 ]Department of Psychology, University of Potsdam

[21 ]Centre for Languages and Literature, Lund University

[22 ]Institute of Psychology, Chinese Academy of Sciences

[23 ]Louvain Research Institute in Management and Organizations (LouRIM), Université catholique de Louvain

[24 ]Center on Consumers and Marketing Strategy (CCMS), Université catholique de Louvain

[25 ]Social Behavior Research Centre, Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

[26 ]Department of Psychology, Brooklyn College & Graduate Center, CUNY

[27 ]Facultad de Psicologia, Universidad del Desarrollo

[28 ]Centro de Apego y Regulacion Emocional, Universidad del Desarrollo

[29 ]Institute for Social Research, University of Michigan

[30 ]Mathematica Policy Research, Princeton, New Jersey

[31 ]Institute of Ethnology, Academia Sinica

[32 ]Department of Psychology, National Taiwan University

[33 ]Department of Psychology, Poitiers University

[34 ]CNRS Unité Mixte de Recherche 7295, Poitiers, France

[35 ]Division of Social Science, The Hong Kong University of Science and Technology

[36 ]Department of Psychology, HELP University

[37 ]Department of Psychology, University of Hawaii at Manoa

[38 ]Directorate of Psychology and Public Health, University of Salford

[39 ]Department of Psychology, Virginia Commonwealth University

[40 ]Department of Psychology, Azusa Pacific University

[41 ]Department of Psychology, University of Florida

[42 ]Department of Psychology, Grand Valley State University

[43 ]Department of Psychology, Harvard University

[44 ]Department of Psychology, Middle East Technical University

[45 ]Department of Psychology, Wittenberg University

[46 ]Leadership and Human Resource Management, WHU – Otto Beisheim School of Management

[47 ]Department of Psychology, San Diego State University

[48 ]Institute for Computing and Information Sciences, Radboud University Nijmegen

[49 ]Tilburg Institute for Behavioral Economics Research, Tilburg University

[50 ]Department of Psychology, Başkent University

[51 ]School of Psychology, The University of Queensland

[52 ]Office of Institutional Diversity, Allegheny College

[53 ]Department of Psychology, Yale University

[54 ]School of Applied Human Sciences, University of KwaZulu-Natal

[55 ]Department of Psychology, University of Virginia

[56 ]Department of Psychology, Rochester Institute of Technology

[57 ]Facultad de Psicología, Universidad Nacional Autónoma de México

[58 ]Shanghai Intercultural Institute, Shanghai International Studies University

[59 ]Departamento de Psicología, Universidad de los Andes, Colombia

[60 ]Department of Political and Juridical Sciences and International Studies, University of Padua

[61 ]Department of Marketing and International Business, Baruch College, CUNY

[62 ]Department of Organisation and Personnel Management, Rotterdam School of Management, Erasmus University

[63 ]Lazaridis School of Business and Economics, Wilfrid Laurier University

[64 ]Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg, Germany

[65 ]Institute of Education and Psychology, Johannes Kepler University Linz

[66 ]Departamento de Psicología Social y de las Organizaciones, Universidad Nacional de Educación a Distancia

[67 ]Escuela de Psicología, Pontificia Universidad Católica de Chile

[68 ]Eccles School of Business, University of Utah

[69 ]Psychology, Pacific Lutheran University

[70 ]Department of Experimental Clinical and Health Psychology, Ghent University

[71 ]Institute of Psychology, Faculty of Social and Political Sciences, University of Lausanne

[72 ]Amity Institute of Psychology and Allied Sciences, Amity University

[73 ]Department of Psychology, Northumbria University

[74 ]Department of Psychology, William Paterson University

[75 ]Department of Psychology, University of Michigan

[76 ]Smith Child Health Research, Outreach, and Advocacy Center, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois

[77 ]Department of Psychological & Brain Sciences, Texas A&M University

[78 ]Department of Economics and Management, Faculty of Social and Economic Studies, Jan Evangelista Purkyne University

[79 ]Department of Psychology, Loyola University Chicago

[80 ]Department of Science and Mathematics, Texas A&M University-San Antonio

[81 ]Department of Psychology, University of Toronto Scarborough

[82 ]Department of Psychology, Lund University

[83 ]Department of Social Science and Policy Studies, Worcester Polytechnic Institute

[84 ]Department of Psychology, University of Amsterdam

[85 ]Department of Management, London School of Economics and Political Science

[86 ]Department of Business Administration, Istanbul Bilgi University

[87 ]Department of Social and Work Psychology, Institute of Psychology, University of Brasilia

[88 ]Department of Social Psychology, ELTE Eötvös Loránd University

[89 ]Department of Psychology, Faculty of Philosophy, University of Belgrade

[90 ]Department of Work, Organizational and Media Psychology, Johannes Kepler University Linz

[91 ]Department of Psychology & Special Education, Texas A&M University-Commerce

[92 ]International Victimology Institute Tilburg, Tilburg University

[93 ]Department of Psychology, James Madison University

[94 ]School of Innovation Science, Eindhoven University of Technology

[95 ]Institute of Psychology, Faculty of Philosophy, University of Belgrade

[96 ]Department of Cognitive Science, Occidental College

[97 ]Department of Communication, Cornell University

[98 ]Department of Psychology, University of Porto

[99 ]Department of Methodology and Statistics, Tilburg University

[100 ]Department of Education Policy Planning and Administration, Faculty of Education, Open University of Tanzania

[101 ]Department of Occupational Therapy, Tufts University

[102 ]Faculty of Media and Communications, Singidunum University

[103 ]Institute of Criminological and Sociological Research, Belgrade, Serbia

[104 ]Department of Psychology, Universidad Latina de Costa Rica

[105 ]Centre for Applied Cross-Cultural Research, Victoria University of Wellington

[106 ]Department of Psychology, McDaniel College

[107 ]Melbourne School of Psychological Sciences, The University of Melbourne

[108 ]Department of Clinical, Neuro- and Developmental Psychology, Vrije Universiteit Amsterdam

[109 ]Amsterdam Public Health Research Institute, Amsterdam, The Netherlands

[110 ]Department of Psychology, University of Central Florida

[111 ]Department of Psychology, George Fox University

[112 ]Department of Psychology, Doshisha University

[113 ]Li Ka Shing Institute of Professional and Continuing Education (LiPACE), The Open University of Hong Kong

[114 ]Department of Psychology, Stanford University

[115 ]Institute for Psychological Research, University of Costa Rica

[116 ]Department of Communication and Cognition, Tilburg University

[117 ]Department of Psychology, California State University, Northridge

[118 ]School of Psychology, University of Sussex

[119 ]BehaviourWorks Australia, Monash Sustainable Development Institute, Monash University

[120 ]Department of Computer Science, University of Chicago

[121 ]Department of Psychology, Southern Illinois University Carbondale

[122 ]Department of Psychology, Ludwig-Maximilians-Universität München

[123 ]Institute of Psychology, Jagiellonian University in Kraków

[124 ]Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

[125 ]Center for Military Psychiatry and Neuroscience, Walter Reed Army Institute of Research, Silver Spring, Maryland

[126 ]Faculty of Psychology and Educational Science and Physical Education, University of Regensburg

[127 ]Occupational & Organisational Psychology and Professional Learning, KU Leuven

[128 ]Department of Psychology, University of Huddersfield

[129 ]Institute of Psychology, University of Graz

[130 ]Department of Psychology, Northeastern University

[131 ]Department of Psychology, University of Winchester

[132 ]Department of Psychology, Universidad de Iberoamerica

[133 ]Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam

[134 ]Department of Obstetrics and Gynaecology, Erasmus MC, Rotterdam, The Netherlands

[135 ]Methodology and Statistics Unit, Institute of Psychology, Leiden University

[136 ]Centro de Investigación Básica en Psicología, Universidad de la República

[137 ]Department of Psychology, Ithaca College

[138 ]Institute of Psychology, Paris Descartes University - Sorbonne Paris Cité

[139 ]Department of Social Psychology, University of Groningen

[140 ]Department of Psychology, Faculty of Arts, Charles University

[141 ]Department of Psychological Science, Western Kentucky University

[142 ]School of Psychology, University of New South Wales

[143 ]Department of Psychology, Washington and Lee University

[144 ]Department of Psychology, Boston College

[145 ]Department of Psychology, Carleton University

[146 ]Zhejiang University of Finance and Economics

[147 ]Center for Open Science, Charlottesville, Virginia

Article

DOI: 10.1177/2515245918810225

SO-VID: bd212a05-ba2a-4c73-863d-3d7eaa2294df

License:

http://journals.sagepub.com/page/policies/text-and-data-mining-license

History

Data availability:

Comments

Comment on this article

scite_

Cited by 221

See all cited by

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Read this article at

Related collections

Remedies to the replicability crisis

Most cited references 55

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Principles of topological psychology.

An Agenda for Purely Confirmatory Research.

Author and article information

Journal

Affiliations

Article

History

Comments

Comment on this article

Similar content 1,128

Cited by 221