Consensus-based guidance for conducting and reporting multi-analyst studies

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Related collections

Most cited references 45

Record: found
Abstract: found
Article: found

Is Open Access

The FAIR Guiding Principles for scientific data management and stewardship

Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg … (2016)

There is an urgent need to improve the infrastructure supporting the reuse of scholarly data. A diverse set of stakeholders—representing academia, industry, funding agencies, and scholarly publishers—have come together to design and jointly endorse a concise and measureable set of principles that we refer to as the FAIR Data Principles. The intent is that these may act as a guideline for those wishing to enhance the reusability of their data holdings. Distinct from peer initiatives that focus on the human scholar, the FAIR Principles put specific emphasis on enhancing the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals. This Comment is the first formal publication of the FAIR Principles, and includes the rationale behind them, and some exemplar implementations in the community.

0 comments Cited 3299 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review.

Lukas Radbruch, Saskia Jünger, Sheila Payne … (2017)

The Delphi technique is widely used for the development of guidance in palliative care, having impact on decisions with relevance for patient care.

0 comments Cited 445 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: not found
Article: not found

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, Colin F Camerer … (2020)

0 comments Cited 366 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Balazs Aczel:

ORCID: https://orcid.org/0000-0001-9364-4988

Barnabas Szaszi:

ORCID: https://orcid.org/0000-0001-7078-2712

Peter Rodgers: Role: Reviewing Editor

Peter Rodgers: Role: Senior Editor

Journal

Journal ID (nlm-ta): eLife

Journal ID (iso-abbrev): Elife

Journal ID (publisher-id): eLife

Title: eLife

Publisher: eLife Sciences Publications, Ltd

ISSN (Electronic): 2050-084X

Publication date (Electronic, pub): 09 November 2021

Publication date Collection: 2021

Volume: 10

Electronic Location Identifier: e72185

Affiliations

[1 ] ELTE Eotvos Lorand University Budapest Hungary

[2 ] Karolinska Institutet Stockholm Sweden

[3 ] Stockholm University Stockholm Sweden

[4 ] Tilburg University Tilburg Netherlands

[5 ] University of Groningen Groningen Netherlands

[6 ] Utrecht University Utrecht Netherlands

[7 ] University Medical Center Groningen, University of Groningen Groningen Netherlands

[8 ] Friesland Mental Health Care Services Leeuwarden Netherlands

[9 ] University of California Los Angeles Los Angeles United States

[10 ] National Bureau of Economic Research Cambridge United States

[11 ] University of Amsterdam Amsterdam Netherlands

[12 ] Dartmouth College Hanover United States

[13 ] University of Münster Muenster Germany

[14 ] University of Rennes, CNRS, Inria and Inserm Rennes France

[15 ] McLean Hospital Belmont United States

[16 ] Harvard Medical School Boston United States

[17 ] University of Missouri Columbia United States

[18 ] National University of Singapore Singapore Singapore

[19 ] University of New South Wales Sydney Australia

[20 ] Stockholm School of Economics Stockholm Sweden

[21 ] University of Innsbruck Innsbruck Austria

[22 ] University Hospital Basel Basel Switzerland

[23 ] Monash University Melbourne Australia

[24 ] University of Wisconsin-Madison Madison United States

[25 ] Ludwig-Maximilians-University Munich Germany

[26 ] University of Innsbruck Innsbruck Austria

[27 ] Maastricht University Maastricht Netherlands

[28 ] Princeton University Princeton United States

[29 ] University Medical Center Groningen, University of Groningen Groningen Netherlands

[30 ] University of Victoria Victoria Canada

[31 ] Université Paris-Saclay Paris France

[32 ] Neurospin, CEA Paris France

[33 ] Amsterdam University Amsterdam Netherlands

[34 ] University of Bristol Bristol United Kingdom

[35 ] Center for Open Science Charlottesville United States

[36 ] University of Virginia Charlottesville United States

[37 ] Stanford University Stanford United States

[38 ] University of Basel Basel Switzerland

[39 ] Amsterdam University Amsterdam Netherlands

[40 ] Tel Aviv University Tel Aviv Israel

[41 ] ESMT Berlin Berlin Germany

[42 ] University College London London United Kingdom

[43 ] University of Sussex Brighton United Kingdom

[44 ] University of Illinois Urbana-Champaign United States

[45 ] University of Virginia Charlottesville United States

[46 ] University of Alberta Edmonton Canada

[47 ] Lund University Lund United States

[48 ] University of Massachusetts Amherst Amherst United States

[49 ] INSEAD Singapore Singapore

eLife United Kingdom

Author notes

[†]

These authors contributed equally to this work.

Author information

Balazs Aczel https://orcid.org/0000-0001-9364-4988

Barnabas Szaszi https://orcid.org/0000-0001-7078-2712

Gustav Nilsonne https://orcid.org/0000-0001-5273-0150

Jojanneke A Bastiaansen https://orcid.org/0000-0003-4831-6402

Daniel Benjamin https://orcid.org/0000-0002-2642-5416

Udo Boehm https://orcid.org/0000-0002-8677-0721

Rotem Botvinik-Nezer https://orcid.org/0000-0003-2669-1877

Laura F Bringmann https://orcid.org/0000-0002-8091-9935

Niko A Busch https://orcid.org/0000-0003-4837-0345

Emmanuel Caruyer https://orcid.org/0000-0002-8547-7726

Andrea M Cataldo https://orcid.org/0000-0003-2787-224X

Nelson Cowan https://orcid.org/0000-0003-3711-4338

Andrew Delios https://orcid.org/0000-0002-6791-227X

Noah NN van Dongen https://orcid.org/0000-0003-0387-7388

Johnny B van Doorn https://orcid.org/0000-0003-0270-096X

Anna Dreber https://orcid.org/0000-0003-3989-9941

Gary F Egan https://orcid.org/0000-0002-3186-4026

Morton Ann Gernsbacher https://orcid.org/0000-0003-0397-3329

Rink Hoekstra https://orcid.org/0000-0002-1588-7527

Sabine Hoffmann https://orcid.org/0000-0001-6197-8801

Felix Holzmeister https://orcid.org/0000-0001-9606-0427

Juergen Huber https://orcid.org/0000-0003-0073-0321

Magnus Johannesson https://orcid.org/0000-0001-8759-6393

Michael Kirchler https://orcid.org/0000-0002-5416-2545

Jean-Francois Mangin https://orcid.org/0000-0002-1612-461X

Ben R Newell https://orcid.org/0000-0003-1898-205X

Russell A Poldrack https://orcid.org/0000-0001-6755-0259

Don van Ravenzwaaij https://orcid.org/0000-0002-5030-4091

Jörg Rieskamp https://orcid.org/0000-0003-2632-8015

Tom Schonberg https://orcid.org/0000-0002-4485-816X

Martin Schweinsberg https://orcid.org/0000-0003-3529-9463

David Shanks https://orcid.org/0000-0002-4600-6323

Samuel St-Jean https://orcid.org/0000-0002-8092-2974

Jelte Wicherts https://orcid.org/0000-0003-2415-2933

Article

Publisher ID: 72185

DOI: 10.7554/eLife.72185

PMC ID: 8626083

PubMed ID: 34751133

SO-VID: 0251402b-3a8d-4238-8a5e-4e2d712bf563

License:

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

History

Date received : 14 July 2021

Date accepted : 07 November 2021

Funding

Funded by: FundRef http://dx.doi.org/10.13039/501100003246, Netherlands Organisation for Scientific Research;

Award ID: 406-17-568

Award Recipient : Alexandra Sarafoglou

Funded by: FundRef http://dx.doi.org/10.13039/501100000038, Natural Sciences and Engineering Research Council of Canada;

Award ID: BP-546283-2020

Award Recipient : Samuel St-Jean

Funded by: FundRef http://dx.doi.org/10.13039/501100003151, Fonds de Recherche du Québec - Nature et Technologies;

Award ID: 290978

Award Recipient : Samuel St-Jean

Funded by: FundRef http://dx.doi.org/10.13039/501100000781, European Research Council;

Award ID: 726361

Award Recipient : Jelte Wicherts Olmo R van den Akker

Funded by: FundRef http://dx.doi.org/10.13039/501100000781, European Research Council;

Award ID: 681466

Award Recipient : Yoram K Kunkels

Funded by: VIDI fellowship organisation;

Award ID: 016.Vidi.188.001

Award Recipient : Don van Ravenzwaaij

Funded by: VENI fellowship grant;

Award ID: Veni 191G.037

Award Recipient : Laura F Bringmann

Funded by: FundRef http://dx.doi.org/10.13039/100000001, National Science Foundation;

Award ID: 1760052

Award Recipient : Matthew J Salganik

Funded by: FundRef http://dx.doi.org/10.13039/501100001735, Weizmann Institute of Science;

Award ID: Israel National Postdoctoral Award Program for Advancing Women in Science

Award Recipient : Rotem Botvinik-Nezer

Funded by: John Templeton Foundation, Templeton World Charity Foundation, Templeton Religion Trust, and Arnold Ventures;

Award Recipient : Brian A Nosek

Funded by: FundRef http://dx.doi.org/10.13039/100012951, Institut Européen d'Administration des Affaires;

Award Recipient : Eric Luis Uhlmann

Funded by: FundRef http://dx.doi.org/10.13039/501100000781, European Research Council;

Award ID: 640638

Award Recipient : Noah NN van Dongen

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Custom metadata

Author impact statement Using multiple analysts to independently analyze the same dataset has the potential to explore and strengthen the robustness of results and conclusions in basic and applied research.

Template 5

ScienceOpen disciplines: Life sciences

Keywords: multi-analyst,metascience,statistical practice,science forum,expert consensus,analytical variability,none

Data availability:

ScienceOpen disciplines: Life sciences

Keywords: multi-analyst, metascience, statistical practice, science forum, expert consensus, analytical variability, none

Consensus-based guidance for conducting and reporting multi-analyst studies

Read this article at

Abstract

Related collections

Remedies to the replicability crisis

Most cited references 45

The FAIR Guiding Principles for scientific data management and stewardship

Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review.

Variability in the analysis of a single neuroimaging dataset by many teams

Author and article information

Contributors

Journal

Affiliations

Author notes

Author information

Article

History

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 525

Cited by 16