51
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: not found
      • Article: not found

      Using prediction markets to estimate the reproducibility of scientific research

      Read this article at

      ScienceOpenPublisherPMC
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.

          Related collections

          Most cited references55

          • Record: found
          • Abstract: not found
          • Article: not found

          Scientific method: statistical errors.

            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Social identity contingencies: how diversity cues signal threat or safety for African Americans in mainstream institutions.

            This research demonstrates that people at risk of devaluation based on group membership are attuned to cues that signal social identity contingencies--judgments, stereotypes, opportunities, restrictions, and treatments that are tied to one's social identity in a given setting. In 3 experiments, African American professionals were attuned to minority representation and diversity philosophy cues when they were presented as a part of workplace settings. Low minority representation cues coupled with colorblindness (as opposed to valuing diversity) led African American professionals to perceive threatening identity contingencies and to distrust the setting (Experiment 1). The authors then verified that the mechanism mediating the effect of setting cues on trust was identity contingent evaluations (Experiments 2 & 3). The power of social identity contingencies as they relate to underrepresented groups in mainstream institutions is discussed. (c) 2008 APA, all rights reserved.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention.

              Recent systematic reviews and empirical evaluations of the cognitive sciences literature suggest that publication and other reporting biases are prevalent across diverse domains of cognitive science. In this review, we summarize the various forms of publication and reporting biases and other questionable research practices, and overview the available methods for probing into their existence. We discuss the available empirical evidence for the presence of such biases across the neuroimaging, animal, other preclinical, psychological, clinical trials, and genetics literature in the cognitive sciences. We also highlight emerging solutions (from study design to data analyses and reporting) to prevent bias and improve the fidelity in the field of cognitive science research. Copyright © 2014 Elsevier Ltd. All rights reserved.
                Bookmark

                Author and article information

                Journal
                Proceedings of the National Academy of Sciences
                Proc Natl Acad Sci USA
                Proceedings of the National Academy of Sciences
                0027-8424
                1091-6490
                December 15 2015
                December 15 2015
                December 15 2015
                November 09 2015
                : 112
                : 50
                : 15343-15347
                Article
                10.1073/pnas.1516179112
                4687569
                26553988
                f788c6e6-85bf-4e28-a696-3f1c471f5ba4
                © 2015
                History

                Comments

                Comment on this article