1,145
views
2
recommends
+1 Recommend
2 collections
    12
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Open Evaluation: A Vision for Entirely Transparent Post-Publication Peer Review and Rating for Science

        1

      Frontiers in Computational Neuroscience

      Frontiers Media S.A.

      peer review, publishing, ratings, social web, open evaluation

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          The two major functions of a scientific publishing system are to provide access to and evaluation of scientific papers. While open access (OA) is becoming a reality, open evaluation (OE), the other side of the coin, has received less attention. Evaluation steers the attention of the scientific community and thus the very course of science. It also influences the use of scientific findings in public policy. The current system of scientific publishing provides only journal prestige as an indication of the quality of new papers and relies on a non-transparent and noisy pre-publication peer-review process, which delays publication by many months on average. Here I propose an OE system, in which papers are evaluated post-publication in an ongoing fashion by means of open peer review and rating. Through signed ratings and reviews, scientists steer the attention of their field and build their reputation. Reviewers are motivated to be objective, because low-quality or self-serving signed evaluations will negatively impact their reputation. A core feature of this proposal is a division of powers between the accumulation of evaluative evidence and the analysis of this evidence by paper evaluation functions (PEFs). PEFs can be freely defined by individuals or groups (e.g., scientific societies) and provide a plurality of perspectives on the scientific literature. Simple PEFs will use averages of ratings, weighting reviewers (e.g., by H-index), and rating scales (e.g., by relevance to a decision process) in different ways. Complex PEFs will use advanced statistical techniques to infer the quality of a paper. Papers with initially promising ratings will be more deeply evaluated. The continual refinement of PEFs in response to attempts by individuals to influence evaluations in their own favor will make the system ungameable. OA and OE together have the power to revolutionize scientific publishing and usher in a new culture of transparency, constructive criticism, and collaboration.

          Related collections

          Most cited references 25

          • Record: found
          • Abstract: found
          • Article: found
          Is Open Access

          Article-Level Metrics and the Evolution of Scientific Impact

          The authors discuss the value of article-level metrics in determining an article's scientific impact.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial.

            To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention. The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review. Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review. Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: found
              Is Open Access

              Open Peer Review by a Selected-Papers Network

              A selected-papers (SP) network is a network in which researchers who read, write, and review articles subscribe to each other based on common interests. Instead of reviewing a manuscript in secret for the Editor of a journal, each reviewer simply publishes his review (typically of a paper he wishes to recommend) to his SP network subscribers. Once the SP network reviewers complete their review decisions, the authors can invite any journal editor they want to consider these reviews and initial audience size, and make a publication decision. Since all impact assessment, reviews, and revisions are complete, this decision process should be short. I show how the SP network can provide a new way of measuring impact, catalyze the emergence of new subfields, and accelerate discovery in existing fields, by providing each reader a fine-grained filter for high-impact. I present a three phase plan for building a basic SP network, and making it an effective peer review platform that can be used by journals, conferences, users of repositories such as arXiv, and users of search engines such as PubMed. I show how the SP network can greatly improve review and dissemination of research articles in areas that are not well-supported by existing journals. Finally, I illustrate how the SP network concept can work well with existing publication services such as journals, conferences, arXiv, PubMed, and online citation management sites.
                Bookmark

                Author and article information

                Journal
                Front Comput Neurosci
                Front Comput Neurosci
                Front. Comput. Neurosci.
                Frontiers in Computational Neuroscience
                Frontiers Media S.A.
                1662-5188
                22 June 2012
                17 October 2012
                2012
                : 6
                Affiliations
                1Medical Research Council, Cognition and Brain Sciences Unit Cambridge, UK
                Author notes

                Edited by: Diana Deca, Technische Universität München, Germany

                Reviewed by: Satrajit S. Ghosh, Massachusetts Institute of Technology, USA; Razvan V. Florian, Romanian Institute of Science and Technology, Romania

                *Correspondence: Nikolaus Kriegeskorte, Medical Research Council, Cognition and Brain Sciences Unit, 15 Chaucer Road, Cambridge CB2 7EF, UK. e-mail: nikolaus.kriegeskorte@ 123456mrc-cbu.cam.ac.uk ; www.mrc-cbu.cam.ac.uk/people/nikolaus.kriegeskorte/
                Article
                10.3389/fncom.2012.00079
                3473231
                23087639
                Copyright © 2012 Kriegeskorte.

                This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.

                Page count
                Figures: 10, Tables: 0, Equations: 0, References: 38, Pages: 18, Words: 14933
                Categories
                Neuroscience
                Methods Article

                Neurosciences

                publishing, peer review, social web, ratings, open evaluation

                Comments

                Comment on this article