270
views
1
recommends
+1 Recommend
4 collections
    16
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations

      research-article
      1 , * , 2
      PLoS Biology
      Public Library of Science

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Because both subjective post-publication review and the number of citations are highly error prone and biased measures of merit of scientific papers, journal-based metrics may be a better surrogate.

          Abstract

          The assessment of scientific publications is an integral part of the scientific process. Here we investigate three methods of assessing the merit of a scientific paper: subjective post-publication peer review, the number of citations gained by a paper, and the impact factor of the journal in which the article was published. We investigate these methods using two datasets in which subjective post-publication assessments of scientific publications have been made by experts. We find that there are moderate, but statistically significant, correlations between assessor scores, when two assessors have rated the same paper, and between assessor score and the number of citations a paper accrues. However, we show that assessor score depends strongly on the journal in which the paper is published, and that assessors tend to over-rate papers published in journals with high impact factors. If we control for this bias, we find that the correlation between assessor scores and between assessor score and the number of citations is weak, suggesting that scientists have little ability to judge either the intrinsic merit of a paper or its likely impact. We also show that the number of citations a paper receives is an extremely error-prone measure of scientific merit. Finally, we argue that the impact factor is likely to be a poor measure of merit, since it depends on subjective assessment. We conclude that the three measures of scientific merit considered here are poor; in particular subjective assessments are an error-prone, biased, and expensive method by which to assess merit. We argue that the impact factor may be the most satisfactory of the methods we have considered, since it is a form of pre-publication review. However, we emphasise that it is likely to be a very error-prone measure of merit that is qualitative, not quantitative.

          Related collections

          Most cited references7

          • Record: found
          • Abstract: not found
          • Article: not found

          Why the impact factor of journals should not be used for evaluating research.

          P O Seglen (1997)
            Bookmark
            • Record: found
            • Abstract: found
            • Article: found
            Is Open Access

            A Principal Component Analysis of 39 Scientific Impact Measures

            Background The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. Methodology We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Conclusions Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              The Impact Factor Game

              The PLoS Medicine editors argue that we need a better measure than the impact factor for assessing the biomedical literature.
                Bookmark

                Author and article information

                Contributors
                Role: Academic Editor
                Journal
                PLoS Biol
                PLoS Biol
                plos
                plosbiol
                PLoS Biology
                Public Library of Science (San Francisco, USA )
                1544-9173
                1545-7885
                October 2013
                October 2013
                8 October 2013
                : 11
                : 10
                : e1001675
                Affiliations
                [1 ]School of Life Sciences, University of Sussex, Brighton, United Kingdom
                [2 ]Hannover, Germany
                University of California Davis, United States of America
                Author notes

                The authors have declared that no competing interests exist.

                The author(s) have made the following declarations about their contributions: Conceived and designed the experiments: AEW NS. Performed the experiments: AEW NS. Analyzed the data: AEW NS. Contributed reagents/materials/analysis tools: AEW NS. Wrote the paper: AEW NS.

                Article
                PBIOLOGY-D-13-00183
                10.1371/journal.pbio.1001675
                3792863
                24115908
                c782a255-931e-4179-b855-225e8c63b875
                Copyright @ 2013

                This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

                History
                : 15 January 2013
                : 26 August 2013
                Page count
                Pages: 8
                Funding
                This work was supported by the salary paid to AEW. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
                Categories
                Research Article

                Life sciences
                Life sciences

                Comments

                Comment on this article