Blog
About

  • Record: found
  • Abstract: found
  • Article: found
Is Open Access

The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations

1 , * , 2

PLoS Biology

Public Library of Science

Read this article at

Bookmark
      There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

      Abstract

      Because both subjective post-publication review and the number of citations are highly error prone and biased measures of merit of scientific papers, journal-based metrics may be a better surrogate.

      Abstract

      The assessment of scientific publications is an integral part of the scientific process. Here we investigate three methods of assessing the merit of a scientific paper: subjective post-publication peer review, the number of citations gained by a paper, and the impact factor of the journal in which the article was published. We investigate these methods using two datasets in which subjective post-publication assessments of scientific publications have been made by experts. We find that there are moderate, but statistically significant, correlations between assessor scores, when two assessors have rated the same paper, and between assessor score and the number of citations a paper accrues. However, we show that assessor score depends strongly on the journal in which the paper is published, and that assessors tend to over-rate papers published in journals with high impact factors. If we control for this bias, we find that the correlation between assessor scores and between assessor score and the number of citations is weak, suggesting that scientists have little ability to judge either the intrinsic merit of a paper or its likely impact. We also show that the number of citations a paper receives is an extremely error-prone measure of scientific merit. Finally, we argue that the impact factor is likely to be a poor measure of merit, since it depends on subjective assessment. We conclude that the three measures of scientific merit considered here are poor; in particular subjective assessments are an error-prone, biased, and expensive method by which to assess merit. We argue that the impact factor may be the most satisfactory of the methods we have considered, since it is a form of pre-publication review. However, we emphasise that it is likely to be a very error-prone measure of merit that is qualitative, not quantitative.

      Related collections

      Most cited references 7

      • Record: found
      • Abstract: not found
      • Article: not found

      Why the impact factor of journals should not be used for evaluating research.

       Per Seglen (1997)
        Bookmark
        • Record: found
        • Abstract: found
        • Article: not found

        The Impact Factor Game

        (2006)
        The PLoS Medicine editors argue that we need a better measure than the impact factor for assessing the biomedical literature.
          Bookmark
          • Record: found
          • Abstract: found
          • Article: found
          Is Open Access

          A Principal Component Analysis of 39 Scientific Impact Measures

          Background The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. Methodology We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Conclusions Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.
            Bookmark

            Author and article information

            Affiliations
            [1 ]School of Life Sciences, University of Sussex, Brighton, United Kingdom
            [2 ]Hannover, Germany
            University of California Davis, United States of America
            Author notes

            The authors have declared that no competing interests exist.

            The author(s) have made the following declarations about their contributions: Conceived and designed the experiments: AEW NS. Performed the experiments: AEW NS. Analyzed the data: AEW NS. Contributed reagents/materials/analysis tools: AEW NS. Wrote the paper: AEW NS.

            Contributors
            Role: Academic Editor
            Journal
            PLoS Biol
            PLoS Biol
            plos
            plosbiol
            PLoS Biology
            Public Library of Science (San Francisco, USA )
            1544-9173
            1545-7885
            October 2013
            October 2013
            8 October 2013
            : 11
            : 10
            3792863
            PBIOLOGY-D-13-00183
            10.1371/journal.pbio.1001675
            (Academic Editor)

            This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

            Counts
            Pages: 8
            Funding
            This work was supported by the salary paid to AEW. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
            Categories
            Research Article

            Life sciences

            Comments

            Comment on this article