Blog
About

6,967
views
4
recommends
+1 Recommend
2 collections
    47
    shares
    • Record: found
    • Abstract: found
    • Article: found
    Is Open Access

    Peer review – issues, limitations, and future development

     1,*

    ScienceOpen Research – Section: SOR-EDU

    ScienceOpen

    Assessment, Evaluation & Research methods, Peer review, Scientific publishing, Open access

    Read this article at

    Bookmark
        There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

        Abstract

        Peer review is almost universally seen as the crux of scientific journal publishing. The role of peer reviewers is (1) to help avoid unnecessary errors in the published article, and (2) to judge publication-worthiness (in the journal that arranges for the review). This happens. Sometimes. But the notion of peer review is rather vague, and since most of it is anonymous, it is very difficult – arguably impossible – for researchers to know if the articles they read have been reliably peer reviewed and which criteria have been used to come to the decision to accept for publication. On top of that, peer review is very expensive. Not the peer review itself, as it is mostly done by researchers without being paid for it, but the process as arranged by publishers. This has several underlying causes, but it is clear that the actual cost of technically publishing an article is but a fraction of the average APC (Article Processing Charge) income or per-article subscription revenues publishers routinely realize. Some (e.g. Richard Smith, ex-Editor of the British Medical Journal) advocate abolishing peer review altogether. This is certainly not without merit, but even without abolishing it, there are ways to make peer review more reliable and transparent, and much cheaper to the scientific community.

        Related collections

        Most cited references 4

        • Record: found
        • Abstract: found
        • Article: found
        Is Open Access

        A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants

        Background This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. Conclusions/Significance Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
          Bookmark
          • Record: found
          • Abstract: not found
          • Article: not found

          The outflow of academic papers from China: why is it happening and can it be stemmed?

            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Conservative attitudes to old-established organs: Oliver Lodge andPhilosophical Magazine

              Bookmark

              Author and article information

              Affiliations
              [1]Independent Scholar, Guildford, Surrey, UK
              Author notes
              [*]Corresponding author's e-mail address: velterop@123456gmail.com
              Contributors
              Journal
              SOR-EDU
              ScienceOpen Research
              ScienceOpen
              2199-1006
              29 September 2015
              : 0 (ID: 1dcfbe69-c30c-4eaa-a003-948c9700da40)
              : 0
              : 1-5
              3101:XE
              10.14293/S2199-1006.1.SOR-EDU.AYXIPS.v1
              © 2015 J. Velterop.

              This work has been published open access under Creative Commons Attribution License CC BY 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Conditions, terms of use and publishing policy can be found at www.scienceopen.com.

              Counts
              Figures: 0, Tables: 0, References: 16, Pages: 5
              Product
              Categories
              Original article
              ScienceOpen disciplines:
              Keywords:

              Comments

              As ScienceOpen has since decided not to offer primary publishing services any longer, the last section in my article is not relevant anymore. I would like to replace it by the following:

              PEER REVIEW BY ENDORSEMENT 

              The principle is quite straightforward, really. Authors themselves invite at least two peers to review their paper (according to some rules to avoid nepotism and friend-bias, such as peer-endorsers having to be active researchers, and not be, or for at least five years have been, at the same institution as, or a co-author of, any of the authors), and when these peers endorse its publication – most likely after some iteration with the authors to clarify questions that arise – they do so openly, fully disclosing their identities. And perhaps also giving a motivation as to why they endorse publication, a motivation which may, of course, be along the lines of what Alan Singleton reported, or of the comments of Oliver Lodge (see above). Many publishers ask authors who should be invited to review their papers anyway, so authors inviting them by themselves would be a relatively small step. Peer review can only operate on trust anyway. It is the openness of peer review that builds trust; not the party who invites the review. 

              Such a peer-review-by-endorsement system is likely to be at least as good as, and quite probably better than, the currently widespread “black box” of anonymous peer review. As reviews/endorsements would be signed and non-anonymous, there is very little danger of sub-standard articles being published, as endorsers/reviewers would not want to put their reputations at risk. Peer Review by Endorsement should be offered as an option for authors. Peer Review by Endorsement would occur, just as usual peer review, before publication, and would be entirely open and transparent. 

              PREPRINTS AND PEER REVIEW BY ENDORSEMENT

              In fact, Peer Review by Endorsement is advisable for preprints as well. Articles published as preprints would, of course, subsequently be available for Post-Publication Peer Review. It’s just that the quality of the initial publication as preprint is likely to be better than without. In the case of preprints, which are inherently less formal than publications in a journal, Peer Review by Endorsement can of course be less formal, too, and given that, these reviews do not necessarily have to be open or subject to the rules to avoid friend-bias or colleague-bias, and the like. They could be anonymous, especially if all they do is help the authors avoid mistakes, ambiguities, language errors and the like. However, if they are open and not anonymous, they are likely to increase the preprint’s quality perception. A preprint’s quality perception may indeed encourage more comments and post-publication reviews.

               

              J Velterop

              15 August 2018

              2018-08-15 10:52 UTC
              +1

              Comment on this article