Blog
About


  • Record: found
  • Abstract: found
  • Article: found
Is Open Access

Peer review – issues, limitations, and future development

Jan Velterop1,*

ScienceOpen Research – Section: SOR-EDU

ScienceOpen

10.14293/S2199-1006.1.SOR-EDU.AYXIPS.v1

Assessment, Evaluation & Research methods, Peer review, Scientific publishing, Open access

Read Bookmark
There is no summary for this article yet

Abstract

Peer review is almost universally seen as the crux of scientific journal publishing. The role of peer reviewers is (1) to help avoid unnecessary errors in the published article, and (2) to judge publication-worthiness (in the journal that arranges for the review). This happens. Sometimes. But the notion of peer review is rather vague, and since most of it is anonymous, it is very difficult – arguably impossible – for researchers to know if the articles they read have been reliably peer reviewed and which criteria have been used to come to the decision to accept for publication. On top of that, peer review is very expensive. Not the peer review itself, as it is mostly done by researchers without being paid for it, but the process as arranged by publishers. This has several underlying causes, but it is clear that the actual cost of technically publishing an article is but a fraction of the average APC (Article Processing Charge) income or per-article subscription revenues publishers routinely realize. Some (e.g. Richard Smith, ex-Editor of the British Medical Journal) advocate abolishing peer review altogether. This is certainly not without merit, but even without abolishing it, there are ways to make peer review more reliable and transparent, and much cheaper to the scientific community.

Related collections

Most cited references 4

  • Record: found
  • Abstract: found
  • Article: found
Is Open Access

A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants

Background This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen's Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta-regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if the information was not conveyed. Conclusions/Significance Studies that report a high level of IRR are to be considered less credible than those with a low level of IRR. According to our meta-analysis the IRR of peer assessments is quite limited and needs improvement (e.g., reader system).
  • Record: found
  • Abstract: not found
  • Article: not found

The outflow of academic papers from China: why is it happening and can it be stemmed?

  • Record: found
  • Abstract: not found
  • Article: not found

The pain of rejection

Author and article information

Affiliations
[1]Independent Scholar, Guildford, Surrey, UK
Author notes
[*]Corresponding author's e-mail address: velterop@123456gmail.com
Contributors
Journal
SOR-EDU
ScienceOpen Research
ScienceOpen
2199-1006
29 September 2015
: 0 (ID: 1dcfbe69-c30c-4eaa-a003-948c9700da40)
: 0
: 1-5
© 2015 J. Velterop.

This work has been published open access under Creative Commons Attribution License CC BY 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Conditions, terms of use and publishing policy can be found at www.scienceopen.com.

Counts
Figures: 0, Tables: 0, References: 16, Pages: 5
Product
Categories
Original article
ScienceOpen disciplines:
Keywords:

Comments

Comment on this article