Blog
About

  • Record: found
  • Abstract: found
  • Article: found
Is Open Access

Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial

Read this article at

Bookmark
      There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

      Abstract

      Objectives To see whether telling peer reviewers that their signed reviews of original research papers might be posted on the BMJ’s website would affect the quality of their reviews.Design Randomised controlled trial.Setting A large international general medical journal based in the United Kingdom.Participants 541 authors, 471 peer reviewers, and 12 editors.Intervention Consecutive eligible papers were randomised either to have the reviewer’s signed report made available on the BMJ’s website alongside the published paper (intervention group) or to have the report made available only to the author—the BMJ’s normal procedure (control group). The intervention was the act of revealing to reviewers—after they had agreed to review but before they undertook their review—that their signed report might appear on the website. Main outcome measures The main outcome measure was the quality of the reviews, as independently rated on a scale of 1 to 5 using a validated instrument by two editors and the corresponding author. Authors and editors were blind to the intervention group. Authors rated review quality before the fate of their paper had been decided. Additional outcomes were the time taken to complete the review and the reviewer’s recommendation regarding publication.Results 558 manuscripts were randomised, and 471 manuscripts remained after exclusions. Of the 1039 reviewers approached to take part in the study, 568 (55%) declined. Two editors’ evaluations of the quality of the peer review were obtained for all 471 manuscripts, with the corresponding author’s evaluation obtained for 453. There was no significant difference in review quality between the intervention and control groups (mean difference for editors 0.04, 95% CI −0.09 to 0.17; for authors 0.06, 95% CI −0.09 to 0.20). Any possible difference in favour of the control group was well below the level regarded as editorially significant. Reviewers in the intervention group took significantly longer to review (mean difference 25 minutes, 95% CI 3.0 to 47.0 minutes).Conclusion Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.

      Related collections

      Most cited references 11

      • Record: found
      • Abstract: found
      • Article: not found

      Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial.

      To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention. The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review. Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review. Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.
        Bookmark
        • Record: found
        • Abstract: found
        • Article: not found

        Open peer review: a randomised controlled trial.

        Most scientific journals practise anonymous peer review. There is no evidence, however, that this is any better than an open system. To evaluate the feasibility of an open peer review system. Reviewers for the British Journal of Psychiatry were asked whether they would agree to have their name revealed to the authors whose papers they review; 408 manuscripts assigned to reviewers who agreed were randomised to signed or unsigned groups. We measured review quality, tone, recommendation for publication and time taken to complete each review. A total of 245 reviewers (76%) agreed to sign. Signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews. Reviewers who signed were more likely to recommend publication. This study supports the feasibility of an open peer review system and identifies such a system's potential drawbacks.
          Bookmark
          • Record: found
          • Abstract: found
          • Article: not found

          The effects of blinding on the quality of peer review. A randomized trial.

          Peer reviewers are blinded sometimes to authors' and institutions' names, but the effects of blinding on review quality are not known. We, therefore, conducted a randomized trial of blinded peer review. Each of 127 consecutive manuscripts of original research that were submitted to the Journal of General Internal Medicine were sent to two external reviewers, one of whom was randomly selected to receive a manuscript with the authors' and institutions' names removed. Reviewers were asked, but not required, to sign their reviews. Blinding was successful for 73% of reviewers. Quality of reviews was higher for the blinded manuscripts (3.5 vs 3.1 on a 5-point scale). Forty-three percent of reviewers signed their reviews, and blinding did not affect the proportion who signed. There was no association between signing and quality. Our study shows that, in our setting, blinding improves the quality of reviews and that research on the effects of peer review is possible.
            Bookmark

            Author and article information

            Affiliations
            [1 ]BMJ, BMA House, London, UK
            [2 ]London School of Hygiene and Tropical Medicine, London, UK
            [3 ]SVR died in February 2005
            Author notes
            Correspondence to: T Delamothe tdelamothe@ 123456bmj.com
            Contributors
            Role: research assistant
            Role: deputy editor
            Role: professor of pharmacoepidemiology
            Journal
            BMJ
            bmj
            BMJ : British Medical Journal
            BMJ Publishing Group Ltd.
            0959-8138
            1468-5833
            2010
            2010
            16 November 2010
            : 341
            2982798
            21081600
            roos644633
            10.1136/bmj.c5729
            © van Rooyen et al 2010

            This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

            Product
            Categories
            Research
            Journalology
            Clinical Trials (Epidemiology)

            Medicine

            Comments

            Comment on this article