Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trialBMJ 2010; 341 doi: https://doi.org/10.1136/bmj.c5729 (Published 16 November 2010) Cite this as: BMJ 2010;341:c5729
- Susan van Rooyen, research assistant13,
- Tony Delamothe, deputy editor1,
- Stephen J W Evans, professor of pharmacoepidemiology2
- 1BMJ, BMA House, London, UK
- 2London School of Hygiene and Tropical Medicine, London, UK
- 3SVR died in February 2005
- Correspondence to: T Delamothe
- Accepted 28 August 2010
Objectives To see whether telling peer reviewers that their signed reviews of original research papers might be posted on the BMJ’s website would affect the quality of their reviews.
Design Randomised controlled trial.
Setting A large international general medical journal based in the United Kingdom.
Participants 541 authors, 471 peer reviewers, and 12 editors.
Intervention Consecutive eligible papers were randomised either to have the reviewer’s signed report made available on the BMJ’s website alongside the published paper (intervention group) or to have the report made available only to the author—the BMJ’s normal procedure (control group). The intervention was the act of revealing to reviewers—after they had agreed to review but before they undertook their review—that their signed report might appear on the website.
Main outcome measures The main outcome measure was the quality of the reviews, as independently rated on a scale of 1 to 5 using a validated instrument by two editors and the corresponding author. Authors and editors were blind to the intervention group. Authors rated review quality before the fate of their paper had been decided. Additional outcomes were the time taken to complete the review and the reviewer’s recommendation regarding publication.
Results 558 manuscripts were randomised, and 471 manuscripts remained after exclusions. Of the 1039 reviewers approached to take part in the study, 568 (55%) declined. Two editors’ evaluations of the quality of the peer review were obtained for all 471 manuscripts, with the corresponding author’s evaluation obtained for 453. There was no significant difference in review quality between the intervention and control groups (mean difference for editors 0.04, 95% CI −0.09 to 0.17; for authors 0.06, 95% CI −0.09 to 0.20). Any possible difference in favour of the control group was well below the level regarded as editorially significant. Reviewers in the intervention group took significantly longer to review (mean difference 25 minutes, 95% CI 3.0 to 47.0 minutes).
Conclusion Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.
This paper has been severely delayed by the sad death of the first author, Susan van Rooyen. Because members of the BMJ’s editorial staff conducted this research, assessment and peer review were carried out entirely by external advisers. No member of BMJ staff was involved in making the decision on the paper. The researchers would like to acknowledge the following people whose help was invaluable during this study: Lisa Bero, for suggesting the study and producing a draft protocol; the members of the research steering group—Richard Smith, Nick Black, Fiona Godlee, Sandra Goldbeck-Wood, and Liz Crossan—for involvement in the conception and design of the study; Richard Smith, then editor of the BMJ, for making it possible to carry out this study in the BMJ offices, for his participation in evaluating reviews, his comments regarding the preparation of the manuscript, and for heading up the BMJ’s research steering group; Maureen Phayer and Daniel Berhane, for their invaluable technical assistance in posting articles’ prepublication history on the website; Sue Minns and Marita Batten, manuscript administrators; Anna Vassilieva, for administrative assistance in data collection and management; BMJ editors—Kamran Abbasi, Douglas Carnall, Sandra Goldbeck-Wood, Trish Groves, Christopher Martyn, Tessa Richards, Roger Robinson, Jane Smith, Tony Smith, Alison Tonks, and Gavin Yamey—for evaluating reviews; the authors who allowed us to use their papers and evaluated the reviews; and all the BMJ reviewers who allowed themselves to be research subjects.
Contributors: SvR was involved in the conception and design of the study; data collection, analysis, and interpretation; day-to-day management of the study; writing the paper; and discussing core ideas, and was a member of the research steering group. TD was involved in the conception and design of the study; interpretation of data; writing the paper; and discussing core ideas, and is the guarantor. SJWE gave statistical advice, was involved in the analysis of the data, and assisted in the writing of the paper.
Funding: SvR was employed by the BMJ as research assistant. The other costs of the study were met by the BMJ.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; TD is employed by the BMJ as deputy editor and the journal is interested in the outcome of this research; no other relationships or activities that could appear to have influenced the submitted work.
Data sharing: The main dataset, in Stata format, and a short Stata programme to reproduce some of the analyses are available from SJWE (stephen.evans[at]lshtm.ac.uk). The data could be converted to other formats if required. Because of the death of SvR, the data dictionary is not available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.