Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial

BMJ 1999;318:23-27 ( 2 January )

Papers

Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial

Editorial by Smith Education and debate p 44

Susan van Rooyen, research assistant, ^a Fiona Godlee, assistant editor, ^a Stephen Evans, visiting professor of medical statistics, ^b Nick Black, professor of health services research, ^b Richard Smith, editor. ^a

^a BMJ Editorial, BMA House, London WC1H 9JR, ^b London School of Hygiene and Tropical Medicine, London WC1E 7HT

Correspondence to: Mrs van Rooyen svanrooyen{at}bmj.com

Objectives: To examine the effect on peer review of askingreviewers to have their identity revealed to the authors of thepaper.
Design: Randomised trial. Consecutive eligible paperswere sent to two reviewers who were randomised to have their identityrevealed to the authors or to remain anonymous. Editors and authorswere blind to theintervention.
Main outcome measures: The quality of the reviews was independentlyrated by two editors and the corresponding author using a validatedinstrument. Additional outcomes were the time taken to completethe review and the recommendation regarding publication. A questionnairesurvey was undertaken of the authors of a cohort of manuscriptssubmitted for publication to find out their views on open peerreview.
Results: Two editors' assessments were obtained for113 out of 125 manuscripts, and the corresponding author's assessmentwas obtained for 105. Reviewers randomised to be asked to be identifiedwere 12% (95% confidence interval 0.2% to 24%) more likely todecline to review than reviewers randomised to remain anonymous(35% v 23%). There was no significant difference in quality (scoredon a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72))and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidenceinterval for difference $-$ 0.19 to 0.12), and no significant differencein the recommendation regarding publication or time taken to reviewthe paper. The editors' quality score for reviews (3.05 (SD 0.70))was significantly higher than that of authors (2.90 (0.87)) (P<0.005,95%confidence interval for difference $-$ 0.26 to $-$ 0.03). Most authorswere in favour of open peerreview.
Conclusions: Asking reviewers to consent to being identifiedto the author had no important effect on the quality of the review,the recommendation regarding publication, or the time taken toreview, but it significantly increased the likelihood of reviewersdeclining toreview.

Key messages

Arguments in favour of open peer review include increased accountability, fairness, and transparency.
Preliminary evidence suggests that open peer review leads to better quality reviews
We conducted a randomised controlled trial to examine the feasibility and impact of asking BMJ reviewers to sign their reviews
There were no differences in the quality of reviews between those who were randomised to be identified and those who were not
Most reviewers agreed to be identified to authors, and most of the authors surveyed were in favour of open peer review

© BMJ 1999

CiteULike

Complore

Connotea

Del.icio.us Add to Digg

Digg

Technorati What's this?

Relevant Articles

Evidence based publishing: Leanne Tite and Sara Schroter
BMJ 2006 333: 366. [Extract] [Full Text] [PDF]

Open peer review is feasible and does not reduce quality of reviews
BMJ 1999 318: 0. [Full Text]

Opening up BMJ peer review: Richard Smith
BMJ 1999 318: 4-5. [Extract] [Full Text] [PDF]

Evidence on peer review $---$ scientific quality control or smokescreen?: Sandra Goldbeck-Wood
BMJ 1999 318: 44-45. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

Diaz-Puente, J. M., Cazorla, A., Dorrego, A. (2007). Crossing National, Continental, and Linguistic Boundaries: Toward a Worldwide Evaluation Research Community in Journals of Evaluation. American Journal of Evaluation 28: 399-415 [Abstract]
Carpenter, J. R., Kenward, M. G., White, I. R. (2007). Sensitivity analysis after multiple imputation under missing at random: a weighting approach. Stat Methods Med Res 16: 259-275 [Abstract]
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, D., McGowan, S., Polter, A., Qadri, Y., Sarfare, S., Schultz, K., Splittgerber, R., Stephenson, J., Tower, C., Walton, R. G., Zotov, A. (2007). The ups and downs of peer review. Adv. Physiol. Educ. 31: 145-152 [Abstract] [Full text]
Pless, I B (2007). Unfinished business. Inj. Prev. 13: 145-146 [Full text]
White, I. R, Carpenter, J., Evans, S., Schroter, S. (2007). Eliciting and using expert opinions about dropout bias in randomized controlled trials. Clin Trials 4: 125-139 [Abstract]
Miller, D. R., Donati, F. (2007). Peer review policies and the Canadian Journal of Anesthesia: an update for authors and readers/La politique d'evaluation par des pairs du Journal Canadien d'Anesthesie: une mise a jour destinee aux auteurs et aux lecteurs. Canadian J. Anesthesia 54: 1-8 [Full text]
Landkroon, A. P., Euser, A. M., Veeken, H., Hart, W., Overbeke, A. J. P. M. (2006). Quality assessment of reviewers' reports using a simple instrument.. Obstet Gynecol 108: 979-985 [Abstract] [Full text]
Tite, L., Schroter, S. (2006). Evidence based publishing. BMJ 333: 366-366 [Full text]
Lange, J. H. (2006). Tradition Dies Hard. Indoor and Built Environment 15: 301-303
Smith, R. (2006). Peer review: a flawed process at the heart of science and journals.. JRSM 99: 178-182 [Full text]
Millett, D. (2006). Training the reviewer?. J. Orthod. 33: 1-2 [Full text]
Allebeck, P. (2006). Necessary steps for a modern scientific journal.. Eur J Public Health 16: 1-1 [Full text]
Schroter, S., Tite, L., Hutchings, A., Black, N. (2006). Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors or by Editors. JAMA 295: 314-317 [Abstract] [Full text]
Savulescu, J, Viens, A M (2005). What makes the best medical ethics journal? A North American perspective. J. Med. Ethics 31: 591-597 [Abstract] [Full text]
Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., Smith, R. (2004). Effects of training on quality of peer review: randomised controlled trial. BMJ 328: 673- [Abstract] [Full text]
Godlee, F. (2002). Making Reviewers Visible: Openness, Accountability, and Credit. JAMA 287: 2762-2765 [Abstract] [Full text]
Jefferson, T., Alderson, P., Wager, E., Davidoff, F. (2002). Effects of Editorial Peer Review: A Systematic Review. JAMA 287: 2784-2786 [Abstract] [Full text]
Jefferson, T., Wager, E., Davidoff, F. (2002). Measuring the Quality of Editorial Peer Review. JAMA 287: 2786-2790 [Abstract] [Full text]
Bacchetti, P. (2002). Peer review of statistics in medical research: the other problem. BMJ 324: 1271-1273 [Full text]
Dick, A D (2002). Are we on a voyage to the unknown?. Br. J. Ophthalmol. 86: 253-254 [Full text]
WALSH, E., ROONEY, M., APPLEBY, L., WILKINSON, G. (2000). Open peer review: a randomised controlled trial. Br. J. Psychiatry 176: 47-51 [Abstract] [Full text]
Smith, R. (1999). Opening up BMJ peer review. BMJ 318: 4-5 [Full text]