BMJ 1999;318:23-27 ( 2 January )

Papers

Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial

Editorial by Smith Education and debate p   44

Susan van Rooyen, research assistant, a Fiona Godlee, assistant editor, a Stephen Evans, visiting professor of medical statistics, b Nick Black, professor of health services research, b Richard Smith, editor. a

a BMJ Editorial, BMA House, London WC1H 9JR, b London School of Hygiene and Tropical Medicine, London WC1E 7HT

Correspondence to: Mrs van Rooyen svanrooyen{at}bmj.com

Objectives: To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper.
Design: Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention.
Main outcome measures: The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
Results: Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference -0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference -0.26 to -0.03). Most authors were in favour of open peer review.
Conclusions: Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.

Key messages

  • Arguments in favour of open peer review include increased accountability, fairness, and transparency.

  • Preliminary evidence suggests that open peer review leads to better quality reviews

  • We conducted a randomised controlled trial to examine the feasibility and impact of asking BMJ reviewers to sign their reviews

  • There were no differences in the quality of reviews between those who were randomised to be identified and those who were not

  • Most reviewers agreed to be identified to authors, and most of the authors surveyed were in favour of open peer review




© BMJ 1999
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to StumbleUpon StumbleUpon   Add to Technorati Technorati    What's this?

Relevant Articles

Evidence based publishing
Leanne Tite and Sara Schroter
BMJ 2006 333: 366. [Extract] [Full Text] [PDF]

Open peer review is feasible and does not reduce quality of reviews
BMJ 1999 318: 0. [Full Text]

Opening up BMJ peer review
Richard Smith
BMJ 1999 318: 4-5. [Extract] [Full Text] [PDF]

Evidence on peer review---scientific quality control or smokescreen?
Sandra Goldbeck-Wood
BMJ 1999 318: 44-45. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

  • Diaz-Puente, J. M., Cazorla, A., Dorrego, A. (2007). Crossing National, Continental, and Linguistic Boundaries: Toward a Worldwide Evaluation Research Community in Journals of Evaluation. American Journal of Evaluation 28: 399-415 [Abstract]  
  • Carpenter, J. R., Kenward, M. G., White, I. R. (2007). Sensitivity analysis after multiple imputation under missing at random: a weighting approach. Stat Methods Med Res 16: 259-275 [Abstract]  
  • Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., Mans, R., Mayhew, D., McGowan, S., Polter, A., Qadri, Y., Sarfare, S., Schultz, K., Splittgerber, R., Stephenson, J., Tower, C., Walton, R. G., Zotov, A. (2007). The ups and downs of peer review. Adv. Physiol. Educ. 31: 145-152 [Abstract] [Full text]  
  • Pless, I B (2007). Unfinished business. Inj. Prev. 13: 145-146 [Full text]  
  • White, I. R, Carpenter, J., Evans, S., Schroter, S. (2007). Eliciting and using expert opinions about dropout bias in randomized controlled trials. Clin Trials 4: 125-139 [Abstract]  
  • Tite, L., Schroter, S. (2006). Evidence based publishing. BMJ 333: 366-366 [Full text]  
  • Lange, J. H. (2006). Tradition Dies Hard. Indoor and Built Environment 15: 301-303  
  • Smith, R. (2006). Peer review: a flawed process at the heart of science and journals.. JRSM 99: 178-182 [Full text]  
  • Millett, D. (2006). Training the reviewer?. J. Orthod. 33: 1-2 [Full text]  
  • Allebeck, P. (2006). Necessary steps for a modern scientific journal.. Eur J Public Health 16: 1-1 [Full text]  
  • Schroter, S., Tite, L., Hutchings, A., Black, N. (2006). Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors or by Editors. JAMA 295: 314-317 [Abstract] [Full text]  
  • Savulescu, J, Viens, A M (2005). What makes the best medical ethics journal? A North American perspective. J. Med. Ethics 31: 591-597 [Abstract] [Full text]  
  • Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., Smith, R. (2004). Effects of training on quality of peer review: randomised controlled trial. BMJ 328: 673- [Abstract] [Full text]  
  • Godlee, F. (2002). Making Reviewers Visible: Openness, Accountability, and Credit. JAMA 287: 2762-2765 [Abstract] [Full text]  
  • Jefferson, T., Alderson, P., Wager, E., Davidoff, F. (2002). Effects of Editorial Peer Review: A Systematic Review. JAMA 287: 2784-2786 [Abstract] [Full text]  
  • Jefferson, T., Wager, E., Davidoff, F. (2002). Measuring the Quality of Editorial Peer Review. JAMA 287: 2786-2790 [Abstract] [Full text]  
  • Bacchetti, P. (2002). Peer review of statistics in medical research: the other problem. BMJ 324: 1271-1273 [Full text]  
  • Dick, A D (2002). Are we on a voyage to the unknown?. Br J Ophthalmol 86: 253-254 [Full text]  
  • Melero, R., Lopez-Santovena, F. (2001). Referees' Attitudes toward Open Peer Review and Electronic Transmission of Papers. Food Science and Technology International 7: 521-527 [Abstract]  
  • WALSH, E., ROONEY, M., APPLEBY, L., WILKINSON, G. (2000). Open peer review: a randomised controlled trial. Br. J. Psychiatry 176: 47-51 [Abstract] [Full text]  
  • Smith, R. (1999). Opening up BMJ peer review. BMJ 318: 4-5 [Full text]  

Rapid Responses:

Read all Rapid Responses

Open peer review: no need for change
Alan S Rigby
bmj.com, 15 Jan 1999 [Full text]
Access jobs at BMJ Careers
Whats new online at Student BMJ