Editorials

Opening up BMJ peer review

A beginning that should lead to complete transparency

Papers p   23 Education and debate p   44

The BMJ has until now used a closed system of peer review, where the authors do not know who has reviewed their papers. The reviewers do, however, know the names of the authors. Most medical journals use the same system, but it's based on custom not evidence. Now we plan to let authors know the identity of reviewers. Soon we are likely to open up the whole system so that anybody interested can see the whole process on the world wide web. The change is based on evidence and an ethical argument.

Peer review is at the heart of the scientific process yet was until recently largely unexamined. Now we begin to have a body of evidence on peer review (www.wame.org), and it illustrates many defects. Peer review is slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud. Evidence to support all these statements can be found in a book by Stephen Lock, my predecessor as editor of the BMJ,¹ three special issues of JAMA,^2-4 and a forthcoming book.⁵ The benefits of peer review are harder to pin down, but it is probably more useful for improving what is eventually published than for sorting the wheat from the chaff.⁶

Those researching peer review have tried to find better methods, and one of the first randomised controlled trials suggested that blinding reviewers to the identity of authors would lead to better opinions.⁷ Two bigger trialsone that included many journals⁸ and one from the BMJ⁹both failed, however, to find any benefit.¹⁰ This led to the idea that open peer review might be a better option, and we publish today a randomised controlled trial of open peer review conducted at the BMJ.¹¹ It found that open peer review does not lead to higher quality opinions, but nor does it lead to poorer quality ones, so we are introducing open reviewfor largely ethical reasons.

The arguments for and against open peer review were explored in depth five years ago in Cardiovascular Research. ^{12 13} Of six editors asked to contribute commentaries all were for more research, none was against open peer review, and three, including Stephen Lock (my predecessor), declared themselves in favour.¹³ Science is progressively moving away from anonymity. Anonymous editorials in scientific journals were common a decade ago; now they look anachronistic.

The primary argument against closed peer review is that it seems wrong for somebody making an important judgment on the work of others to do so in secret. A court with an unidentified judge makes us think immediately of totalitarian states and the world of Franz Kafka. A related argument is, in the words of Drummond Rennie (deputy editor of JAMA), that identifying the reviewer links "privilege and duty, by reminding the reviewer that with power comes responsibility: that the scientist invested with the mantle of the judge cannot be arbitrary in his or her judgment and must be a constructive critic." All editors have seen curt, abusive, destructive reviews and assumed that the reviewer would not have written in that way if he or she were identifiable. Openness also links accountability with credit. One important defect of closed review is that reviewers don't receive academic credit. Finally, openness should eliminate some of the worst abuses of peer review, where reviewersunder the cloak of anonymitysteal ideas or procrastinate.

The main argument against open peer reviewa sad oneis that junior reviewers will be reluctant to criticise the work of senior researchers for fear of reprisals. This fear is particularly acute for researchers whose livelihoods depend on winning grants. Junior reviewers, those under 40, have time and again been shown to give the best opinions.¹⁴ By moving to open review we may thus be ruling out the best reviewers. We recognise these arguments, but we don't think that they outweigh the arguments for open review; in particular, BMJ authors seem broadly in favour of open peer review.¹¹ A few reviewers have said that they don't want to review if they will be identified, and anyone can decline to review a particular paper. Nevertheless, we hope our small move will contribute to a broader culture change so that junior researchers cease to fear reprisals from senior ones.

From this week, for all new papers that we review, the BMJ will identify to authors the names of those who have reviewed their papers, including the names of our in house editorial and statistical advisers. But we expect to go further, researching as we go. Soon we will probably start to list reviewers at the end of articles. Then we may move to a system where authors and readers can watch the peer review system on the world wide web as it happens and contribute their comments. Peer review will become increasingly a scientific discourse rather than a summary judgment. Through such openness we will hope to show that peer review by journals does add value to the scientific process and that we will thus have a place in an electronic world where authors can potentially go straight to readers.

Richard Smith, Editor. 

BMJ

1. Lock S. A difficult balance: editorial peer review in medicine. London: Nuffield Provincials Hospital Trust , 1985.
2. Guarding the guardians: research on editorial peer review: selected proceedings from the First International Congress on Peer Review in Biomedical Publication. JAMA 1990; 263(theme issue): 1309-1456.
3. Second International Congress on Peer Review in Biomedical Publication. JAMA , 1994:272(theme issue):91-173.
4. Third International Congress on Peer Review in Biomedical Publication. JAMA , 1998:280(theme issue):213-306.
5. Godlee F, Jefferson T. Peer review in medicine. London: BMJ Books (in press).
6. Pierie JP, Walvoort HC, Overbeke JA. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 1996; 348: 1480-1483[Medline].
7. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA 1990; 263: 1371-1376[Abstract].
8. van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomised trial. JAMA 1998; 280: 234-237[Abstract/Free Full Text].
9. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, the PEER investigators. Does masking author identity improve peer review quality: a randomised controlled trial. JAMA 1998; 280: 240-242[Abstract/Free Full Text].
10. Goldbeck-Wood S. Evidence on peer reviewscientific quality control or smokescreen? BMJ 1999; 318: 44-45[Free Full Text].
11. van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ 1999; 318: 23-27[Abstract/Free Full Text].
12. Fabiato A. Anonymity of reviewers. Cardiovasc Res 1994; 28: 1134-1139[Free Full Text].
13. Fletcher RH, Fletcher SW, Fox R, Horrobin DF, Lock S, Pepper K, et al. Anonymity of reviewers. Cardiovasc Res 1994; 28: 1340-1345.
14. Goldbeck-Wood S. What makes a good reviewer of manuscripts. BMJ 1998; 316: 86[Free Full Text].