Intended for healthcare professionals

Head To Head

Is open peer review the fairest system? No

BMJ 2010; 341 doi: https://doi.org/10.1136/bmj.c6425 (Published 16 November 2010) Cite this as: BMJ 2010;341:c6425
  1. Karim Khan, editor
  1. 1British Journal of Sports Medicine, London, UK
  1. karim.khan{at}familymed.ubc.ca

Trish Groves (doi:10.1136/bmj.c6424) argues that telling authors who has reviewed their paper has helped to make the process fairer, but Karim Khan is concerned that it stops reviewers from being completely frank

Open peer review makes perfect sense in the ideal world. But it is not an ideal world. As editor of a BMJ Group specialist journal, I am concerned that open review provides more scope for power relationships to favour “the great and the good.”

Anonymity creates a safe place

Open peer review is associated with the risk that an inferior paper written by a senior authority in the field may receive a “soft” or generous review from a junior reviewer who either seeks to curry favour or fears an honest review would lead to payback at some future time.1 2 Van Rooyen and colleagues’ classic randomised controlled trial found a 5% greater positive recommendation among identified reviewers (open) than among anonymous reviewers (closed). Although that difference seems unremarkable, the authors of that study, undertaken at the BMJ, acknowledge that results could differ for a small specialist journal where the authors and reviewers are more likely to come into regular contact.

Reviewers for the British Journal of Sports Medicine (BJSM) have the option of remaining anonymous (closed review), and about a quarter do. I received this unsolicited note from a reviewer: “I don’t want my name used. I know [author] and she will likely know these are my comments. I’d just rather not reveal that to her. We have known each other for quite some time and I have a great deal of respect for her and her work. I wouldn’t want to jeopardise that relationship if she perceived these comments to be negative.” Importantly, the closed review was fair and constructive, and the paper was accepted. Open peer review, in this case, would have either prevented the reviewer from accepting the task or led to a modified review. In either case, an inferior outcome.

Open review can cause reviewers to blunt their opinions for fear of causing offence and so produce poorer reviews.3 Authors and reviewers are human. Psychology professor Robert Cialdini argues that reciprocity is the most powerful of all the forces that influence behaviour—give what you want to receive.4 Open review assumes that reciprocity has no role.

Does closed review empower the jealous rival?

As humans are imperfect, occasionally reviewers will use their cloak of invisibility to savage a paper for personal reasons; jealousy is not eradicated by climbing the academic ladder. Editors may suspect this and obtain additional reviews, but if not there should be a robust system of appeal managed at arm’s length from the first review. At BJSM editors involved in the first rejection are not part of the appeals process. Of 32 appeals, four were successful—this compares with the journal’s overall acceptance of 10-15% for new papers.

Aren’t all reviews fair?

BJSM has a specific “peer review: fair review” section for authors who feel that their manuscripts have not been fairly treated at other journals. Although our aspiration is that all reviews are fair, this section allows the author to highlight reviewers who should be excluded; we encourage them to submit previous reviews without prejudice. This process mirrors what has been happening at grant review panels in many places. To date we have accepted six papers from eight submissions through this method.

A practical note—finding reviewers

The proponents of open review argue that it should “increase the credit” for the reviewer.3 Although this aspiration is laudable, my experience from sitting on university promotion and tenure committees suggests that the universities who should confer credit have a long way to go before they will value peer review. Importantly, a portion of the potential reviewer pool clearly considers review more of a burden than a gift; in the BMJ open review experiment, 23% of closed and 35% of open reviewers declined the invitation.3 Using BJSM as a specialist journal comparator, 28% of 600 requested referees have declined the invitation so far this year. If that became 40% (arguably conservative), the editorial office would need to invite about 10-12 additional reviewers a month.

Peer review, either open or closed, has limitations; indeed it has become popular to draw the parallel between Churchill’s opinion regarding democracy and the state of peer review (seriously, almost fatally flawed, but better than any alternative). The ongoing commitment to studying peer review is to be encouraged, and journal editors must apply existing evidence and contribute to further research where possible.

Notes

Cite this as: BMJ 2010;341:c6425

Footnotes

  • Research,doi:10.1136/bmj.c5729
  • Competing interests: All authors have completed the unified competing interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare no support from any organisation for the submitted work; no financial relationships with any organisation that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.

  • Provenance and peer review: Commissioned; not externally peer reviewed.

References

View Abstract