Education and debate

Evidence on peer reviewscientific quality control or smokescreen?

Sandra Goldbeck-Wood, assistant editor. 

BMJ, London WC1H 9JR

10176.2501{at}compuserve.com

Peer reviewthe process by which experts advise editors on the value of scientific manuscripts submitted for publicationis traditionally surrounded by an almost religious mystique. Published papers are an important part of most assessment systems that decide how academic posts and research grants are distributed. Peer review confers legitimacy not only on scientific journals and the papers they publish but on the people who publish them. But if peer review is so central to the process by which scientific knowledge becomes canonised, it is ironic that science has little to say about whether it works.

Editors have described peer review as "indispensable for the progress of biomedical science."¹ They argue that peer review helps them distinguish between good and bad papers and between good and bad research, that it improves the presentation of what is being published, and even that it educates editors and authors.² When they ask reviewers to comment on a paper's scientific reliability, originality, relevance, appropriateness to the journal, and other matters, editors hope they are providing some kind of intellectual quality control, allowing the best science to be selected and improved. But is this belief more than just wishful thinking and self aggrandisement by editors and other beneficiaries of the peer review system? The question is all the more relevant because peer review is so time consuming, complex, expensive, and prone to abuse.³

	The evidence so far

In 1990, at the first international congress on biomedical peer review, some editors began to examine critically their own activities. ^{4 5} The most recent insights into what, if anything, is achieved by peer review and how it might be improved were presented at the third such congress in Prague last autumn and were brought together in the July 15 issue of JAMA.⁶

Before 1990, most articles on biomedical peer review reported descriptive or observational studies. Many were heavy on opinion and speculation and light on evidence. Some speculated that blinding reviewers to authors' identity ("blinding"), asking reviewers to sign their reviews ("signing"), or passing the comments of one reviewer to other reviewers ("unmasking") might improve the quality of reviews by increasing objectivity and eliminating prejudice and bias. Others said that there might be special characteristics associated with high quality reviews, such as age, seniority, holding an academic post, or having published widely. There were numerous anecdotes of biases and of abuses of the peer review system. ^{7 8}

Blinding, signing, and unmasking
McNutt et al were the first to use a randomised controlled trial to examine the issues of blinding and signing.⁹ In their 1990 study of 127 consecutive manuscripts submitted to an American internal medicine journal, blinding increased the quality of reviews in a statistically significant way, though the improvement failed to reach their predefined threshold for administrative significance. The authors found no association between signing of reviews and review quality. Limitations of this trial were its small size, the specialist nature of the journal, the fact that reviewers were not randomly assigned to signing or not signing their reviews, the lack of a previously validated instrument for assessing review quality, and inability to exclude a "Hawthorn effect" (the possibility that reviewers' behaviour changed merely as a result of being studied).

Reviewer characteristics
In a separate analysis of data from the same trial, Black et al found that the characteristics of reviewers, such as demographic factors, specialty, seniority, or academic appointments, had little association with the quality of the reviews they produced, explaining only 8% of review quality.¹⁴ A logistic regression analysis found that training in epidemiology and statistics, and younger age, were the only characteristics significantly associated with higher quality ratings. Paradoxically, membership of an editorial board was associated with lower, not higher, review quality.

Bias
Other contributors to JAMA's issue on peer review illustrate the worrying number of biases by which peer review is beset, including nationality bias,¹⁵ language bias,¹⁶ specialty bias,¹⁷ and perhaps even gender bias,¹⁸ as well as the recognised bias toward the publication of positive results.^19-21

	Major challenges

For all the progress that has been made since 1990, some of the most important questions remain unanswered. The greatest challenge for peer review researchers is perhaps the quest for an instrument capable of measuring the most interesting and least accessible outcome of allmanuscript quality. Up to now, researchers have had little choice but to study the intermediate outcome of review quality; but to discover whether peer review is an effective intervention, we want to be able to trace its effects on the manuscript.

The other major challenge is obtaining funding for this new area of research, which falls outside the sphere of interest of almost all grant giving bodies. Much of the research so far has been conducted "on a shoestring," using small, one-off grants and time borrowed from researchers' other, paid commitments.

	Where does this leave us?

Researchers have an interest in knowing about the fairness of the systems by which their research is judged. If the peer review process should turn out to be worthless or, worse still, hopelessly corrupt, researchers would be better off committing their findings to the internet. Meanwhile, it may be some small comfort to those who conduct research and submit papers to journals that editors, forced to grapple with the challenges of designing their own trials, are now receiving a salutary taste of their own medicine.

	References

1. Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable. JAMA 1994; 272: 96-97[Medline].
2. Goldbeck-Wood S. What makes a good reviewer of manuscripts? BMJ 1998; 316: 86[Free Full Text].
3. Rennie D. Peer review in Prague. JAMA 1998; 280: 214-215[Free Full Text].
4. Bailar JC, Patterson K. Journal peer review: the need for a research agenda. N Engl J Med 1985; 312: 645-657.
5. Guarding the guardians: research on editorial peer review. Selected proceedings from the first international congress on peer review in biomedical publication. JAMA 1990; 263: 1317-1441[Medline].
6. Peer review theme issue. JAMA 1998; 280: 203-306.
7. Sharp DW. What can and should be done to reduce publication bias? JAMA 1990; 263: 1390-1391[Abstract].
8. Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias. JAMA 1990; 263: 1392-1393[Abstract].
9. McNutt MD, Evans A, Fletcher R, Fletcher S. The effects of blinding on the quality of peer review. JAMA 1990; 263: 1371-1376[Abstract].
10. Van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review. JAMA 1998; 280: 234-237[Abstract/Free Full Text].
11. Justice AC, Cho MK, Winker M, Berlin JA, Rennie D, the PEER investigators. Does masking author identity improve peer review quality? A randomised controlled trial. JAMA 1998; 280: 240-242[Abstract/Free Full Text].
12. Cho MK, Justice AC, Winker MA, Berlin JA, Waekerle JF, Callaham ML, et al. Masking author identity in peer review: what factors influence masking success? JAMA 1998; 280: 243-245[Abstract/Free Full Text].
13. Van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ 1998; 317: 23-27.
14. Black N, Van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA 1998; 280: 231-233[Abstract/Free Full Text].
15. Link AM. US and non-US submissions, an analysis of reviewer bias. JAMA 1998; 280: 246-247[Abstract/Free Full Text].
16. Junker CA. Adherence to published standards of reporting, a comparison of placebo-controlled trials published in English or German. JAMA 1998; 280: 247-249[Abstract/Free Full Text].
17. Joyce J, Rabe-Hesketh S, Wessely S. Reviewing the reviews: the example of chronic fatigue syndrome. JAMA 1998; 280: 264-266[Abstract/Free Full Text].
18. Dickersin K, Fredman L, Flegal KM, Scott J, Crawley B. Is there a sex bias in choosing editors? Epidemiology journals as an example. JAMA 1998; 280: 260-263[Abstract/Free Full Text].
19. Misakinan AL, Bero LA. Publication bias and research on passive smoking, comparison of published and unpublished studies. JAMA 1998; 280: 250-253[Abstract/Free Full Text].
20. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA 1998; 280: 254-257[Abstract/Free Full Text].
21. Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998; 280: 257-259[Abstract/Free Full Text].