BMJ No 7111 Volume 315
News Saturday 27 September 1997
The peer review process is viewed as the arbiter of quality in published research, yet is itself based on little evidence. In Prague last week an international congress on biomedical peer review and global communications debated the value and purpose of peer review. Sandra Goldbeck-Wood reports.
Blinding reviewers to authors' identity does not improve quality
| Reviewers of research manuscripts do not produce better reports if they are blinded to the authors' identity and institution, according to research presented by investigators from the London School of Hygiene and Tropical Medicine and the BMJ in Prague last week.
The study contradicts the suspicion common among authors and editors that reviewers' opinions of a paper may be biased according to the renown of its authors or the prestige of the research institution.
|
 | |
| Little seems to be gained from blind peer review |
In a randomised controlled trial of 487 papers submitted to the BMJ, reviewers were randomly allocated a blinded or unblinded copy of papers. Reviewers were also randomly assigned, whether or not they would be asked to allow their signed reviews to be passed to another reviewer. Independent evaluation of each review found no significant difference between blinded and unblinded reviewers in the overall quality, recommendation about publication, or time taken to complete the review. There was also no difference in the quality of the review between reviewers whose reports were sent to another reviewer and those whose reviews were not passed on.
Similar conclusions were reached by a multicentre randomised trial from Cleveland in the United States. Based at five leading US medical journals including JAMA and the Annals of Internal Medicine, Dr Amy Justice and colleagues also allocated blinded or unblinded manuscripts to reviewers. Among the 58% of papers for which both reviews were returned, the researchers found no significant difference in the quality of review between the two groups of reviewers. But this study also showed that blinding is not always possible. Checks to ensure the efficacy of the blinding procedure showed that a significant number of reviewers were able to guess the authors' identity. This was a particular problem for specialist journals representing small research communities, in which researchers are more likely to know one another
Dr Fiona Godlee, one of the authors of the BMJ study, said: "Taken together, the two studies sugget that there is little to be gained from blind peer review especially if there is a bias towards better known authors being easier to identify."
But the New England Journal of Medicine's editor, Marcia Angell, said that the experiments violated the Nuremberg Code and the Declaration of Helsinki, which called for the best standard of care and not the local standard.
In a joint statement the Centers for Disease Control and Prevention and the National Institutes of Health responded that the zidovudine regimen used in the US "is not feasible for poorer nations" and that the studies offer the best hope of finding a safe and affordable alternative.
Gender bias exists in research
Only 13% of medical journal editors are women, compared with 27% of reviewers and 29% of authors, according to research presented in Prague.
Investigating the hypothesis that gender bias exists in research, Dr Kay Dickersin from the University of Maryland School of Medicine in Baltimore examined four major journals in the United States for the years 1982, 1987, 1992, and 1994. Among the 2415 reports, 8005 authors, 2982 reviewers, and 695 editors studied, women were underrepresented at all levels but more so as editors than elsewhere in the publication process.
On the basis of a pyramid of influence with editors at the top, reviewers in the middle, and authors at the bottom, Dr Dickersin concluded that women are most underrepresented in the most influential areas of the publication process. "It is not clear why this is," said Dr Dickersin. "The fact that the proportion of editors who were women in 1994 is still considerably lower than the proportion of women who were authors in 1982 does not support a cohort effect as the sole explanation. Another possibility is a selection bias favouring men for editorial positions."
Dr Liselotte Højgaard, the first woman editor in chief of the Danish medical journal Ugeskrift for Læger said: "There are now more women than ever at medical school and more in publishing, but still we are underrepresented at the highest levels. I think one solution is that women need to like each other a little more, instead of just enjoying the payoffs of being a woman in a man's world."
Dr George Lundberg, editor in chief of JAMA, said: "In some fields it is very difficult to find women candidates, but I think things will improve. But it will take affirmative action as well."
Citing of research shows national bias
Authors from the United States and the United Kingdom preferentially cite research from within their own countries, according to a retrospective data review presented in Prague by Simon Wessely, professor of epidemiological and liaison psychiatry at King's College School of Medicine.
Investigating the effect of specialty and nationality on the selection of references, Professor Wessely studied 89 review articles on the chronic fatigue syndrome that had been found in three major databases of medical literature.
Each reference cited was categorised by specialty and nationality of its author(s), and associations were tested using analysis of variance. Authors of the review articles showed bias towards research not only from their own country but also from their own specialty. Thus despite the multidisciplinary nature of the subject, psychiatrists tended to cite psychiatric papers while laboratory based researchers tended to cite other laboratory research. Only 3.4% of the review articles described the literature search method that had been used.
According to Professor Wessely, the size of the bias was substantial. "The mean difference between the percentage of US and UK references showed that US authors quoted 54% more US research and UK authors 20% more UK work. This gives an overall spread for the difference in preference of 75%. What this shows is that in a controversial area, where we need more reason and less passion, most overviews singularly fail to fulfil basic criteria for scientific acceptability," said Professor Wessely.
Home | Current issue | Past issues | Classified ads | Career Focus | Feedback
Collections | About this site | About the BMJ | BMA | Medline