Peer review system needs thorough evaluation, MPs hear

BMJ 2011; 342 doi: (Published 16 May 2011) Cite this as: BMJ 2011;342:d3046
  1. Adrian O’Dowd
  1. 1London

The peer review system used by many medical journals before they publish new research needs to be thoroughly evaluated to confirm its value, MPs have been told.

Several medical and scientific journal editors appearing before the parliamentary science and technology select committee on 11 May spoke of the many merits of peer review, but they raised some concerns about the variability of its quality and a lack of adequate evaluation of the system.

The committee was holding the second oral evidence session as part of its inquiry into the peer review process in science (BMJ 2011;342:d2858 doi:10.1136/bmj.d2858).

MPs asked about the BMJ’s written evidence to the committee, which mentioned a study that had concluded that little empirical evidence was available to support the use of editorial peer review as a mechanism to ensure the quality of biomedical research.

Fiona Godlee, editor in chief of the BMJ Group, giving evidence, said, “It’s long been felt that a system as important as peer review has been to medicine and science is remarkably underevaluated.

“There have been studies, but I think the overall level of evaluation of peer review is very poor—not only journal editorial peer review but also peer review of grant applications. We should have it [an evaluation]. We can improve it. There are huge flaws.”

MPs asked whether the peer review process had a “conservative impact” on science and whether this was a problem.

Robert Campbell, senior publisher at Wiley-Blackwell, also giving evidence, said, “I don’t see it as particularly conservative. A good editor will encourage the author to write a better paper and to develop those ideas and get them over more effectively. It is a positive process.

“If you have a very conservative editorial board the journal will suffer. It is a market, and the more proactive, entrepreneurial editorial teams will win out.”

Fellow witness Philip Campbell, editor in chief of Nature Publishing Group, said, “What peer review really helps us to deliver is robust publication, and we at Nature are more conservative than other journals. We do make researchers go the extra mile to demonstrate what they are saying.

“I also celebrate that fact that some papers go against the status quo, and in that sense we do not want to be conservative and we do want to encourage radical discoveries.”

Dr Godlee told the MPs: “Conservatism is not a bad thing in science and medicine in terms of making sure that what we publish is robust, relevant, and quality controlled, but we should not be conservative in how we go about that.”

Quality, however, was an issue, she added: “We have to acknowledge that there is a huge variety in the quality of peer review across the publishing sector.”

She said, “Journals like Nature, BMJ and the Lancet, which have got big editorial teams within them, do a very different type of peer review to those with much less resource.

“At its worst peer review has had words used about it such as ‘slow,’ ‘expensive,’ ‘biased,’ ‘open to abuse,’ ‘stifles innovation,’ and ‘bad at detecting fraud,’ but at its very best it does improve the quality of scientific reporting and can improve . . . the quality of the science itself.”

MPs asked for the witnesses’ opinions on written evidence the committee had received claiming that the peer review system was in crisis, as it was characterised by academics and researchers with too little time and few incentives to participate in peer review.

Mr Campbell said, “There is no quantitative evidence that it is in crisis. The peer review system as a whole is more robust than ever.”

Nature’s Dr Campbell said, “Nature and the Royal Society co-hosted a discussion of Royal Society research fellows, and there was definitely a sense that their lives were getting more burdensome.”

Dr Godlee added: “Scientists are under a lot of pressure on a whole host of things—getting funding and the bureaucracy surrounding scientific research—and peer review is just one other thing, so the more we can do to make it something that they can gain proper recognition for, the better.”


Cite this as: BMJ 2011;342:d3046


View Abstract

Sign in

Log in through your institution