Problems with peer review

Mistakes will happen

The charges against the IPCC, the Lancet, and the stem cell journals reflect a well rehearsed criticism of peer review: that it fails to root out error. Yet even the most rigorous refereeing procedures cannot prevent every inaccuracy, and they can achieve still less when conflicts go undeclared or outright fraud is involved. The best and most conscientious reviewers cannot spot every slip.

Though the IPCC’s error was indefensibly glaring, many of its scientists have reasonably pointed out that it would be remarkable for a 3000 page report to be completely error-free. As Jürgen Willebrand, an IPCC lead author, told Nature: “IPCC reports are written by humans. I have no doubt that similar errors could be found in earlier IPCC reports, but nobody has bothered to look in detail.”5 No mistake in the IPCC’s work has yet been identified that alters its fundamental conclusions. And for all Professor Jones’s bluster, the papers to which he objected were in fact considered by the appropriate working group.

In the Lancet case, Evan Harris, the Liberal Democrat member of parliament, led calls for a retraction six years ago, when Wakefield’s undeclared legal aid funding was first revealed. At the time, however, the journal ruled that no misrepresentations in the paper itself had been proved.

The Lancet, it might be argued, ought not to have published a paper with such significant implications for public health without checking these details. Yet when a researcher is not candid, it can be difficult for even the most assiduous reviewer or editor to find flaws. Submitted data must generally be taken on trust, though its interpretation must always be checked.

Genuinely bad behaviour is more usually identified after publication, when others replicate experiments or pick over the published research in detail. Hwang’s work, for example, fell under scrutiny when rivals failed to repeat his techniques, ethical doubts emerged over his egg collection procedures, and a former colleague turned whistleblower. This invited fresh analysis that revealed much of his data had been faked. Short of insisting that experiments are independently repeated before acceptance (as Nature did with a monkey cloning paper after the Hwang affair6 7), peer review can only do so much to detect fraud.

Reviewing the reviewers

Of the three recent incidents, the criticisms by Professors Smith and Lovell-Badge are most challenging. Their concern is that in the eyes of editors and reviewers, some scientists are more equal than others. Some papers thus do not get the scrutiny they need, while others are unfairly rejected.

“On the one hand, papers are held up by referees asking for experiments that no reasonable person would demand,” Professor Smith said. “On the other, people are making important and extraordinary claims without going the extra mile and providing the critical bit of data. Most people in the field have had one or more bad experiences.”

Some editors, they say, are reluctant to upset favourite scientists by overturning their reviews, for fear that they will stop submitting their work to that journal. That can give them excessive and unaccountable power. Anonymity also means that some referees do not declare their interests and review the work of a fierce rival or a collaborator. “If I receive a paper which someone in my lab has worked on, or even a good friend, I will say there is a conflict of interest and decline to review,” Professor Lovell-Badge said. “I’m sure not everybody does that.”

Philip Campbell, the editor of Nature, rejects the charge. His journal uses more than 400 referees in stem cell research alone, and he cites cases where editors have published a paper they think is important despite three unfavourable reviews. “We try to avoid all situations where referees abuse their positions,” he said. “Our editors keep in good touch with the research community, they never get dependent on a small group. I’m in no way denying that there are concerns out there, but it isn’t the case that referees are keeping good research out of Nature.”

For Mark Walport, director of the Wellcome Trust, it is good editors that should make the system tick. “It is the job of scientific editors, who usually have two or three reviews in front of them, to spot when people are misbehaving,” he said. “A good editor undoubtedly can.”

Despite its perceived weaknesses, improvements to peer review are notoriously hard to find. A double blind approach by which neither reviewer nor author knows the other’s identity, for instance, is difficult because authors can usually be guessed from their citations and subject matter.

Professor Smith accepts there is no easy answer, and Dr Walport likes to quote Winston Churchill’s famous dictum about democracy: that it is the worst form of government, “except for all those other forms that have been tried from time to time.” Yet two new models that are starting to gain ground do have some potential to address the most common complaint: that the system is unnecessarily opaque and unaccountable.

Open review

The BMJ has adopted one radical approach—opening up peer review so that referees are no longer anonymous. In most other journals reviews are unsigned to encourage candour and so that junior researchers can take part without fear that a negative opinion might be held against them by a senior figure. Drs Campbell and Walport both reject open review for just this reason. But Fiona Godlee, the BMJ’s editor, says the journal has not had this problem.

“We did a randomised controlled trial of signed versus unsigned reviews and found that it was acceptable to authors and reviewers, and that it made no significant difference to the reviews,” she said. “The quality was unchanged, though there was a slightly greater tendency to recommend acceptance.8 Since implementing open review, we have had one or two reviewers saying they won’t review for us, but the vast majority of reviewers are fine with it. And authors like it.”

She accepts that open review may not work for every journal, particularly those covering very specialist areas in which researchers tend to know each other well. She also highlights the importance of the BMJ’s editors: “They make the final decision on papers, so we are not reliant on the recommendation of the peer reviewer about whether to publish.”

There is also an intermediate solution, which has been pioneered by the European Molecular Biology Organisation Journal. Although it does not name reviewers, it publishes their reports. It is an approach that appeals to Smith, Lovell-Badge, and Walport. “If you publish a package of supplementary material, including anonymous reviews, it provides a paper trail and another level of accountability,” Dr Walport said. “It would place pressure on reviewers to be scrupulously fair, because anything openly hostile or ridiculous would be out there, and on journal editors to think very carefully about their comments.”

The BMJ is about to take this one step further—publishing its signed reviews alongside published papers after a second randomised trial found this feasible and acceptable to authors and reviewers. Meanwhile Nature is considering the anonymous publication of referees’ reports. “We’ve been thinking about that for a few years,” Dr Campbell said. “There are questions we need to be careful about, such as does this change the relationship between the editor and the referee, but it is absolutely something we are looking at.”

It may be true that peer review is the worst system of scrutinising science, except for all the others that have been tried from time to time. But like democracy, that does not mean it can’t be tweaked to make it fairer.