Time to open up the black box of peer review

As recently as 10 years ago we had almost no evidence on peer review, a process at the heart of science. Then a small group of editors and researchers began to urge that peer review could itself be examined using scientific methods. The result is a rapidly growing body of work, much of it presented at the third international congress on peer review held in Prague last week. The central message from the conference was that there is something rotten in the state of scientific publishing and that we need radical reform.

The problem with peer review is that we have good evidence on its deficiencies and poor evidence on its benefits. We know that it is expensive, slow, prone to bias, open to abuse, possibly anti-innovatory, and unable to detect fraud. We also know that the published papers that emerge from the process are often grossly deficient. Research presented at the conference showed, for instance, that reports of randomised controlled trials often fail to mention previous trials and do not place their work in the context of what has gone before; that routine reviews rarely have adequate methods and are hugely biased by specialty and geography in the references they quote (p 766); and that systematic reviews rarely define a primary outcome measure.

Perhaps because scientific publishing without peer review seems unimaginable, …