The Cochrane Collaboration at 20BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f7383 (Published 18 December 2013) Cite this as: BMJ 2013;347:f7383
Just as Archimedes leapt naked from his bath on discovering his principle and August Kekulé dreamt the structure of the benzene ring while sleeping beside a fire, so Iain Chalmers had a vision of the Cochrane Collaboration at 6 am in May 1991 while walking beside a tributary of the Thames in Oxford. The collaboration would fulfil the vision of Archie Cochrane and clean up the Augean stables of medical studies. Specifically it would prepare, maintain, and promote the accessibility of systematic reviews of the effects of health interventions.1 2 The enterprise would need to be global because it was such a huge task.
Twenty years after it was founded the collaboration has more than 31 000 contributors from 120 countries and has published more than 5000 systematic reviews.3 Many see Cochrane reviews as the gold standard, and the collaboration has played a major role in promoting evidence based practice. The collaboration is clearly a success. But like any organisation it has problems and challenges, and Chalmers, who gives an annual prize for the best criticism of the collaboration, described some of them at the 21st gathering of the collaboration in Quebec. Challenges included finding more efficient means of preparing and updating reviews and avoiding duplication of reviews.4
Perhaps the main challenge is whether to extend the collaboration’s mission. It has mainly covered treatments, but should it be extended to, for example, diagnostic tests, qualitative studies of implementation, and products derived from the systematic reviews that might compete with tools like UpToDate and the BMJ’s evidence products.3 Clearly, extension would be desirable because all elements of healthcare need to be evidence based. In addition, evidence based tools beyond systematic reviews can promote evidence based healthcare and provide additional sources of revenue for the collaboration.
The argument against such a move is that the collaboration has much to do to achieve its current mission. Coverage by the systematic reviews is patchy—some topics are thinly covered and some reviews answer questions that interest only the authors. Some reviews conclude that there is no reliable evidence, which is important to know but not useful to clinicians. This patchiness is not surprising because Cochrane is a bottom-up organisation, with 53 review groups mostly answering the questions that are passively brought to them, and many reviewers are more researchers than clinicians. Some reviews are commissioned, but most are produced by unpaid people who cannot easily be told what to do. There are now attempts to prioritise topics for review and to be more directed. Chalmers’s latest creation, the James Lind Alliance, could help because it aims to identify uncertainties in healthcare.
But how many reviews would be needed to answer all questions on just treatment? Jon Brassey, a supporter of evidence based practice, points out that of 358 questions asked in dermatology only three could be answered by a single systematic review.5 Healthcare is increasingly about people with multiple conditions,6 yet most randomised trials exclude patients with more than one condition. This is a problem for all of healthcare, not just for the Cochrane Collaboration and evidence based medicine.
Another factor that affects all of healthcare but is a particular problem for systematic reviews is that about half of clinical trials are not published.7 This was pointed out in a 1992 BMJ editorial that described the work of the collaboration,2 but little progress has been made since then, although a major campaign is now under way.7 Because published trials are biased towards the positive, systematic reviews probably exaggerate the benefits of interventions. Tom Jefferson, a longstanding reviewer for the collaboration, has struggled to obtain all the data on neuraminidase inhibitors for treating influenza and concluded that “Cochrane reviews based on publications should really be a thing of the past.”5
The epidemiologist Rod Jackson and others have argued that to be truly useful to clinicians systematic reviews should be based on individual patient data so that patients can be stratified by risk. Such information helps clinicians to know when to treat patients. The limited amount of information on adverse effects also reduces the usefulness of reviews; the lack of information results from poor reporting of adverse effects in trials and trials not being the best way to identify such effects.
Even with existing reviews there are worries about quality. The “methodological expectations of Cochrane intervention reviews” has 80 standards for conducting and 108 for reporting reviews, and few reviews meet all of these standards.3 A recent audit showed some basic deficiencies in reviews, particularly in abstracts and plain language summaries.8 Another worry is that only around a third of reviews have been updated in the past two years.9 In some cases this doesn’t matter—because the evidence hasn’t changed or the question is no longer relevant—but in others it renders the review useless. The proportion of reviews that are up to date has fallen over the past three years,9 and the collaboration may need to find a way of identifying the reviews that need updating the most. No other provider of systematic reviews does better, perhaps because updating can be tiresome and unrewarding, and often arrives at something close to the original conclusion.
Efficiency is a problem because it takes an average of 30 months to complete and publish a Cochrane review.9 Brassey points out that Cochrane has received something like £150m (€179m; $246m) in funding over its 20 years,5 around £30 000 a review. And remember that most reviews are conducted by people in their own time.
Both the quality of the reviews and the efficiency of their production urgently need to be raised. Some in the collaboration have begun to produce more rapid reviews, and Chalmers and Paul Glasziou, a longstanding member of the collaboration, have suggested categorising questions into those needing just a rapid review, those that merit a full review, and some that demand a review based on individual patient data.4 Andy Oxman, who has been in the collaboration since the beginning, has listed 13 ways to improve reviews, starting with “Ensure that any important potential adverse effects of the interventions are addressed (whether the included studies report those outcomes or not).”3 Brassey suggests that it might be possible to depend simply on the best trial or trials reported in core journals.5 These strategies are unlikely to be acceptable, not least because if those core journals are the “top” journals, they would be most likely to exaggerate the benefits of treatments.10
Another challenge is to make systematic reviews more useful and accessible. Reviews are currently difficult for clinicians, so the collaboration is exploring more useful “derivative products,” although it might struggle against established competitors. The collaboration is committed to achieving open access, which is a more realistic goal. Many people have free access though HINARI and national licences, but at least half of the world does not, and open access is more than free access—it also allows reuse and repurposing. Although institutions in some low income countries have free access, few reviews cover problems relevant to those countries, partly because most research is from the developed world. About a fifth of Cochrane review authors come from developing countries,9 and the collaboration is keen to increase this proportion and find ways to be more useful in low income countries.
The Cochrane Collaboration is a special organisation, more like a church than a corporation. Muir Gray, one of the founders, says that he saw the Jesuits as a good model. “Cochranites,” as they are called, believe passionately in the collaboration’s mission, often work for free, and are signed up for life. The passion is palpable at annual colloquiums and has driven many of the collaboration’s achievements, but there is room to be more business like. The Cochrane Library got its first editor a few years ago, and the collaboration appointed a new and modernising chief executive just over a year ago. As a consequence, a strategy that has goals and objectives and will soon have targets is being finalised. The collaboration is also considering whether 53 review groups is the best way for it to be structured.
Rightfully proud of its achievements, the collaboration is well prepared to meet its many challenges.
Cite this as: BMJ 2013;347:f7383
Competing interests: I have read and understood the BMJ Group policy on declaration of interests and declare the following interests: I have attended and spoken at several of the Cochrane colloquiums, including the first one. I am chair of the Cochrane Library oversight committee and have presented a series of films on the history of evidence based medicine for JAMA that will be released early in 2014.
Provenance and peer review: Commissioned; not externally peer reviewed.