The problem of duplicate systematic reviewsBMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f5040 (Published 14 August 2013) Cite this as: BMJ 2013;347:f5040
- David Moher, senior scientist
- 1Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, Canada ON K1H 8L6
Systematic reviews occupy a central position in evidence based medicine. They are the starting point of a well developed practice guideline. Some funders of randomised trials ask investigators for a strong rationale for their proposed trial, indicating that the best evidence is likely to be a well conducted and completely reported systematic review.1 These reasons, and others, probably explain the popularity and publication trajectory of systematic reviews.2 Does this translate into duplication of effort and waste? In a linked paper (doi:10.1136/bmj.f4501), using sound methodology and complete and transparent reporting, Siontis and colleagues examined this question.3
Having selected 73 meta-analyses published during 2010, the authors identified two thirds of them as having at least one overlapping meta-analysis. The good news is that duplication does not seem to have been a major problem. The authors report a median of two overlapping meta-analyses per topic. However, for several clinical topics there were multiple duplicates, and in 17 instances at least one author was involved in more than one overlapping meta-analysis. These findings provide another opportunity to pause and reflect on the development of systematic reviews and meta-analyses.
Not all duplication is bad. Indeed, replication is essential and has uncovered some unfortunate behaviour by scientists.4 5 With a judicious amount of duplication, clinicians—and other decision makers—can be more confident in the consistency, or lack thereof, of initially reported results. Regulators often require two randomised trials conducted in different jurisdictions as part of a drug’s approval process. In the context of systematic reviews and meta-analyses, there is no magic number regarding the “correct” amount of replication: most people would probably consider that two or three systematic reviews on the same topic with similar eligibility criteria and outcomes is reasonable, whereas four or more would definitely be too many. The question that needs clarification is: when does replication become unnecessary duplication? Unnecessary duplication is usually associated with waste,6 a big problem, particularly when funding comes from a shrinking public purse.
One factor that contributes to duplication is that systematic reviewers may not be aware of what their colleagues are planning. Until recently, there was no single source from which systematic reviewers (and others) could identify existing protocols of ongoing systematic reviews. However, an international prospective register for systematic review protocols now exists (PROSPERO; www.crd.york.ac.uk/PROSPERO/), funded by the National Institute of Health Research and administered through the Centre for Reviews and Dissemination.
The database contains 18 mandatory items and 22 discretionary ones.7 8 This information can be used to search for existing systematic review protocols. At the time of writing, 1871 records of review protocols exist from 65 countries and duplication seems to be rare. This number will probably increase substantially later this summer when Cochrane protocols are added to the register. With increasing international endorsement of the register by journals and funders,9 it is likely to help reduce unnecessary duplication.
Another factor contributing to duplication is the relentless pressure on academics to publish. Many institutions still use the archaic system of publication quantity, rather than quality, as a measure for promotion and tenure. When income and professional advancement may ultimately depend on publication output, systematic reviewers may be unwilling to forgo publication, regardless of whether the review has already been done. Unnecessary duplication will be reduced only when researchers are evaluated on quality, and academic institutions come together, globally, to promote a policy whereby quality trumps quantity.
Expert peer reviewers of content might also be expected to help identify unnecessary duplication. However, in a fast moving clinical field, content experts may have a hard time keeping abreast of their own literature.10 Better training for peer reviewers on ways to identify duplicate reports might help reduce the problem.
The best way to reduce unnecessary duplication of systematic reviews may be to make it compulsory for systematic reviewers to identify existing reviews, either protocols or completed reviews, before conducting their own review. This could form the first step of the review process. After the question has been clearly and precisely formulated,11 systematic reviewers should search for existing systematic review protocols and completed reviews. PROSPERO is a good place to identify existing protocols and the Cochrane Library is an excellent place to identify high quality completed reviews.12
If existing protocols are identified, the review team should review the eligibility criteria and outcomes to see whether they meet their interest or whether there are relevant nuances, such as different patient subgroups or outcomes, that might warrant continuation of the review. Sometimes duplicate reviews are completed to satisfy sensitive political and jurisdictional needs. Even if completed duplicate reviews exist, their methodology might be weak,13 and a new review might be conducted to overcome these deficiencies. Open access publication (or web posting of unpublished reports) will probably help reviewers examine identified protocols and completed reviews more thoroughly.
The PRISMA statement, which aims to help authors improve the reporting of systematic reviews and meta-analyses, does not specifically ask authors to report on existing duplicate reviews. However, item 3 (rationale) of the checklist asks authors to “describe the rationale for the review in the context of what is already known.”14 The PRISMA checklist and its background paper could be more comprehensive, flagging up the problem of unnecessary duplication in its guidance to authors.15
Cite this as: BMJ 2013;347:f5040
Competing interests: I have read and understood the BMJ group policy on declaration of interests and declare the following interests: None.
Provenance and peer review: Commissioned; not externally peer reviewed.