Ideas to improve the knowledge system stimulated by BMJ Rapid Responses
Roberts and colleagues reported that systematic reviews are often biased, out of date, and unreadably long and proposed multiple ideas (including limiting evaluation to only prospectively registered trials) as a solution. Cochrane leadership responded noting the challenges associated with the specific proposals, and that many might not work “as suggested” but the ideas behind them may stimulate and contribute to solutions, such as further ideas shared by Gluud and colleagues.[2,3] We recommend more direct considerations for solving these challenges and suggest a new approach for Cochrane reviews related to this new idea.
Because the medical literature is highly biased for so many reasons, reviewers often need to be more critical of the evidence and not accept conclusions and statements from publications without critical appraisal. A large amount of high-quality work to produce a systematic review can be wasted by introducing a single concept uncritically in the subsequent discussion and conclusion. We need to be critical regarding all information sources and data, but we also need to be critical of our critical appraisal. Claiming a risk of bias based on a technicality that is not likely to have any influence on results is of limited benefit. We need to be vigilant for “true risk of bias” and not slight variations with potentially minimal impact.
Many reviews are out of date but removing substantial portions of the evidence to only view the “very best” may not provide the best summary of the body of evidence. The problem being addressed by Roberts and colleagues is that the workload to produce reviews and keep them current is higher than the resources applied to it in our current workflow. More efficient strategies for keeping current, critical appraisal, and data sharing are better solutions for overcoming this workflow challenge as opposed to ignoring some of the work.
Many reviews are unreadably long. However, reducing the reading length by eliminating the number of included studies negates the concept of being comprehensive in what is reviewed. A focus on what is needed for reading for a particular purpose can guide what and how to present for reading, and a focus on what is needed for documentation for transparency can guide what facts, data and analyses are presented for looking up for detail checking.
Getting back to the statement that “Including only prospectively registered trials in systematic reviews will improve validity and readability” – This is not true. If the best evidence is not prospectively registered the validity will be reduced by excluding it. Even if unregistered trials are not the best evidence but provide inconsistent results compared to registered trials, ignoring the unregistered trials could provide a less valid answer that appears more valid by missing this inconsistency in the body of evidence.
The concept that absence of prospective registration of trials represents a risk of bias, and that systematic reviews may be biased by uncritically accepting this bias in the evidence, come together for a new consideration. Consider expanding the Risk of Bias domains in Cochrane reviews to include additional sources of bias with explicit criteria rather than subsuming these under the domain of ‘Other bias’ which are often left to the discretion of reviewers who may make less systematic judgments regarding which risks of bias to assess and report. Adding more Risk of Bias domains has implications on the work of change (work to test, work to change systems, work to educate) and implications on the work of conducting systematic reviews. However, considering a greater set of Risk of Bias domains that are practical to include and meaningful for representing important bias (and changes in the medical evidence production and publishing environment may mean this is not always constant) may be a substantial improvement that helps apply the suggestion by Roberts and colleagues to meet the primary problem they reported. 'Prospective registration' could be considered by the Cochrane Bias Methods Group and the Cochrane Handbook editors, as well as other risks commonly included among the 'Other' Risk of Bias domains for review authors to consider. A drop-down list in RevMan could facilitate the selection of this and other relevant 'Other' risk of bias domains for review authors to select, standardize on terminology, and facilitate subgroup analyses in individual reviews and meta-epidemiologic research across reviews.
1. Roberts I, Ker K, Edwards P, Beecher D, Manno D, Sydenham E: The knowledge system underpinning healthcare is not fit for purpose and must change. BMJ 2015 Jun 3;350:h2463. http://www.bmj.com/content/350/bmj.h2463
2. Tovey DI, Bero L, Farquhar C, Lasserson T, MacLehose H, Macdonald G, et al. A response to Ian Roberts and his colleagues. Rapid response 2015 Jun 7 to BMJ 2015;350:h2463. http://www.bmj.com/content/350/bmj.h2463/rr-3
3. Gluud C, Jakobsen JC, Imberger G, Lange T, Wetterslev J. Re: The knowledge system underpinning healthcare is not fit for purpose and must change - Responses to the opposing viewpoints of Roberts and colleagues and Tovey and colleagues. Rapid response 2015 Jun 12 to BMJ 2015;350:h2463. http://www.bmj.com/content/350/bmj.h2463/rr-6
Brian S. Alper, MD, MSPH, FAAFP (firstname.lastname@example.org) – Founder of DynaMed; Vice President of EBM Research and Development, Quality and Standards, EBSCO Health; Ipswich, Massachusetts, USA; Clinical Research Assistant Professor, Department of Family and Community Medicine, University of Missouri School of Medicine, Columbia, Missouri, USA
Esther J. van Zuuren, MD - Leiden University Medical Centre, Leiden, the Netherlands
Zbys Fedorowicz, MSc, DPH, BDS, LDS, RCS - Advisor to the Supreme Council of Health, Manama, Kingdom of Bahrain; Director for The Bahrain Branch, The Cochrane Collaboration, Bahrain
Eric Manheimer, PhD – Medical Writer, DynaMed (anticipated)
No direct competing interests. All authors are employees or contractors for EBSCO Health which publishes DynaMed (www.dynamed.com) and DynaMed provides evidence summaries which may be used by systematic review authors to reduce the risk of bias and being out of date.
Competing interests: No direct competing interests. All authors are employees or contractors for EBSCO Health which publishes DynaMed (www.dynamed.com) and DynaMed provides evidence summaries which may be used by systematic review authors to reduce the risk of bias and being out of date.