Living network meta-analysis compared with pairwise meta-analysis in comparative effectiveness research: empirical study
BMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k585 (Published 28 February 2018) Cite this as: BMJ 2018;360:k585
- Adriani Nikolakopoulou, research associate in biostatistics1,
- Dimitris Mavridis, assistant professor of statistics2 3,
- Toshi A Furukawa, professor of clinical epidemiology4,
- Andrea Cipriani, associate professor of psychiatry5 6,
- Andrea C Tricco, research scientist7 8,
- Sharon E Straus, professor of medicine7 9,
- George C M Siontis, consultant cardiologist10,
- Matthias Egger, professor of epidemiology and public health1,
- Georgia Salanti, associate professor of biostatistics and epidemiology1
- 1Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland
- 2Department of Primary Education, University of Ioannina, Ioannina, Greece
- 3Centre de Recherche Épidémiologie et Statistique Sorbonne Paris Cité, Inserm/Université Paris Descartes, Paris, France
- 4Departments of Health Promotion and Human Behavior and of Clinical Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
- 5Department of Psychiatry, University of Oxford, Warneford Hospital, Oxford, UK
- 6Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, UK
- 7Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Ontario, Canada
- 8Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- 9Department of Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- 10Department of Cardiology, Bern University Hospital, Bern, Switzerland
- Correspondence to: G Salanti georgia.salanti{at}ispm.unibe.ch
- Accepted 22 January 2018
Abstract
Objective To examine whether the continuous updating of networks of prospectively planned randomised controlled trials (RCTs) (“living” network meta-analysis) provides strong evidence against the null hypothesis in comparative effectiveness of medical interventions earlier than the updating of conventional, pairwise meta-analysis.
Design Empirical study of the accumulating evidence about the comparative effectiveness of clinical interventions.
Data sources Database of network meta-analyses of RCTs identified through searches of Medline, Embase, and the Cochrane Database of Systematic Reviews until 14 April 2015.
Eligibility criteria for study selection Network meta-analyses published after January 2012 that compared at least five treatments and included at least 20 RCTs. Clinical experts were asked to identify in each network the treatment comparison of greatest clinical interest. Comparisons were excluded for which direct and indirect evidence disagreed, based on side, or node, splitting test (P<0.10).
Outcomes and analysis Cumulative pairwise and network meta-analyses were performed for each selected comparison. Monitoring boundaries of statistical significance were constructed and the evidence against the null hypothesis was considered to be strong when the monitoring boundaries were crossed. A significance level was defined as α=5%, power of 90% (β=10%), and an anticipated treatment effect to detect equal to the final estimate from the network meta-analysis. The frequency and time to strong evidence was compared against the null hypothesis between pairwise and network meta-analyses.
Results 49 comparisons of interest from 44 networks were included; most (n=39, 80%) were between active drugs, mainly from the specialties of cardiology, endocrinology, psychiatry, and rheumatology. 29 comparisons were informed by both direct and indirect evidence (59%), 13 by indirect evidence (27%), and 7 by direct evidence (14%). Both network and pairwise meta-analysis provided strong evidence against the null hypothesis for seven comparisons, but for an additional 10 comparisons only network meta-analysis provided strong evidence against the null hypothesis (P=0.002). The median time to strong evidence against the null hypothesis was 19 years with living network meta-analysis and 23 years with living pairwise meta-analysis (hazard ratio 2.78, 95% confidence interval 1.00 to 7.72, P=0.05). Studies directly comparing the treatments of interest continued to be published for eight comparisons after strong evidence had become evident in network meta-analysis.
Conclusions In comparative effectiveness research, prospectively planned living network meta-analyses produced strong evidence against the null hypothesis more often and earlier than conventional, pairwise meta-analyses.
Footnotes
Contributors: GS and ME conceived and designed the study. TAF, AC, ACT, SES, and GCMS assisted in the design of the study, indicated the relevant comparisons of highest interest for their specialties of expertise, and provided a relevant reference to guidelines. AN compiled the database of eligible networks and performed the analysis. AN drafted the manuscript. TAF and AC assessed network meta-analyses pertaining to mental health, ACT and SES assessed networks in the specialties of endocrinology, dermatology, gastroenterology, obstetrics, oncology, anaesthesiology, and hepatology, and GCMS assessed network meta-analyses from the specialty of cardiology. For networks examining other conditions, external medical doctors or experienced researchers were approached. All authors critically revised the manuscript, interpreted the results, and performed a critical review of the manuscript for intellectual content. GS and ME produced the final version of the submitted article, and all coauthors approved it. AN and GS had full access to all data in the study and take responsibility for the integrity of the data and the accuracy of the analysis. GS and ME are the guarantors.
Funding: ACT is funded by a tier 2 Canada research chair in knowledge synthesis. SES is funded by a tier 1 Canada research chair in knowledge translation. GS received funding from a Horizon 2020 Marie-Curie individual fellowship (grant No 703254). AC is supported by the National Institute for Health Research Oxford cognitive health clinical research facility. The sponsors had no influence on the design, analysis, and reporting of this study, neither on the preparation of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; TAF has received lecture fees from Eli Lilly, Janssen, Meiji, Mitsubishi-Tanabe, MSD, and Pfizer and consultancy fees from Takeda Science Foundation. He has received research support from Mochida and Mitsubishi-Tanabe; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: List of included studies and details about the data are reported in the appendix. Study level data and R codes are available in a GitHub repository. Instructions to access them can be found in appendix N.
Transparency: The lead author (GS) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.