Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.d7202 (Published 03 January 2012) Cite this as: BMJ 2012;344:d7202
- Beth Hart, Doris Duke clinical research fellow and medical student1,
- Andreas Lundh, PhD student2,
- Lisa Bero, professor1
- 1Department of Clinical Pharmacy, Institute for Health Policy Studies, University of California, San Francisco, 3333 California St, Suite 420, San Francisco, CA 94118, USA
- 2Nordic Cochrane Centre, Rigshospitalet and University of Copenhagen, Copenhagen, Denmark
- Correspondence to: L Bero berol{at}pharmacy.ucsf.edu
- Accepted 2 November 2011
Abstract
Objective To investigate the effect of including unpublished trial outcome data obtained from the Food and Drug Administration (FDA) on the results of meta-analyses of drug trials.
Design Reanalysis of meta-analyses.
Data sources Drug trials with unpublished outcome data for new molecular entities that were approved by the FDA between 2001 and 2002 were identified. For each drug, eligible systematic reviews containing at least one meta-analysis were identified by searches of Medline, Embase, and the Cochrane Library in November 2010.
Selection criteria Eligible systematic reviews were done after FDA approval of the drug, were published in English, and had outcomes and comparators that were the same as those of the trials with unpublished FDA trial outcomes, and the characteristics of participants in the systematic reviews were consistent with the FDA approved indication for the drug. Clinical guidelines, conference proceedings, duplicate systematic reviews, and systematic reviews in which included trials were not referenced or that combined trials across multiple drug classes were excluded. Systematic reviews using non-standard meta-analytic techniques (such as Bayesian or network meta-analyses) and those that used inappropriate or invalid methods for calculation of summary statistics (such as unweighted pooled analyses) were also excluded.
Data extraction Two authors independently extracted data from both the published systematic reviews and the FDA’s medical and statistical reviews of the trials submitted to FDA.
Main outcome measure Summary statistics (risk ratios, odds ratios, or weighted mean differences) for relevant outcomes with and without unpublished FDA trial data.
Results 42 meta-analyses (41 efficacy outcomes, one harm outcome) for nine drugs across six drug classes were reanalysed. Overall, addition of unpublished FDA trial data caused 46% (19/41) of the summary estimates from the meta-analyses to show lower efficacy of the drug, 7% (3/41) to show identical efficacy, and 46% (19/41) to show greater efficacy. The summary estimate of the single harm outcome showed more harm from the drug after inclusion of unpublished FDA trial data.
Conclusion The effect of including unpublished FDA trial outcome data varies by drug and outcome. Unpublished FDA trial outcome data should be available and included in meta-analysis. Making these data easily accessible is particularly important because the effects of including unpublished data vary.
Footnotes
We thank Peter Bacchetti for statistical guidance, Kristin Rising for advice and data extraction from the FDA reviews, Yuliya Blyakherova for data collection, Britta Tendal and Asbjørn Hróbjartsson for advice on data analysis, and Erika Campbell for administrative support.
Contributors: BH designed the study, collected and analysed data, and drafted the paper. AL contributed to designing the study, collected and analysed data, and contributed to drafting the paper. LB conceived and designed the study, contributed to analysing data, and drafted the paper. LB is the guarantor.
Funding: This work was supported by a grant from the Doris Duke Charitable Foundation to the University of California, San Francisco, to fund clinical research fellow BH. The sponsor had no role in the design, conduct, or publication of the research. The authors retain full control of all the data.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: This study involves secondary data analysis of publicly available information. Therefore, the University of California, San Francisco Committee on Human Research classifies this research as exempt from review.
Data sharing: All data from this study—including literature searches, additional explanatory material, and data extraction forms—are available on request.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.