Sex based subgroup differences in randomized controlled trials: empirical evidence from Cochrane meta-analysesBMJ 2016; 355 doi: https://doi.org/10.1136/bmj.i5826 (Published 24 November 2016) Cite this as: BMJ 2016;355:i5826
- Joshua D Wallach, doctoral student in epidemiology and clinical research1,
- Patrick G Sullivan, resident physician1,
- John F Trepanowski, postdoctoral fellow2,
- Ewout W Steyerberg, professor3,
- John P A Ioannidis, professor4
- 1Department of Health Research and Policy, and Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA
- 2Stanford Prevention Research Center, Stanford University, Stanford, CA, USA
- 3Department of Public Health, Erasmus MC, Rotterdam, Netherlands
- 4Departments of Medicine, Health Research and Policy, and Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, CA 94305, USA
- Correspondence to: J P A Ioannidis
- Accepted 20 October 2016
Objective To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses.
Design Meta-epidemiological study.
Data sources Cochrane Database of Systematic Reviews (CDSR) and PubMed.
Eligibility criteria for study selection Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials.
Data extraction Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered.
Results Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses (“topics”). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions.
Conclusion Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.
We thank Ruth Foxlee (Information Specialists, Cochrane Editorial Unit, Cochrane Central Executive) and Rasmus Moustgaard (Cochrane Informatics and Knowledge Management Department) for searching the Cochrane Database of Systematic Reviews (CDSR) and providing guidance on CDSR search procedures.
Contributors: JDW, PGS, JFT, EWS, and JPAI designed the study. JDW, PGS, and JT screened the Cochrane reviews for eligibility. JDW extracted the data. PGS and JFT checked all extracted data for accuracy. JDW and PGS performed the statistical analysis. JDW, PGS, JFT, and JPAI interpreted the results. JDW wrote the first draft of the manuscript, with major contributions from PGS, JFT, and JPAI. All authors made revisions to the manuscript and have read and approved the final version. JPAI is the guarantor. All authors had full access to all of the data.
Funding: METRICS (Meta-Research Innovation Center at Stanford) is supported by a grant from the Laura and John Arnold Foundation. PGS received support from the Stanford Clinical and Translational Science Award to Spectrum (NIH UL1 TR 001085). JFT is supported by the NIH (T32 HL007034). EWS is partly supported by the NIH (PRICES project, U01 NS086294). JPAI is supported by an unrestricted gift by Sue and Bob O’Donnell to Stanford Prevention Research Center. The research was conducted independent of any involvement of the funders. Funders were not involved in any aspect of the study design, data collection, data interpretation, writing, or the decision to submit the article for publication.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: METRICS (Meta-Research Innovation Center at Stanford) is supported by a grant from the Laura and John Arnold Foundation, PGS received support from the Stanford Clinical and Translational Science Award to Spectrum (NIH UL1 TR 001085), JFT is supported by the NIH (T32 HL007034), EWS is partly supported by the NIH (PRICES project, U01 NS086294), and JPAI is supported by an unrestricted gift by Sue and Bob O’Donnell to Stanford Prevention Research Center; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: Data are available from the lead author on request.
Transparency: The guarantor (JPAI) affirms that the manuscript is a honest, accurate, and transparent account of the study bring reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.