Credibility of claims of subgroup effects in randomised controlled trials: systematic reviewBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e1553 (Published 15 March 2012) Cite this as: BMJ 2012;344:e1553
- Xin Sun, assistant professor12,
- Matthias Briel, assistant professor23,
- Jason W Busse, scientist24,
- John J You, assistant professor25,
- Elie A Akl, associate professor26,
- Filip Mejza, research fellow7,
- Malgorzata M Bala, research fellow8,
- Dirk Bassler, associate professor9,
- Dominik Mertz, assistant professor2510,
- Natalia Diaz-Granados, doctoral candidate2,
- Per Olav Vandvik, researcher1112,
- German Malaga, associate professor13,
- Sadeesh K Srinathan, assistant professor14,
- Philipp Dahm, professor15,
- Bradley C Johnston, assistant professor216,
- Pablo Alonso-Coello, researcher17,
- Basil Hassouneh, research fellow2,
- Stephen D Walter, professor2,
- Diane Heels-Ansdell, statistician2,
- Neera Bhatnagar, librarian18,
- Douglas G Altman, professor19,
- Gordon H Guyatt, professor2
- 1Center for Health Research, Kaiser Permanente Northwest, Portland, OR, USA
- 2Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, 1280 Main Street West, Hamilton, ON, Canada, L8S 4K1
- 3Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Switzerland
- 4Institute for Work and Health, Toronto, ON, Canada
- 5Department of Medicine, McMaster University, Hamilton, ON, Canada
- 6Departments of Medicine and Family Medicine, State University of New York at Buffalo, NY, USA
- 7Department of Pulmonary Diseases, Jagiellonian University School of Medicine, Krakow, Poland
- 8II Department of Internal Medicine, Jagiellonian University School of Medicine, Krakow, Poland
- 9Department of Neonatology and Center for Pediatric Clinical Studies, University Children’s Hospital Tuebingen, Tuebingen, Germany
- 10Michael G DeGroote Institute for Infectious Diseases Research, McMaster University, Hamilton, Canada
- 11Norwegian Knowledge Centre for the Health Services, Oslo, Norway
- 12Department of Medicine, Innlandet Hospital Trust, Gjøvik, Norway
- 13Department of Medicine, Universidad Peruana Cayetano Heredia, Lima, Peru
- 14Department of Surgery, University of Manitoba, Winnipeg, MB, Canada
- 15Department of Urology, University of Florida, College of Medicine, Gainesville, FL, USA
- 16Department of Anaesthesiology and Pain Medicine, The Hospital for Sick Children, Toronto, Ontario, Canada
- 17IberoAmerican Cochrane Centre, Biomedical Research Institute-CIBER of Epidemiology and Public Health, Barcelona, Spain.
- 18Health Sciences Library, McMaster University, Hamilton, ON, Canada
- 19Centre for Statistics in Medicine, University of Oxford, Oxford, UK
- Correspondence to: G H Guyatt
- Accepted 19 January 2012
Objective To investigate the credibility of authors’ claims of subgroup effects using a representative sample of recently published randomised controlled trials.
Design Systematic review.
Data source Core clinical journals, as defined by the National Library of Medicine, in Medline.
Study selection Randomised controlled trials published in 2007. Using prespecified criteria, teams of trained reviewers independently judged whether authors claimed subgroup effects and the strength of their claims. Reviewers assessed each of these claims against 10 predefined criteria, developed through a search of existing criteria and a consensus process.
Results Of 207 randomised controlled trials reporting subgroup analyses, 64 (31%) made claims for the primary outcome. Of those, 20 were strong claims and 28 claims of a likely effect. Authors included subgroup variables measured at baseline in 60 (94%) trials, used subgroup variable as a stratification factor at randomisation in 13 (20%), clearly prespecified their hypotheses in 26 (41%), correctly prespecified direction in 4 (6%), tested a small number of hypotheses in 28 (44%), carried out a test of interaction that proved statistically significant in 6 (9%), documented replication of a subgroup effect with previous related studies in 21 (33%), identified consistency of a subgroup effect across related outcomes in 19 (30%), and provided a compelling indirect evidence for the effect in 14 (22%). In the 19 trials making more than one claim, only one (5%) checked the independence of the interaction. Of the 64 claims, 54 (84%) met four or fewer of the 10 criteria. For strong claims, more than 50% failed each of the individual criteria, and only three (15%) met more than five criteria.
Conclusion Authors often claim subgroup effects in their trial report. However, the credibility of subgroup effects, even when claims are strong, is usually low. Users of the information should treat claims that fail to meet most criteria with scepticism. Trial researchers should report the conduct of subgroup analyses and provide sufficient evidence when claiming a subgroup effect or suggesting a possible effect.
We thank Jessica Truong and Neil Dattani for screening part of the study abstracts and title; Monica Owen for administrative assistance; and Aravin Duraikannan for developing the electronic data abstraction forms.
Contributors: XS and GHG conceived the study, had full access to all of the data in the study, and take responsibility for the integrity of the data and the accuracy of the data analysis. GHG is the guarantor. XS, GHG, MB, JWB, EAA, SDW, and DGA designed the study. MB, EAA, JWB, ND-G, JJY, FM, MMB, DB, DM, POV, GM, SKS, PD, BCJ, PA-C, BH, XS, J Truong, N Dattani, and NB acquired the data. XS, GHG, SDW, and DH-A analysed and interpreted the data. XS drafted the manuscript. XS, JWB, MB, GHG, JJY, BH, EAA, DB, DM, POV, BCJ, PD, SDW, DGA, GM, FM, DH-A, ND-G, PA-C, SKS, MMB, and NB critically revised the manuscript. XS provided administrative, technical, and material support.
Funding: This study was supported by the National Natural Science Foundation of China (project No: 70703025). The funder had no role in the study design, writing of the manuscript, or decision to submit this or future manuscripts for publication. MB is supported by Santésuisse and the Gottfried and Julia Bangerter-Rhyner Foundation. JWB is funded by a new investigator award from the Canadian Institutes of Health Research and the Canadian Chiropractic Research Foundation. DB is supported by the European Union (grant award health-F5-2009-223060). DM is supported by a research scholarship from the Swiss National Science Foundation (PBBSP3-124436 and PASMP3-132571) and the Lichtenstein-Stiftung, Basel, Switzerland. PD is supported by a Dennis W Jahnigan Career Development Award by the American Geriatrics Society. BCJ holds a SickKids Foundation postdoctoral fellowship. PA-C is funded by a Miguel Servet contract by the Instituto de Salud Carlos III (CP09/00137). JJY is supported by a career scientist award from the Ontario Ministry of Health and Long-Term Care. DGA is supported by Cancer Research UK (grant No C5529).
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, and no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.