We thank Perol and colleagues for their reflections of our work on how RCD might complement RCTs. We agree that RCD may be used to analyze any number of associations for combinations of medical interventions, patient characteristics, settings, and concurrent treatments. They may be very well useful to inform further research activities and generate hypotheses. Nevertheless, the concomitant problems of spurious findings and notoriously misleading subgroup effects are well described (1-4), and Perol et al. highlight that there are numerous novel issues and types of bias associated with routine data, which we need to better understand. This further underlines our call for caution. Our empirical findings do not increase our confidence in findings from RCD on treatment effects in general. Much more research and empirical evidence is necessary to determine what makes RCD more or less “robust”.
There is, however, much evidence that pragmatic RCTs are possible which evaluate complex interventions and strategies and allow clarifying whether a medical intervention is beneficial “in the context of all ongoing medical interventions that occurred in the same patients throughout the study period, whether they were dedicated to the underlying disease itself, to supportive care, or to other concomitant conditions.”
We agree that RCD offer the opportunity to complement randomized evidence and address evidence gaps that are unlikely to be closed by RCTs, for various reasons (1). However, we believe that many of these situations would be better addressed by an improved RCT research agenda.
References:
1. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016 Feb 16. pii: cmaj.150653.
2. Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000;355(9209):1064-1069.
3. Bhandari M, Devereaux PJ, Li P, et al. Misuse of baseline comparison tests and subgroup analyses in surgical trials. Clin Orthop Relat Res. 2006;447:247-251.
4. Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials. N Engl J Med. 2007;357:2189-2194.
Competing interests:
No competing interests
29 September 2016
Lars G. Hemkens
Senior Scientist
Despina G. Contopoulos-Ioannidis, John P.A. Ioannidis
Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel
Rapid Response:
The authors respond
We thank Perol and colleagues for their reflections of our work on how RCD might complement RCTs. We agree that RCD may be used to analyze any number of associations for combinations of medical interventions, patient characteristics, settings, and concurrent treatments. They may be very well useful to inform further research activities and generate hypotheses. Nevertheless, the concomitant problems of spurious findings and notoriously misleading subgroup effects are well described (1-4), and Perol et al. highlight that there are numerous novel issues and types of bias associated with routine data, which we need to better understand. This further underlines our call for caution. Our empirical findings do not increase our confidence in findings from RCD on treatment effects in general. Much more research and empirical evidence is necessary to determine what makes RCD more or less “robust”.
There is, however, much evidence that pragmatic RCTs are possible which evaluate complex interventions and strategies and allow clarifying whether a medical intervention is beneficial “in the context of all ongoing medical interventions that occurred in the same patients throughout the study period, whether they were dedicated to the underlying disease itself, to supportive care, or to other concomitant conditions.”
We agree that RCD offer the opportunity to complement randomized evidence and address evidence gaps that are unlikely to be closed by RCTs, for various reasons (1). However, we believe that many of these situations would be better addressed by an improved RCT research agenda.
References:
1. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016 Feb 16. pii: cmaj.150653.
2. Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000;355(9209):1064-1069.
3. Bhandari M, Devereaux PJ, Li P, et al. Misuse of baseline comparison tests and subgroup analyses in surgical trials. Clin Orthop Relat Res. 2006;447:247-251.
4. Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials. N Engl J Med. 2007;357:2189-2194.
Competing interests: No competing interests