Explanatory trials versus pragmatic trialsBMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g6694 (Published 13 November 2014) Cite this as: BMJ 2014;349:g6694
All rapid responses
In the BMJ Endgames “Explanatory versus pragmatic trials”, Sedgwick stated that the example pragmatic trial for leg ulcers undertaken in the community “… would NOT be expected to have high internal validity” (1). Our view is that internal validity as understood by Sedgwick, is not relevant as explanatory trials ask a different question than pragmatic trials.
Internal validity asks whether a trial answers the question it posed. Pragmatic trials ask about real-world, unadorned happenings in the setting of the study, effectiveness. They do not ask about efficacy as in explanatory trials that are carried out under ideal circumstances such as total adherence to the protocol by trial participants and practitioners, total follow-up of trial outcomes and detailed and extensive eligibility criteria. The Sedgwick assertion that pragmatic trials would have a lower internal validity is not justified.
The Cardiovascular Health Awareness Program (CHAP) study was a cluster randomization trial—with the community as the unit of randomization, intervention, analysis and inference—designed to ask the effectiveness question whether an easy to implement community program would have a positive impact on cardiovascular disease problems created by hypertension (2). Thirty-nine communities were selected, stratified by size and geographical location, and randomly allocated to intervention (20 communities) and control arm (19 communities). A detailed description of the design and development of the CHAP trial has been published elsewhere [3-6] along with the main results . The CHAP intervention was followed by a statistically significant 9% relative reduction [RR] in our composite endpoint of hospitalization for acute myocardial infarction (MI), congestive failure or stroke. There were statistically significant reductions favouring the intervention communities in hospital admissions for acute MI and congestive HF, but not for stroke. Extrapolating to the general population aged 65 years, the 9% RR means that the CHAP intervention would result in approximately 5000 fewer hospital admissions for cardiovascular disease annually in the province of Ontario, Canada.
For pragmatic trials, “internal validity” would be compromised if the real-world conditions were altered—that is, if no cross-overs were allowed, patients or clinicians were blinded, or artificial follow-up visits were created to minimize loss-to-follow up. In fact, the trial would then no longer be pragmatic, would not answer the questions it posed and therefore lose “internal validity”. A problem with this criticism is that the critics inappropriately judge the internal validity of a pragmatic trial by applying criteria that are appropriate for explanatory trials; this is a major source of confusion about pragmatic trials. We ensured the internal validity of the Cardiovascular Health Awareness Program (CHAP) trial by preserving the natural process of care in the real-world, and by using other techniques to minimize the potential for a bias in its conclusions.
1. Segwick P: Explanatory trials versus pragmatic trials. BMJ 2014, 349.
2. Kaczorowski J, Chambers LW, Dolovich L, Paterson JM, Karwalajtys T, Gierman T, Farrell B, McDonough B, Thabane L, Tu K, et al: Improving cardiovascular health at population level: 39 community cluster randomised trial of Cardiovascular Health Awareness Program (CHAP). BMJ 2011, 342:d442.
3. Karwalajtys T, McDonough B, Hall H, Guirguis-Younger M, Chambers LW, Kaczorowski J, Lohfeld L, Hutchison B: Development of the volunteer peer educator role in a community Cardiovascular Health Awareness Program (CHAP): a process evaluation in two communities. J Community Health 2009, 34:336-345.
4. Kaczorowski J, Chambers LW, Karwalajtys T, Dolovich L, Farrell B, McDonough B, Sebaldt R, Levitt C, Hogg W, Thabane L, et al: Cardiovascular Health Awareness Program (CHAP): a community cluster-randomised trial among elderly Canadians. Prev Med 2008, 46:537-544.
5. Chambers LW, Kaczorowski J, Dolovich L, Karwalajtys T, Hall HL, McDonough B, Hogg W, Farrell B, Hendriks A, Levitt C: A community-based program for cardiovascular health awareness. Can J Public Health 2005, 96:294-298.
6. . Jones C, Simpson SH, Mitchell D, Haggarty S, Campbell N, Then K, Lewanczuk RZ, Sebaldt RJ, Farrell B, Dolovitch L, et al: Enhancing hypertension awareness and management in the elderly: lessons learned from the Airdrie Community Hypertension Awareness and Management Program (A-CHAMP). Can J Cardiol 2008, 24:561-567.
Competing interests: No competing interests
In his recent BMJ Endgames “Explanatory trials versus pragmatic trials” Sedgwick stated that the example pragmatic trial for leg ulcers undertaken in the community “…would NOT be expected to have high internal validity” .
I disagree. Although others have made similar statements [2, 3] I am not aware of an evidence base supporting (or refuting) the belief that trials that take a more pragmatic approach sacrifice internal validity. The CONSORT extension for pragmatic trials  makes the same demands regarding reporting of information relevant for assessing internal validity as the full CONSORT Statement ; it does not suggest that these can be compromised in a trial taking a pragmatic approach. The only item that is different is blinding - “if blinding was not done, or was not possible, explain why” so even here the expectation is that to ensure internal validity, blinding should be carried out wherever possible.
Rothwell states that internal validity is independent of “all aspects of the design and performance that impact on the external usefulness of the result of a trial”, which he calls the external validity . He also suggests internal validity underpins the rules for randomised trial design and is thus a given whether pragmatic or explanatory design, whereas external validity and applicability requires clinical judgements from medical practitioners who have clinical expertise .
Trialists need to ensure internal validity in all trials, “real world setting” or “ideal setting”.
It is important that we do not confuse confounding with the reality of practice.
1. Sedgwick P: Explanatory trials versus pragmatic trials. BMJ 2014, 349.
2. Ware JH, Hamel MB: Pragmatic trials--guides to better patient care? N Engl J Med 2011, 364:1685-1687.
3. Ernst E, Canter PH: Limitations of "pragmatic" trials. Postgrad Med J 2005, 81:203.
4. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D: Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008, 337:a2390.
5. Rothwell PM: Commentary: External validity of results of randomized trials: disentangling a complex concept. Int J Epidemiol 2010, 39:94-96.
Competing interests: No competing interests