ReviewRealist randomised controlled trials: A new approach to evaluating complex public health interventions
Highlights
► ‘Realists’ argue that RCTs ignore the complexity of causation and fail to ask which interventions work, for whom and when. ► We disagree but believe RCTs should examine for whom and when interventions work. ► To this end, RCTs could aim to examine intervention mechanisms of change and how intervention effects vary with context. ► “Realist” RCTs should examine the validity of intervention theory to better inform policy and practice in the long term.
Introduction
In this paper, we outline problems with the way complex public health interventions are sometimes evaluated using randomized controlled trials (RCTs) before examining ‘realist’ critiques of and proposed alternatives to RCTs. Realism in evaluation represents a paradigm through which the world is seen as an open system of dynamic structures, mechanisms and contexts that intricately influence the change phenomena that evaluations aim to capture (Kazi, 2003). Realistic evaluators argue that RCTs fail to test hypotheses rooted in theory and embrace a crude notion of causality based on comparison groups and statistical association rather than understanding mechanisms. They argue that evaluators must develop a priori theories about how, for whom and under what conditions interventions will work and then use observational data to examine how context and intervention mechanism interact to generate outcomes. While we dispute the realists' rejection of experimental designs in the social sciences (Pawson & Tilley, 1997), we agree with their arguments concerning the need for evaluation: to examine how, why and for whom interventions work; to give more attention to context; and to focus on the elaboration and validation of program theory. Some previous authors (Blackwood, O'Halloran et al., 2010) have argued for a synergistic, rather than oppositional, relationship between realist and randomized evaluation:
The RCT can be used to ascertain whether, all other things being equal, a particular causal mechanism (intervention) is efficacious [i.e. effective under optimum conditions], while realistic evaluation can establish what effect the interaction of other mechanisms operating in the open contexts studied has upon its effectiveness, and identify which mechanisms promote, and which inhibit that effectiveness (Blackwood et al., 2010, p. 519).
We would go further than this, however, to propose that RCTs themselves could contribute to a realist approach to evaluation. We examine the extent to which some RCTs are already embracing many of these issues and, bringing together some of these existing innovations alongside our own ideas, sketch out what ‘realist RCTs’ might look like. We argue that it is possible to benefit from the insights provided by realist evaluation without relinquishing the RCT as the best means of examining intervention causality.
Section snippets
Current approaches to evaluating complex public health interventions
RCTs aim to generate minimally biased estimates of intervention effects by ensuring that intervention and control groups are not systematically different from each other in terms of measured and/or unmeasured characteristics. RCTs may randomly allocate individuals or ‘clusters’ of individuals, such as schools or villages; a method that should ensure that the groups are similar other than differences that occur due to chance. Random allocation is widely regarded as ethical if there is
The ‘realist’ critique of, and alternative to, RCTs
The realist tradition provides a critique of, and apparent alternative to, the RCT tradition, but we argue it might be more useful in informing how RCTs could be modified to be more useful in the evaluation of complex public health interventions. Realism asserts that the objects of knowledge exist independently of our minds and that unobservable objects and structures can exert causal influence (Bhaskar, 1975); in contrast to a positivist tradition, which would limit science to directly
Problems with the ‘realist’ position
One might question Pawson and Tilley's claim that the use of RCTs inevitably embodies a set of strong epistemological and ontological commitments, such as the positivistic account of scientific knowledge and the ‘successionist’ theory of causality (Bonell et al., 2003). However, we will not address this broader point here. Instead, firstly, we will concentrate on Pawson and Tilley's critique of the use of counterfactuals to examine causality. Pawson and Tilley are incorrect in arguing that
A synthesis: realist RCTs of complex public health interventions
What might a ‘realist’ program of RCTs look like? We set out some suggestions below illustrated by cases in which RCTs are already employing these approaches, and we suggest how these may be further enhanced.
First, realist RCTs would place emphasis on understanding the effects of intervention components separately as well as in combination. For example, this could involve use of ‘multi-arm studies’ with various combinations of intervention components in each arm. Factorial trials (Montgomery,
Conclusions
This paper has scrutinized apparent tensions between RCT and realist approaches to evaluating complex public health interventions and proposed opportunities for synergy between the two. Some argue against anything other than an approach to complex interventions being experimentally evaluated and those found to be effective being replicated with maximum fidelity, because we have so little understanding of key causal mechanisms (Kemple & Willner, 2008). However, given the uncertain and mixed
References (72)
- et al.
Pilot multi-method trial of a school-ethos intervention to reduce substance use: building hypotheses about upstream pathways to prevention
Journal of Adolescent Health
(2010) - et al.
An informal school-based peer-led intervention for smoking prevention in adolescence (ASSIST): a cluster randomised trial
The Lancet
(2008) - et al.
School effects on young people's drug use: a systematic review of intervention and observational studies
Journal of Adolescent Health
(2008) - et al.
Patients or research subjects? A qualitative study of participation in a randomised controlled trial of a complex intervention
Patient Education and Counseling
(2006) - et al.
A proposal to speed translation of healthcare research into practice: dramatic change is needed
American Journal of Preventive Medicine
(2011) - et al.
Empowerment-based non-formal education for Arab youth: a pilot randomized trial
Children and Youth Services Review
(2012) - et al.
Effect of a structural intervention for the prevention of intimate-partner violence and HIV in rural South Africa: a cluster randomised trial
The Lancet
(2006) - et al.
Improving the reporting of public health intervention research: advancing TREND and CONSORT
Journal of Public Health
(2008) - et al.
The role of therapist adherence, therapist competence and allowance in predicting outcome of individual drug counselling: results from the national institute drug abuse collaborative cocaine treatment study
Psychotherapy Research
(2006) - et al.
The moderator-mediator variable distinction in social psychological research: conceptual, strategic and statistical considerations
Journal of Personality and Social Psychology
(1986)
Changed lives: The effects of the Perry Preschool Program on youths through age 19
A realist theory of science
On the problems of mixing RCTs with qualitative research: the case of the MRC framework for the evaluation of complex healthcare interventions
Journal of Research in Nursing
The fidelity-adaptation debate: implications for the implementation of public sector social programs
American Community Psychology
Sexual health interventions should be subject to experimental evaluation
Alternatives to randomisation in the evaluation of public-health interventions: design challenges and solutions
Journal of Epidemiology and Community Health
Trials of health interventions and empirical assessment of generalizability: suggested framework and systematic review
British Medical Journal
Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population
Journal of the American Medical Association
Evolutionary operation: a method for increasing industrial productivity
Applied Statistics
Implementation fidelity in community-based interventions
Research in Nursing & Health
Positive youth development in the United States: research findings on evaluations of positive youth development programs
Prevention & Treatment
Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems
Developing and evaluating complex interventions: the new Medical Research Council guidance
British Medical Journal
Program integrity in primary and early secondary prevention: are implementation effects out of control?
Clinical Psychology Review
A factorial-design cluster randomised controlled trial investigating the cost-effectiveness of a nutrition supplement and an exercise programme on pneumonia incidence, walking capacity and body mass index in older people living in Santiago, Chile: the CENEX study protocol
Nutrition Journal
When interventions harm
American Psychologist
Why intervention implementation is important
Journal of Prevention and Intervention in the Community
Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation
American Journal of Community Psychology
Community based programs for youth: lessons learned from general development research and from experimental and quasi-experimental evaluations
Bridging science to practice: achieving prevention program implementation fidelity in the community youth development study
American Journal of Community Psychology
School-based programs to reduce bullying and victimization
Campbell Systematic Reviews
Effects of 2 prevention programs on high-risk behaviors among African American youth: a randomized trial
Archives of Pediatric and Adolescent Medicine
Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: outcomes and mechanisms of change
Journal of Child Psychology and Psychiatry
Evaluating the overall impact of health promotion programs: using the RE-AIM framework for decision making and to consider complex issues
Health Education Research
Cited by (370)
Complex interventions in frail older adults
2024, Archives of Gerontology and GeriatricsFirst, do no harm? Dark logic models, social injustice, and the prevention of iatrogenic conservation outcomes
2024, Biological ConservationAn introduction to realist evaluation and synthesis for kidney research
2024, Kidney InternationalPsychosocial interventions to improve the mental health of survivors of human trafficking: a realist review
2023, The Lancet PsychiatryTheory-based evaluation and programme theories in nursing: A discussion on the occasion of the updated Medical Research Council (MRC) Framework
2023, International Journal of Nursing StudiesEvaluating population health interventions: The contributions of theory-driven evaluations
2023, Revue d'Epidemiologie et de Sante Publique