BMJ  2006;332:413-416 (18 February), doi:10.1136/bmj.332.7538.413

Analysis and comment

Health services research

Process evaluation in randomised controlled trials of complex interventions

¹ Social Science Research Unit, Institute of Education, University of London, London WC1H ONR, ² Public and Environmental Health Research Unit, London School of Hygiene and Tropical Medicine, London WC1E 7HT, ³ Centre for Sexual Health and HIV Research, London WC1E 6AU

Most randomised controlled trials focus on outcomes, not on the processes involved in implementing an intervention. Using an example from school based health promotion, this paper argues that including a process evaluation would improve the science of many randomised controlled trials

"Complex interventions" are health service interventions that are not drugs or surgical procedures, but have many potential "active ingredients."¹ A complex intervention combines different components in a whole that is more than the sum of its parts.² Randomised controlled trials (RCTs) are the most rigorous way to evaluate the effectiveness of interventions, regardless of their complexity. Because of their multifaceted nature and dependence on social context, complex interventions pose methodological challenges, and require adaptations to the standard design of such trials.³ This paper outlines a framework for using process evaluation as an integral element of RCTs. It draws on experience from a cluster randomised trial of peer led sex education.

Why evaluate processes?

Conventional RCTs evaluate the effects of interventions on prespecified health outcomes. The main question is, "Does it work?" Process evaluations within trials explore the implementation, receipt, and setting of an intervention and help in the interpretation of the outcome results. They may aim to examine the views of participants on the intervention; study how the intervention is implemented; distinguish between components of the intervention; investigate contextual factors that affect an intervention; monitor dose to assess the reach of the intervention; and study the way effects vary in subgroups.⁴ Process evaluation can help to distinguish between interventions that are inherently faulty (failure of intervention concept or theory) and those that are badly delivered (implementation failure).⁵ Process evaluations are especially necessary in multisite trials, where the "same" intervention may be implemented and received in different ways.

Although "process" and "qualitative" are often used interchangeably, data for process evaluation can be both quantitative and qualitative. For example, in a series of systematic reviews of research on health promotion and young people, 33 (62%) of 53 process evaluations collected only quantitative data, 11 (21%) collected only qualitative data, and nine (17%) collected both.⁶ Process evaluation in RCTs is common in health promotion and public health, but often data are not collected systematically and are used mostly for illustration.⁵

Baby talk: the most effective sex education? Credit: TONY KYRIACOU/REX

 

Process evaluation in the RIPPLE study

The RIPPLE (randomised intervention of pupil peer led sex education) study is a cluster RCT designed to investigate whether peer delivered sex education is more effective than teacher delivered sessions at decreasing risky sexual behaviour. It involves 27 English secondary schools and follow-up to age 19. In the schools randomised to the experimental arm, pupils aged 16-17 years (given a brief training by an external team) delivered the programme to two successive year cohorts of 13-14 year olds. Control schools continued with their usual teacher led sessions. The trial was informed by a systematic review and pilot study in four schools.^{7 8} Its design by a multidisciplinary research team includes an integral process evaluation that has four aims: to document the implementation of the peer led intervention and sex education in control schools; to describe and compare processes in the two forms of sex education; to collect information from study participants (schools and students) about the experience of taking part in the trial; and to collect data on individual school contexts.⁹

Box 1: Methods for data collection in the RIPPLE trial process evaluation Questionnaire surveys of students and peer educators Focus groups with students and peer educators Interviews with teachers Researcher observation of peer led and teacher led sex education

Several methods were used to collect process data (box 1), including questionnaire surveys, focus groups, interviews, researcher observations, and structured field notes. Some methods, such as questionnaire surveys, were also used to collect outcome data. Other methods, such as focus groups and interviews with school staff and peer educators, were specific to process evaluation.

The outcome results by age 16 showed that the peer led approach improved some knowledge outcomes; increased satisfaction with sex education; and reduced intercourse and increased confidence about the use of condoms in girls. Girls in the peer led arm reported lower confidence about refusing unwanted sexual activity (borderline significance). The incidence of intercourse before age 16 in boys and of unprotected first sex, regretted first intercourse (or other quality measure of sexual experiences or relationships), confidence about discussing sex or contraception, and some knowledge outcomes for both girls and boys were not affected.¹⁰

We analysed process data in two stages. Before analysing outcome data (box 2), we analysed the data to answer questions arising from the aims outlined above. The hypotheses generated were tested in statistical analyses that integrated process and outcome data to address three questions:

• What was the relation between trial outcomes and variation in the extent and quality of the implementation of the intervention?
  
• What processes might mediate the observed relation between intervention and outcomes?
  
• Did subgroups of students and schools differ in their responses to the intervention?
  

We used several strategies to combine the different types of data. These included on-treatment analyses, in which results for students who received the peer led intervention were compared with results from the standard intention to treat approach (where allocation to, rather than receipt of, the intervention forms the basis of the analysis). We then carried out regression analyses and, where appropriate, tests for interactions, to examine the relation between key dimensions of sex education, subgroups of schools and students most and least likely to benefit from the peer led programme, and study outcomes (further details are available elsewhere).^{11 12} We used regression analyses because they are best for dealing with many types of outcome, assessing the impact of mediating factors, and analysing subgroups.^{13 14}

These strategies provided answers to our three questions.

• More consistent implementation of the peer led programme might have had a greater impact on several knowledge outcomes and reduced the proportion of boys having sex by age 16, but it would not have changed other behavioural outcomes.
  
• There were key interactions between the extent to which sex education is participative and skills based and who provided it: when sex education was participative and skills based, the peer led intervention was more effective, but when these methods were not used, teacher led education was more effective.
  
• Peer led sex education was less good at engaging students most at risk of poor sexual health (risk was assessed using housing tenure, attitude to school, and educational aspirations); the peer led approach was better at increasing knowledge in schools serving "medium" rather than "low" risk populations (assessed using the proportion of students receiving free school meals).
  

Integrating process and outcome evaluation

Process evaluation in the RIPPLE trial is an example of the trend to move beyond the rhetoric of quantitative versus qualitative methods. By integrating process and outcome data, we maximised our ability to interpret results according to empirical evidence. Importantly, an on-treatment approach to outcomes using process data made little difference to the results. Using process data to identify key dimensions of sex education and examining these in relation to the trial arm revealed the circumstances in which peer led sex education was most effective, as did analysis of risk for both individual schools and students. The conclusion that the peer led approach is not a straightforward answer to the problem of school disengagement and social exclusion is an important policy message.

Other recent trials of complex interventions in which integral process evaluations helped explain the outcome findings include a trial of peer education for homosexual men that had no apparent impact on HIV risk behaviour; the process evaluation established that the intervention was largely unacceptable to the peer educators recruited to deliver it.¹⁵ In trials of day care for preschool children and of supportive health visiting for new mothers, integral process evaluations revealed unpredicted economic effects of day care and the reasons for low uptake of health visiting in a culturally diverse population.^{16 17} In a trial of a secondary prevention programme for cardiovascular disease, interviews and focus groups showed that because patients viewed heart attacks as self limited episodes, they were less willing to adopt long term lifestyle changes, and that the ability of practice nurses to provide skilled continuity of care was affected by their lack of training and low status in the primary health care team.¹⁸ Such trials involve issues of service and organisation and the frameworks of understanding that inform the behaviour of healthcare users—features of the interactions between intervention and environment that characterise complex interventions.¹⁹

Box 2: Methodological issues of integrating process evaluation within trials Process data should be collected from all intervention and control sites Data should be both qualitative and quantitative Process data should be analysed before outcome data to avoid bias in interpretation Steps should be taken to minimise the possibility of bias and error in interpreting the findings from statistical approaches, such as on-treatment analyses and regression and subgroup analyses

"Good" randomised controlled trials have been defined as those having a high quality intervention, adequate evaluation of the intervention and its delivery, documentation of external factors that may influence outcome, and a culturally sensitive intervention.²⁰ Thus, many RCTs would be enhanced by an integral process evaluation. The additional costs (such as collecting and analysing qualitative data) would probably be balanced by greater explanatory power and understanding of the generalisability of the intervention.²¹ Complexity as a quality of interventions is a matter of degree, not an absolute. The argument that process evaluation is most useful in cluster trials, and where the intervention is non-standardised, also applies to many "pragmatic" clinical RCTs,²² as does the idea that process evaluation in feasibility studies is crucial to developing appropriate and effective interventions.²³

A "social" model of RCTs?

The move towards combining process and outcome evaluation builds on other, related methodological developments. For example, more attention is being paid to the need for pilot and consultation work in the development phase of RCTs,²³ the importance of a more theory based approach to evaluation,²⁴ and the modification of intervention effects by context in community intervention trials.²⁵ Statistical methods to integrate process and outcome data, such as those developed in the RIPPLE study, are a move forward.

Some of the methods we suggest—such as allowing hypotheses about the effectiveness of interventions to be derived from process data collected during a trial, and drawing conclusions (with reservations) from on-treatment and subgroup analyses—go against the conventions of clinical RCTs. Perhaps the effort devoted to developing "how to do it" guidelines for clinical trials could be matched by a similar effort in the field of complex interventions.

Summary points A detailed process evaluation should be integral to the design of many randomised controlled trials Process evaluations should specify prospectively a set of process research questions and identify the processes to be studied, the methods to be used, and procedures for integrating process and outcome data Expanding models of evaluation to embed process evaluations more securely in the design of randomised controlled trials is important to improve the science of testing approaches to health improvement It is also crucial for persuading those who are sceptical about using randomised controlled trials to evaluate complex interventions not to discard them in favour of non-randomised or non-experimental studies

---

Contributors: The RIPPLE study group includes the authors and Abdel Babiker, Andrew Copas, and Anne Johnson. The RIPPLE study was designed by AJ, AO, and JS. JS is principal investigator and is responsible for the overall conduct of the trial. The process evaluation is led by AO and VS. VS, S Forrest, and S Charleston developed methods for collecting and integrating qualitative and quantitative data, and coordinated data collection. EA and AC were the trial statisticians; S Black, M Ali, and AB contributed to statistical analyses. AJ contributed to quantitative data collection methodology. CB led data analyses related to social exclusion and sexual health. AO was the guarantor.

Funding: Medical Research Council.

Competing interests: None declared.

Ethical approval: Committee on ethics of human research, University College, London.

References

1. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321: 694-6.[Free Full Text]
2. Hawe P, Shiell A, Riley T. Complex interventions: how "out of control" can a randomised controlled trial be? BMJ 2004;328: 1561-3.[Free Full Text]
3. Gueron JM. The politics of random assignment: implementing studies and affecting policy. In: Mosteller F, Boruch R, eds. Evidence matters: randomized trials in education research. Washington, DC: Brookings Institution Press, 2002: 15-49.
4. Wight D, Obasi A. Unpacking the "black box": the importance of process data to explain outcomes. In: Stephenson J, Imrie J, Bonell C, eds. Effective sexual health interventions: issues in experimental evaluation. Oxford: Oxford University Press, 2002: 151-66.
5. Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health 2002;56: 119-27.[Abstract/Free Full Text]
6. Oakley A. Design and analysis of social intervention studies in health. In: Bowling A, Ebrahim S, eds. Handbook of health research methods. Maidenhead: Open University Press (in press).
7. Harden A, Oakley A, Oliver S. Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J 2001;60: 363-70.
8. Stephenson JM, Oakley A, Charleston S, Brodala A, Fenton K, Petruckevitch A, et al. Behavioural intervention trials for HIV/STD prevention in schools: are they feasible? Sex Transm Infect 1998;74: 405-8.[Abstract]
9. Stephenson JM, Oakley A, Johnson AM, Forrest S, Strange V, Charleston S, et al. A school-based randomized controlled trial of peer-led sex education in England. Control Clin Trials 2003;24: 643-57.[CrossRef][ISI][Medline]
10. Stephenson JM, Strange V, Forrest S, Oakley A, Copas A, Allen E, et al. Pupil-led sex education in England (RIPPLE study): cluster-randomised intervention trial. Lancet 2004;364: 338-46.[CrossRef][ISI][Medline]
11. Oakley A, Strange V, Stephenson J, Forrest S, Monteiro H. Evaluating processes: a case study of a randomised controlled trial of sex education. Evaluation 2004;10: 440-62.[Abstract]
12. Strange V, Allen E, Oakley A, Stephenson J, Johnson A. Integrating process and outcome data in a randomised controlled trial of sex education. Evaluation (in press).
13. Victora CG, Huttly S, Fuchs SC, Olinto MTA. The role of conceptual factors in epidemiological analysis. A hierarchical approach. Int J Epidemiol 1997;26: 224-7.[Abstract/Free Full Text]
14. Oxman AD, Gordon H, Guyatt A. Consumers' guide to subgroup analyses. Ann Intern Med 1992;116: 78-84.[Medline]
15. Elford J, Sherr L, Bolding G, Serie F, Maguire M. Peer-led HIV prevention among gay men in London: process evaluation. AIDS Care 2002;14: 351-60.[CrossRef][ISI][Medline]
16. Toroyan T, Oakley A, Laing G, Roberts I, Mugford M, Turner J. The impact of day care on socially disadvantaged families: an example of the use of process evaluation within a randomised controlled trial. Child Care Health Dev 2004;30: 691-8.[CrossRef][ISI][Medline]
17. Wiggins M, Oakley A, Roberts I, Turner H, Rajan L, Austerberry H, et al. The social support and family health study: a randomised controlled trial and economic evaluation of two alternative forms of postnatal support for mothers living in disadvantaged inner city areas. Health Technology Assessment Monograph 8: 32.
18. Bradley F, Wiles R, Kinmonth A-L, Mant D, Gantley M. Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care programme (SHIP). BMJ 1999;318: 711-5.[Free Full Text]
19. Wolf N. Randomised trials of socially complex interventions: promise or peril? J Health Serv Res Policy 2001;6: 123-6.[CrossRef][Medline]
20. Ross DA, Wight D. The role of randomized controlled trials in assessing sexual health interventions. In: Stephenson J, Imrie J, Bonell C, eds. Effective sexual health interventions: issues in experimental evaluation. Oxford: Oxford University Press, 2002: 35-48.
21. Helitzer DL, Davis SM, Gittelsohn J, Going SB, Murray DM, Snyder P, et al. Process evaluation in a multisite primary obesity-prevention trial in American Indian schoolchildren. Am J Clin Nutr 1999;69: 816S-24S.[Abstract/Free Full Text]
22. Hotopf M. The pragmatic randomised controlled trial. Adv Psychiatr Treat 2002;8: 326-33.[Free Full Text]
23. Power R, Langhaug LF, Nyamurera, Wilson D, Bassett MT, Cowan FM. Developing complex interventions for rigorous evaluation—a case study from rural Zimbabwe. Health Educ Res 2004;19: 570-5.[Abstract/Free Full Text]
24. Stame N. Theory-based evaluation and varieties of complexity. Evaluation 2002;10: 58-76.
25. Hawe, P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Community Health 2004;58: 788-93.[Abstract/Free Full Text]

(Accepted 14 October 2005)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Complex interventions or complex systems? Implications for health economic evaluation

Alan Shiell, Penelope Hawe, and Lisa Gold
BMJ 2008 336: 1281-1283. [Extract] [Full Text] [PDF]

Internal and external validity of cluster randomised trials: systematic review of recent trials

Sandra Eldridge, Deborah Ashby, Catherine Bennett, Melanie Wakelin, and Gene Feder
BMJ 2008 336: 876-880. [Abstract] [Full Text] [PDF]

Equitable utilisation of Indian community based health insurance scheme among its rural membership: cluster randomised controlled trial

M Kent Ranson, Tara Sinha, Mirai Chatterjee, Fenil Gandhi, Rupal Jayswal, Falguni Patel, Saul S Morris, and Anne J Mills
BMJ 2007 334: 1309. [Abstract] [Full Text] [PDF]

Review of stroke rehabilitation

John Young and Anne Forster
BMJ 2007 334: 86-90. [Extract] [Full Text] [PDF]

Strategies to prevent falls and fractures in hospitals and care homes and effect of cognitive impairment: systematic review and meta-analyses

David Oliver, James B Connelly, Christina R Victor, Fiona E Shaw, Anne Whitehead, Yasemin Genc, Alessandra Vanoli, Finbarr C Martin, and Margot A Gosney
BMJ 2007 334: 82. [Abstract] [Full Text] [PDF]

Assessment of generalisability in trials of health interventions: suggested framework and systematic review

C Bonell, A Oakley, J Hargreaves, V Strange, and R Rees
BMJ 2006 333: 346-349. [Full Text] [PDF]

Disentangling separate effects

Jane Smith
BMJ 2006 332: 0. [Extract] [Full Text] [PDF]

Complex interventions: how "out of control" can a randomised controlled trial be?

Penelope Hawe, Alan Shiell, and Therese Riley
BMJ 2004 328: 1561-1563. [Extract] [Full Text] [PDF]

Framework for design and evaluation of complex interventions to improve health

Michelle Campbell, Ray Fitzpatrick, Andrew Haines, Ann Louise Kinmonth, Peter Sandercock, David Spiegelhalter, and Peter Tyrer
BMJ 2000 321: 694-696. [Extract] [Full Text] [PDF]

Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP)

Fiona Bradley, Rose Wiles, Ann-Louise Kinmonth, David Mant, and Madeleine Gantley
BMJ 1999 318: 711-715. [Full Text] [PDF]

This article has been cited by other articles:

• Shiell, A., Hawe, P., Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. BMJ 336: 1281-1283 [Full text]  
• Brown, C, Hofer, T, Johal, A, Thomson, R, Nicholl, J, Franklin, B D, Lilford, R J (2008). An epistemology of patient safety research: a framework for study design and interpretation. Part 2. Study design. Qual Saf Health Care 17: 163-169 [Abstract] [Full text]  
• Brown, C, Hofer, T, Johal, A, Thomson, R, Nicholl, J, Franklin, B D, Lilford, R J (2008). An epistemology of patient safety research: a framework for study design and interpretation. Part 3. End points and measurement. Qual Saf Health Care 17: 170-177 [Abstract] [Full text]  
• Brown, C, Hofer, T, Johal, A, Thomson, R, Nicholl, J, Franklin, B D, Lilford, R J (2008). An epistemology of patient safety research: a framework for study design and interpretation. Part 4. One size does not fit all. Qual Saf Health Care 17: 178-181 [Abstract] [Full text]  
• Slade, M., Gask, L., Leese, M., McCrone, P., Montana, C., Powell, R., Stewart, M., Chew-Graham, C. (2008). Failure to improve appropriateness of referrals to adult community mental health services--lessons from a multi-site cluster randomized controlled trial. Fam Pract 25: 181-190 [Abstract] [Full text]  
• Eldridge, S., Ashby, D., Bennett, C., Wakelin, M., Feder, G. (2008). Internal and external validity of cluster randomised trials: systematic review of recent trials. BMJ 336: 876-880 [Abstract] [Full text]  
• Protheroe, J., Bower, P., Chew-Graham, C. (2007). The use of mixed methodology in evaluating complex interventions: identifying patient factors that moderate the effects of a decision aid. Fam Pract 24: 594-600 [Abstract] [Full text]  
• Clark, A. M., MacIntyre, P. D., Cruickshank, J. (2007). A critical realist approach to understanding and evaluating heart health programmes. Health (London) 11: 513-539 [Abstract]  
• Ranson, M K., Sinha, T., Chatterjee, M., Gandhi, F., Jayswal, R., Patel, F., Morris, S. S, Mills, A. J (2007). Equitable utilisation of Indian community based health insurance scheme among its rural membership: cluster randomised controlled trial. BMJ 334: 1309-1309 [Abstract] [Full text]  
• Oliver, D., Connelly, J. B, Victor, C. R, Shaw, F. E, Whitehead, A., Genc, Y., Vanoli, A., Martin, F. C, Gosney, M. A (2007). Strategies to prevent falls and fractures in hospitals and care homes and effect of cognitive impairment: systematic review and meta-analyses. BMJ 334: 82-82 [Abstract] [Full text]  
• Bonell, C, Oakley, A, Hargreaves, J, Strange, V, Rees, R (2006). Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ 333: 346-349 [Full text]  
• Green, J. (2006). The evolving randomised controlled trial in mental health: studying complexity and treatment process. Adv. Psychiatr. Treat. 12: 268-279 [Abstract] [Full text]