Education And Debate

Framework for design and evaluation of complex interventions to improve health

BMJ 2000; 321 doi: http://dx.doi.org/10.1136/bmj.321.7262.694 (Published 16 September 2000) Cite this as: BMJ 2000;321:694
  1. Michelle Campbell, senior policy analysta,
  2. Ray Fitzpatrick, professor of public health and primary care (raymond.fitzpatrick{at}nuffield.ox.ac.uk)b,
  3. Andrew Haines, professor of primary health carec,
  4. Ann Louise Kinmonth, professor of general practiced,
  5. Peter Sandercock, professor of medical neurologye,
  6. David Spiegelhalter, senior scientistf,
  7. Peter Tyrer, professor of community psychiatryg
  1. a Office of the President, Medical Research Council of Canada, 1600 Scott Street, Ottawa, Ontario, Canada K1 OW9,
  2. b Division of Public Health and Primary Health Care, Institute of Health Sciences, University of Oxford, Oxford OX3 7LF,
  3. c Department of Primary Care and Population Sciences, Royal Free and University College Medical School, London NW3 2PF,
  4. d General Practice and Primary Care Research Unit, Department of Public Health and Primary Care, Institute of Public Health, Cambridge CB2 2SR
  5. e Neuroscience Trials Unit, Department of Clinical Neurosciences, Western General Hospitals NHS Trust, Edinburgh EH4 2XU
  6. f MRC Biostatistics Unit, Institute of Public Health, Cambridge CB2 2SR,
  7. g Department of Public Mental Health, Imperial College of Science, Technology, and Medicine, St Mary's Campus, London W2 1PD
  1. Correspondence to: R Fitzpatrick
  • Accepted 31 May 2000

Randomised controlled trials are widely accepted as the most reliable method of determining effectiveness, but most trials have evaluated the effects of a single intervention such as a drug. Recognition is increasing that other, non-pharmacological interventions should also be rigorously evaluated.1-3 This paper examines the design and execution of research required to address the additional problems resulting from evaluation of complex interventions—that is, those “made up of various interconnecting parts.”4 The issues dealt with are discussed in a longer Medical Research Council paper (www.mrc.ac.uk/complex_packages.html). We focus on randomised trials but believe that this approach could be adapted to other designs when they are more appropriate.

Summary points

Complex interventions are those that include several components

The evaluation of complex interventions is difficult because of problems of developing, identifying, documenting, and reproducing the intervention

A phased approach to the development and evaluation of complex interventions is proposed to help researchers define clearly where they are in the research process

Evaluation of complex interventions requires use of qualitative and quantitative evidence

Challenges of trials of complex interventions

There are specific difficulties in defining, developing, documenting, and reproducing complex interventions that are subject to more variation than a drug. A typical example would be the design of a trial to evaluate the benefits of specialist stroke units. Such a trial would have to consider the expertise of various health professionals as well as investigations, drugs, treatment guidelines, and arrangements for discharge and follow up. Stroke units may also vary in terms of organisation, management, and skill mix. The active components of the stroke unit may be difficult to specify, making it difficult to replicate the intervention. The box gives other examples of complex interventions.

Examples of complex interventions

Service delivery and organisation:

Stroke units

Hospital at home

Interventions directed at health professionals' behaviour:

Strategies for implementing guidelines

Computerised decision support

Community interventions:

Community based programmes to prevent heart disease

Community development approaches to improve health

Group interventions:

Group psychotherapies or behavioural change strategies

School based interventions—for example, to reduce smoking or teenage pregnancy

Interventions directed at individual patients:

Cognitive behavioural therapy for depression

Health promotion interventions to reduce alcohol consumption or support dietary change

Framework for trials of complex interventions

Problems often arise in the evaluation of complex interventions because researchers have not fully defined and developed the intervention. It is useful to consider the process of development and evaluation of such interventions as having several distinct phases. These can be compared with the sequential phases of drug development (fig 1) or may be seen as more iterative (fig 2). Either way a phased approach separates the different questions being asked.

Progression from one phase to another may not be linear. In many cases an iterative process occurs—for example, if an exploratory trial finds that a complex intervention is unacceptable to potential recipients, the theoretical basis and components of the intervention may have to be re-examined. Preliminary work is often essential to establish the probable active components of the intervention so that they can be delivered effectively during the trial. Identifying which stage of development has been reached in specifying the intervention and outcome measures will give researchers and funding bodies reasonable confidence that an appropriately designed and relevant study is being proposed.

Preclinical or theoretical phase

The first step is to identify the evidence that the intervention might have the desired effect. This may come from disciplines outside the health sciences (such as theories of organisational change). Review of the theoretical basis for an intervention may lead to changes in the hypothesis and improved specification of potentially active ingredients. In addition, previous studies may have provided some empirical evidence—for example, an intervention may have been found effective for a closely related condition or in another country with a different organisation of health care.5

Fig 1
Fig 1

Sequential phases of developing randomised controlled trials of complex interventions

Fig 2
Fig 2

Iterative view of development of randomised controlled trials of complex interventions

Phase I: defining components of the intervention

Modelling or simulation techniques can improve understanding of the components of an intervention and their interrelationships. Qualitative testing through focus groups, preliminary surveys, or case studies can also help define relevant components. Descriptive studies may help to delineate variants of a service. For example, hospital at home schemes vary in purpose. Some are designed to hasten hospital discharge, others to avoid hospital admissions, and yet others to provide palliative care in the home.6

Qualitative research can also be used to show how the intervention works and to find potential barriers to change in trials that seek to alter patient or professional behaviour.7 For example, if health professionals see the main barrier to changing their practice as being lack of time or resources, an intervention that focuses only on improving their knowledge will not work.

Phase II: defining trial and intervention design

Acceptability and feasibility

In phase II the information gathered in phase I is used to develop the optimum intervention and study design. This often involves testing the feasibility of delivering the intervention and acceptability to providers and patients. Different versions of the intervention may need to be tested or the intervention may have to be adapted to achieve optimal effectiveness—for example, if the proposed intensity and duration of the intervention are found to be unacceptable to participants.

It is also important to test for evidence of a learning curve, leading to improved performance of the intervention over time. If a learning curve exists a run-in period might be needed before formal recruitment to the trial to ensure that the intervention is provided effectively.

The exploratory trial is also an opportunity to determine the consistency with which the intervention is delivered. Consultations could be audio or video taped to give feedback of performance to providers together with training to promote consistency.

Defining the control intervention

The content of the comparative arm (control group) of the main trial will be decided during the preparatory phase. It may be an alternative package of care, standard care, or placebo. Although standard practice is often an appropriate control, it can be as complex as the intervention being evaluated and may change with time. It is thus important to monitor the care that is being delivered to the control group. The use of a no treatment control group may be unacceptable to patients. One possible solution is a randomised waiting list study in which all participants ultimately receive the intervention.

Designing the main trial

The exploratory phase should ideally be randomised to allow assessment of the size of the effect. This initial assessment will provide a sound basis for calculating sample sizes for the main trial. Other design variables can also be established in an exploratory trial.

Outcomes

Outcome measures for the main trial will also generally be piloted during the exploratory phase. Investigators should include outcomes that not only are relevant to patients with the disease or condition being studied but also encompass measures of wider relevance to the health system, including economic measures.8 Collection of data to assess a full range of costs to patients, carers, and society adds considerably to the workload and costs of researchers and may challenge the feasibility of a trial. Strategic choice of outcomes is therefore needed.9

An important decision in trials of complex interventions is whether health outcome needs to be assessed. For studies such as those evaluating strategies to change professional behaviour, it may be sufficient to show that the intervention changed behaviour, provided that clear evidence exists that the changed behaviour—for example, prescribing particular treatments—is effective.

Phase III: methodological issues for main trial

The main trial will need to address the issues normally posed by randomised controlled trials, such as sample size, inclusion and exclusion criteria, and methods of randomisation, as well as the challenges of complex interventions. Individual randomisation may not always be feasible or appropriate. For example, cluster randomisation is often used for trials of interventions directed at a practice or hospital team. 10 11 Randomised incomplete block designs have also been used to evaluate different approaches to promoting change in professional behaviour.12

It is often not possible to conceal allocation of treatment from the patient, practitioner, and researcher in complex intervention trials. The potential biases of unblinded trials therefore have to be taken into account. Dissimilar levels of patient commitment between intervention and control groups may cause differential dropout, making interpretation of results difficult. When patients have strong preferences, a preference trial design may be used; patients without strong preferences are randomised as usual but those with strong preferences receive their preferred treatment.13 The results of such trials can, however, be difficult to interpret.

The findings of trials of complex interventions are more generalisable if they are performed in the setting in which they are most likely to be implemented. Eligibility criteria must not lead to the exclusion of patients—for example, on the grounds of age—who constitute a substantial portion of those to whom the intervention is likely to be offered when implemented in the health system. Poor recruitment to a trial can also raise doubts about generalisability.

Qualitative study of the processes of implementation of interventions in study arms of the main trial can further show the validity of findings.14

Phase IV: promoting effective implementation

The purpose of the final phase is to examine the implementation of the intervention into practice, paying particular attention to the rate of uptake, the stability of the intervention, any broadening of subject groups, and the possible existence of adverse effects. As in the case of drug trials, this might be carried out by long term surveillance, although currently there is no established mechanism for funding such activities.

Conclusions

Trials of complex interventions are of increasing importance because of the drive to provide the most cost effective health care. Although these trials pose substantial challenges to investigators, the use of an iterative phased approach that harnesses qualitative and quantitative methods should lead to improved study design, execution, and generalisability of results.

Acknowledgments

We thank the participants at a MRC workshop on complex interventions for their contribution. This article represents the views of the authors and does not represent the official view of the Medical Research Council.

Footnotes

  • Competing interests None declared.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.