For healthcare professionals only

Analysis

Complex interventions or complex systems? Implications for health economic evaluation

BMJ 2008; 336 doi: https://doi.org/10.1136/bmj.39569.510521.AD (Published 05 June 2008) Cite this as: BMJ 2008;336:1281
  1. Alan Shiell, professor1,
  2. Penelope Hawe, professor1,
  3. Lisa Gold, senior research fellow2
  1. 1Population Health Intervention Research Centre, University of Calgary, Calgary T2N 4N1, Canada
  2. 2Health Economics Unit, School of Health and Social Development, Deakin University, Melbourne, Australia
  1. Correspondence to: A Shiell ashiell{at}ucalgary.ca
  • Accepted 15 April 2008

Although guidelines exist for evaluating complex interventions, they may be of little help in dealing with the multiple effects of interventions in complex systems such as hospitals. Alan Shiell, Penelope Hawe, and Lisa Gold explain why it is important to distinguish the two types of complexity

Health researchers commonly use the notion of complexity to indicate the problems faced in evaluating the effectiveness of many non-drug interventions.1 2 3 However, although it is rarely delineated, complexity has two meanings. In the first it is a property of the intervention, and in the second it is a property of the system in which the intervention is implemented. We examine the implications of these two views for economic evaluation.

What do we mean by complex?

The first view of complexity, in effect, means complicated. This is the meaning used in the Medical Research Council’s framework for the evaluation of complex interventions.4 5 A complex intervention is “built up from a number of components, which may act both independently and inter-dependently.”4 This makes it hard to define the “active ingredients” and to be sure which component or combinations of components is more important.

The second view makes reference to the insights offered by complexity science.6 7 8 9 Complexity is a property of a system not an intervention. A complex system is one that is adaptive to changes in its local environment, is composed of other complex systems (for example, the human body), and behaves in a non-linear fashion (change in outcome is not proportional to change in input).

Complex systems include primary care, hospitals, and schools. Interventions in these settings may be simple or complicated, but the complex systems approach makes us consider the wider ramifications of intervening and to be aware of the interaction that occurs between components of the intervention as well as between the intervention and the context in which it is implemented. This includes the operations, structures, and relations that exist in each setting10 11 and the implications that contextual effects have for designing and evaluating interventions.12 13

The distinction between the two approaches (complex interventions versus complex systems) is easily blurred because they share common features—for example, non-standardisation, multiplicity, interactions. Analysts working with complex interventions, for example, also recognise the importance of context.14 Furthermore, complicated interventions can take on the characteristics of complex systems, since it is impossible to separate the intervention from the human agency required for its delivery.15 However, it is important to recognise the differences between the two approaches and to identify when each one is being applied correctly when thinking about economic evaluation.

Implications for economic evaluation

The main challenge in evaluating complex interventions arises because the active elements of the intervention are subject to more variation than in typical drug trials. Campbell and colleagues,5 citing the operation of a stroke unit, point to variation among units in staff characteristics, clinical practices, management protocols, and infrastructure. This makes it difficult to specify what the intervention is, what is most effective, or how to replicate the intervention beyond the original trial.5

When it comes to economic evaluation, however, the problem of specifying the intervention is less of an issue. Economists compare the value of what goes in (the resources) with what comes out (the outcomes). If you can specify the inputs and outcomes with sufficient clarity to ensure that changes in resource use and benefits can be measured and valued, then it is not necessary to understand how the intervention works.

To illustrate this we can use the example of group psychotherapy.5 This is a complex intervention because the content of each consultation is tailored to the individual needs of the patients in the group and is adapted as the program of consultations unfolds and each client responds in their different ways to treatment. We do not know whether the treatment effect is the result of the content of the consultation, the personal style of the therapist, the dynamic of the group, a combination of all three, or some other consideration. However, to evaluate economic efficiency the economist need only know how long each session lasts, how many sessions there are, how many people are in each group (so that costs can be apportioned to patients), what skill level is required of the therapist (so that salary costs can be generalised), what other resources are required (the venue for example), and what effect treatment has on health outcomes and use of health services. The content of the consultation is immaterial.

Of course, economic evaluation of multicomponent interventions does present challenges. It is more difficult to draw boundaries around the evaluation. Multicomponent interventions to reduce excessive alcohol consumption, for example, will benefit people beyond the problem drinker, including family members and the community at large, which raises questions about how to include such benefits in the appraisal. But simple interventions tackling the same problem also generate these externalities. Multicomponent interventions will also have an effect on multiple dimensions of health and have non-health benefits as well, but then so too do many simple interventions (vaccination being a good example).

Thus, complex interventions of the sort discussed by the MRC are more difficult to evaluate, but there is nothing substantively different about their economic evaluation. No new economic methods are required, and the problems can all be solved with time, effort, and resources.16

In contrast, evaluating the economic efficiency of interventions directed at changing the properties of complex systems presents big challenges. Complex systems have several defining characteristics including the tendency to be self organising, be sensitive to initial conditions, and make non-linear phase transitions (to jump quickly from one position to another very different position); the existence of emergent properties; and the importance of interaction effects and feedback.17 These characteristics affect what measures of effectiveness should be included in the economic evaluation and how the consequences of the intervention are valued.

What should we evaluate?

The economist’s concern with value will always mean looking for improvement in final (health) outcomes. However, the characteristics of complex systems suggest the need to do much more than this.

Firstly, evaluation of outcomes typically involves measuring health changes at the individual level and simply summing these to capture the “social” effect. In a complex system this is no longer wholly appropriate. Complex systems have emergent properties that are a feature of the system as a whole.18 These properties are not seen in any one part of a complex system nor are they summations of individual parts (community empowerment,19 social exclusion, and income inequality are noted emergent properties relevant to population health). So outcomes should be measured at multiple levels within the complex system, with tools designed specifically for this purpose.

Secondly, the relatively short follow-up periods of most intervention studies and the fact that non-linear change in complex systems is difficult to observe in its early stages means there is high risk of missing important outcomes and concluding prematurely that the intervention is not effective. The impact of public health advocacy on public health policy such as gun control is a case in point. Multiple “advocacy episodes” may have no discernable impact on policy, but then a tipping point is reached, a phase transition occurs, and new laws are introduced. In the search for cause and effect, the role played by advocates in creating the conditions for change is easily overlooked in favour of prominent and immediately prior events.20 To minimise the risk of premature evaluation and wrongful attribution, economists must become comfortable working with evidence of intermediate changes in either process or impact that act as preconditions for a phase transition.

One important indicator of system level change is movement in the positions of key actors within the structures that make up the complex system, and with it changes in their relationships with other actors and agencies. Relational data (collected at the individual level but analysed at the network level using social network methods) are needed to capture these effects. In community based interventions to improve access to primary care, for example, we might wish to see family practitioners become more influential in the network of providers. In interventions in schools, a reduction in the number of children or teachers who are socially isolated, and corresponding increases in the density of support networks, might provide evidence of effect.21 Such organisational and social network measures are not final outcomes favoured by economists (the economic test depends on whether such changes lead to improvements in health and wellbeing) but they provide reassuring evidence that the intervention is having an effect on the system, which will in turn hopefully lead to improvements in health. We are beginning to see these network analytical methods introduced into cluster randomised controlled trials.22

How should we evaluate benefits?

The consequences of intervention in a complex system will not be the small scale, marginal changes usually examined by economists. Since everything is interconnected, changes in one part of the system feed through to other parts of the system and feedback on themselves. The economist’s usual approach assumes that the effects of the intervention can be examined in isolation of changes in the broader context. Feedback loops are ignored. With interventions in complex systems, this no longer applies. Nothing can be assumed constant as everything is linked to everything else.

Two consequences follow. Firstly, spin-off effects are to be expected. The consequences of system level change are both multiple and multiplied, with induced costs and outcomes beyond those originally envisaged in the research protocol. The practical challenge of identifying and capturing these effects within an evaluation is substantial.22

Secondly, one of the things that economists assume is unchanging is the value (that is, the importance) that people assign to the intervention. This assumption is unlikely to hold with system level change. We see this most notably in tobacco control, where the concerted action of public health advocates to reduce the harm associated with tobacco use has changed behaviours and social norms. Support for banning smoking in public places often increases after the policy is implemented.23 This means that the value of an intervention that changes the dynamic of a complex system is likely to be a function of that intervention: people value the intervention more after implementation than before it. Preferences are no longer stable, and this undermines the validity of the methods economists use to ascertain value. More collective, deliberative methods of eliciting social value are needed.24

New approaches?

The view that complexity refers to the systems in which interventions are implemented affects all efforts to evaluate interventions, not just those of economists. For example, it is difficult to attribute causality in a complex system, not least because such systems are sensitive to initial conditions and miniscule differences at baseline can lead to very large differences in outcome. Thus, randomisation (even at the cluster level) may not eliminate all causes of bias, even if it removes all observable differences between groups.25

The economic evaluation of interventions aimed at changing systems requires new ways of thinking: one sensitive to ecological theory, interactions between microlevel and macrolevel variables, non-linearities, multiplier effects, and the fact that individual values are shaped by the interventions we seek to evaluate and the contexts we seek to change.

The methodological agenda is huge, and the proper evaluation of systems level change will be expensive. We should remember therefore that existing methods have served us relatively well thus far. Linear approximation may be sufficient to assess non-linear change (and it is easier and less expensive). Our concerns do not rule out the use of current economic approaches. They do, however, point to the need for extensive prospective data collection alongside cluster trials to capture signs of non-linear change, unintended consequences, and multiplier effects,13 and for more extensive use of modelling to assess the sensitivity of economic evaluations to the inclusion of these effects.

We need to recognise whether we have a complex intervention or an intervention in a complex system, and whether the dynamic characteristics of the system matter enough for us to change our evaluation approach. Neither question is easy to answer, making efforts to develop the means of diagnosing complexity especially important.

Summary points

  • Health research often uses complex to refer to multicomponent interventions

  • An alternate view is that complexity refers to systems

  • Interventions implemented in complex systems are likely to have diverse, far-reaching, and non-linear effects

  • Distinguishing the two types of complexity is important for economic evaluation

Footnotes

  • We thank colleagues in the International Collaboration on Complex Interventions funded by the Canadian Institutes of Health Research (CIHR) and colleagues at an MRC workshop convened to consider revisions to the original MRC Framework. We also thank the reviewers for helpful suggestions. The views expressed here are those of the authors alone.

  • Contributors and sources: PH is the Markin chair in health and society. AS holds a Canadian Institutes of Health Research chair in the economics of public health. Both are also health scientists funded by the Alberta Heritage Foundation for Medical Research. AS had the original idea to write the paper based on ideas developed in discussion with PH and LG. AS wrote the first draft and revisions. PH and LG provided comments on early drafts, helped to refine the arguments and suggested edits. AS is guarantor.

  • Competing interests: None declared.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract