A blueprint for synthetic control methodology: a causal inference tool for evaluating natural experiments in population health
BMJ 2022; 379 doi: https://doi.org/10.1136/bmj.o2712 (Published 23 November 2022) Cite this as: BMJ 2022;379:o2712Linked Research
Impact of community asymptomatic rapid antigen testing on covid-19 related hospital admissions
- Ben Barr, chair in applied public health research1,
- Xingna Zhang, research associate1,
- Mark Green, reader in health geography2,
- Iain Buchan
, chair in public health and clinical informatics1
- 1Department of Public Health, Policy and Systems, University of Liverpool, Liverpool, UK
- 2Department of Geography and Planning, University of Liverpool, Liverpool, UK
- Correspondence to: I Buchan buchan{at}liverpool.ac.uk
Causal effect
A causal effect is defined as the difference between what happened in an observed population experiencing the intervention versus what might have happened without it. Two alternative situations are compared—one where the intervention happened, and a counterfactual where it did not.1 Causal methods use information on groups that did not experience the intervention to try and mimic this counterfactual. Trials can use randomisation to estimate this counterfactual. When trials are impractical, other causal methods can harness the observed characteristics of intervention and control populations and subpopulations to estimate what might have happened without the intervention—the synthetic control method (SCM) is one such approach.
Synthetic control method
SCM compares the outcomes of an intervention in a given population to an artificially created control population not experiencing the intervention but having similar characteristics to the intervention population. A predecessor to SCM selected the control group then estimated the effect by subtracting the change in outcomes before versus after the intervention between the intervention and control groups—the difference-in-differences approach. If time trends in outcomes would have tracked in parallel across the groups without the intervention, then the estimate derived from the difference-in-differences approach is an unbiased estimate of the causal effect of the intervention. But this assumption of parallel outcome trajectories depends on selecting the right control group, and so to minimise bias in selecting this control group, the SCM was introduced as a generalisation of the difference-in-differences approach.2 The authors proposed weighting the potential control units (subgroups comprising the control populations) such that the weighted average of outcomes and confounders during the pre-intervention period mimics the outcome path and other characteristics in the intervention population. The difference in weighted outcomes post-intervention between this synthetic control group and the intervention group allows an estimate of the intervention’s effect. Various approaches are used to derive optimal weights. Most studies that use SCM have focused on a single treated unit (usually a geographical place, such as a city) experiencing the intervention, and derived weights for other units not experiencing the intervention to minimise pre-intervention differences between intervention and control groups. Another approach extended this to multiple intervention units, such as small neighbourhoods or census tracts.34 We applied this synthetic control approach for microdata to the evaluation of the Liverpool covid-19 community testing pilot (doi:10.1136/bmj-2022-071374).5
When and how to use the SCM
SCMs are best suited to evaluating population level interventions using a panel of aggregate data across similar units. This is because SCM requires continuous, sequential data at consistent and regular time points, with limited random fluctuations over time.6 SCM using aggregate data can be applied when individual level data are not available (eg, to preserve privacy). No major event or intervention should have occurred in any group before the intervention, and the intervention should not “leak” into the synthetic control population. SCM conventionally requires a discrete time point for when the intervention started, although staggered interventions can be accommodated.7
Application to covid-19 action research
During a public health emergency, such as the covid-19 pandemic, policy decisions need to be made quickly based on imperfect evidence. New interventions need to be evaluated rapidly. Although potential scenarios can be simulated using current knowledge and assumptions, retrospective evaluation should be informed by real world data when available. Policy interventions create natural experiments that can be evaluated to inform the next steps in responses. Limited access to sufficiently granular data may impair these important rapid evaluations, and SCM is useful for maximising causal information from small area aggregate data that may be more readily available. The UK’s response to covid-19 resulted in many natural experiments with potentially important learning for future public health emergencies. In supporting local and national covid-19 responses we applied SCM to evaluating the impact of tiered restrictions,8 assessing the effectiveness of vaccination outreach activities,7 and the world’s first pilot of voluntary, mass, asymptomatic rapid antigen testing, as reported in the linked paper (doi:10.1136/bmj-2022-071374).5910
Issues with interpretation and bias
Causal inference with SCM assumes that differences that could affect the outcome other than from the intervention have been accounted for (ie, minimal confounding) between intervention and control groups. By weighting control units and areas to match the intervention units in the pre-intervention period, SCM adjusts for observed and some unobserved confounders, provided these confounders have the same effect on outcomes across the intervention and control groups, and evolved similarly in intervention and control groups following the intervention. Weighting can incorporate additional covariates that predict post-intervention outcomes in absence of the intervention, and this may improve causal inference.6 The appropriateness of covariates can be assessed by visualising them in causal graphical methods reflecting expert knowledge or previous evidence. Causal interpretation of SCM could be impaired by events in the post-intervention observation period that affect intervention and control groups differently. Other potential biases include anticipation effects of the intervention and contamination (spill-over) to the control group. Traditional approaches to measuring the uncertainty of intervention effects are not used in SCM owing to the constraints placed on weights. Instead, confidence intervals and P values are constructed using placebo permutations, such that the analysis is repeated through multiple iterations that randomly allocate control units to the intervention group to estimate the sampling distribution of the treatment effect.4
Conclusion
When designed experiments with randomisation are impractical, SCM is a powerful causal tool for evaluating natural experiments. Whether evaluating the roll-out of public health policies outside of emergency situations or pilots of urgent public health responses during a pandemic, SCM offers important methodological advantages over other observational research methods. In rapidly evolving situations such as pandemics, small area data can be harnessed with SCM to understand the effects of urgent public health measures. We therefore encourage sharing of small area data and use of SCM to improve understanding of population level interventions.
Key features of synthetic control methods
Important causal inference tool when randomisation is impractical
Good for evaluating population level interventions using aggregate data from control and intervention units (eg, neighbourhoods)
Can be used even when only one unit has received the intervention
Control group is synthesised as a weighted combination of potential control units
Can account for observed and some unobserved confounding
Can be promptly used to evaluate urgent public health interventions
Acknowledgments
BB and XZ are joint first authors.
Footnotes
Funding and competing interests available in the linked paper on bmj.com.
Provenance and peer review: Commissioned; not externally peer reviewed.