CCBY Open access
Research Methods & Reporting

Regression discontinuity designs: an approach to the evaluation of treatment efficacy in primary care using observational data

BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g5293 (Published 08 September 2014) Cite this as: BMJ 2014;349:g5293
  1. Aidan G O’Keeffe, lecturer in statistics1,
  2. Sara Geneletti, lecturer in statistics2,
  3. Gianluca Baio, lecturer in statistics1,
  4. Linda D Sharples, professor of statistics and director of CHRD3,
  5. Irwin Nazareth, professor of primary care and population science4,
  6. Irene Petersen, reader in epidemiology and statistics4
  1. 1Department of Statistical Science, University College London, London WC1E 6BT, UK
  2. 2Department of Statistics, Columbia House, London School of Economics, London WC2A 2AE, UK
  3. 3Leeds Clinical Trials Research Unit, University of Leeds, Leeds LS2 9JT, UK
  4. 4Department of Primary Care and Population Health, University College London, UCL Royal Free Campus, London NW3 2PF, UK
  1. Correspondence to: A G O’Keeffe a.o%E2%80%99keeffe{at}ucl.ac.uk
  • Accepted 6 June 2014

Regression discontinuity designs use observational data to examine treatment efficacy. This article uses the example of statin prescription in primary care to explain the concept of the method and how it can be used

Introduction

Randomised controlled trials are, in most scenarios, the best scientific method for evaluating the efficacy of treatment.1 However, some randomised controlled trials may have poor external validity. Therefore, complementing any evidence on treatment efficacy from randomised controlled trials with that from observational studies is often valuable.2 In this article, we draw attention to a method, known as the regression discontinuity design (RDD), that examines treatment efficacy by using observational data. We describe how this method could be useful to evaluate treatment strategies in primary care and outline the concept of RDD to a clinical audience.

We consider an observational dataset in which information is available on a particular treatment and a specific, continuous, outcome variable. Our interest lies in determining the effect of the treatment on the outcome. Furthermore, treatment allocation occurs according to the value of a chosen, continuous, “assignment variable,” whereby patients are given the treatment if their assignment variable lies above a pre-specified “threshold.” For example, a drug for hypertension (the treatment) designed to reduce blood pressure (the outcome) might be given only to those patients whose systolic blood pressure (the assignment variable) lies above 140 mm Hg (the threshold). In clinical practice, the threshold may not always be fixed.

An RDD makes use of the treatment threshold and assumes that individuals whose assignment variable lies “just above” or “just below” this threshold belong to the same population. With this assumption, the threshold might be seen as a random intervention that assigns the treatment to people whose assignment variable lies just above the threshold and withholds treatment from those whose assignment variable lies just below the threshold. We note that, unlike in a randomised controlled trial, this is not true randomisation, but it may be interpreted as a form of quasi-randomisation in an observational setting. Hence, the RDD could be used, in some cases, to estimate the effect of treatment on a specific outcome.

Since its inception in the 1960s in educational economics,3 the RDD has been applied in labour economics,4 politics,5 psychology,6 and other areas.7 However, with a few notable exceptions,8 9 10 11 12 13 the RDD has yet to be exploited fully to improve understanding of treatment efficacy in medicine. Previous applications of RDDs to healthcare have tended to tailor the design to specific problems and have often used RDDs for health economic analyses rather than for the assessment of a specific treatment effect. We focus on how an RDD may be used with observational data within a medical context more generally, where a treatment is applied at a given threshold, such as the prescription of a drug according to specific guidelines (for example, National Institute for Heath and Care Excellence (NICE) guidelines). In addition, we discuss the problem of non-adherence to treatment. Throughout, we use the prescription of statins on the basis of cardiovascular risk score as an example to illustrate these concepts.

Example: prescription of statins in UK primary care

Statins are one of the most widely prescribed classes of drugs in UK primary care, accounting for more than £800m (€500m; $664m) of National Health Service expenditure in 2011.14 The effect of statins in primary prevention of cardiovascular disease has been widely investigated in randomised controlled trials and observational studies.15 16

NICE guidance on the use of statins for the primary prevention of cardiovascular disease suggests that statins should be prescribed to people whose risk of experiencing a cardiovascular event in the subsequent 10 years exceeds 20% (revised downwards to a 10% risk score threshold in July 2014), as calculated using an appropriate cardiovascular risk indicator (for example, the Framingham risk calculator at www.framinghamheartstudy.org/risk-functions/cardiovascular-disease/10-year-risk.php.17 Hence, the RDD could be used to estimate the effect of statins on low density lipoprotein cholesterol concentration for people around this 20% threshold under the assumption that those “just below” and “just above” the threshold are similar. If this assumption is valid, then we have two groups of treated and untreated individuals whom we consider to belong to the same population. In this example, the assignment variable is the 10 year cardiovascular risk score, the outcome variable is the low density lipoprotein cholesterol concentration, and the threshold is a 10 year cardiovascular risk score of 20%. Figure 1 illustrates some example plots of low density lipoprotein cholesterol concentration against cardiovascular risk score. In plots A, B, and C, patients receive treatment only when they lie above the 20% risk score threshold. In plot D, for some patients (represented by the square icons), treatment is contrary to that inferred from their risk scores.

Figure1

Fig 1 Graphical representations of various regression discontinuity design (RDD) scenarios using simulated data with 10 year cardiovascular risk score as assignment variable and low density lipoprotein (LDL) cholesterol concentration as outcome variable. d represents arbitrary maximum distance from 20% cardiovascular risk score threshold at which data points would be included in associated RDD analysis

Figure 1 panels A-C show different approaches to the analysis of these measurements, when treatment is applied strictly according to the guidelines. A “naïve” approach to estimating the treatment effect would be to take the difference in sample mean low density lipoprotein cholesterol concentrations between people above and people below the threshold, within a particular distance, d, either side of the threshold (fig 1, panel A). The distance, d, defines the population to which the RDD is applied and quantifies the notion of similar people being just below and just above the threshold.

This naïve approach to estimation depends on the chosen value of d. A more appropriate estimate of the treatment effect at the threshold would be obtained by fitting linear regression models on either side of the threshold,18 because one might assume a linear relation between risk score and low density lipoprotein cholesterol concentration. If a constant relation between risk score and low density lipoprotein cholesterol concentration is assumed, then two separate linear regression models with the same slope would be fitted, one on each side of the threshold (fig 1, panel B). Then, the difference in fitted low density lipoprotein cholesterol concentrations at the 20% threshold, estimated using these two models, would be an estimate of the effect of statins on low density lipoprotein cholesterol concentration. If the slopes cannot be considered the same below and above the threshold, as in figure 1 panel C, then linear regression models with different slopes should be fitted on each side of the threshold or, perhaps, more flexible models could be considered.19

Figure 1 panels A-C represent situations in which guidelines are strictly adhered to. In this context, an RDD is termed “sharp.” Many situations may exist in which guidelines are not interpreted strictly. For statin prescriptions, individual general practitioners may prescribe statins by using other information about patients, or patients might request a prescription for a statin irrespective of their risk score. In such scenarios, there will be people below the threshold who receive the treatment and people above the threshold who do not. In this case, the RDD is termed “fuzzy.” Figure 1 panel D shows an example plot from a fuzzy RDD, where the square icons represent those patients who received treatment contrary to that indicated by their 10 year cardiovascular risk score.

Generally, most RDDs in clinical settings will be fuzzy and must account for the possibility that the threshold does not correspond to a strict separation between treated and untreated patients. A fuzzy RDD is analogous to a randomised controlled trial with partial compliance: there are patients allocated to a treatment arm (equivalent to being above the threshold in an RDD) who have not received a treatment and patients allocated to the control arm (below the threshold in an RDD) who have received a treatment. These patients are unlikely to represent a random sample in either a randomised controlled trial or an RDD setting. This problem can be alleviated by using a method for estimation of treatment effect that accounts for whether or not a patient actually receives the treatment above or below the threshold. This estimate is termed a “local average treatment effect,”19 20 and we view this as the treatment effect estimate from a sharp RDD but re-weighted to account for the fuzziness in the RDD. This method may be unreliable in some cases, typically where the threshold is not strongly associated with the treatment received. In this scenario, an RDD may not be appropriate for the estimation of a treatment effect.

Our estimates of treatment effect depend on the threshold being a reliable indicator of patients’ treatment. Figure 2 provides some guidance on appropriate data plots to consider when attempting to use an RDD. Such plots can be helpful in exploring whether the threshold defines an acceptable separation between treated and untreated patients. In the statins example, if we plot risk score against low density lipoprotein cholesterol concentration and observe an obvious “jump” or discontinuity at the threshold level, this indicates that the threshold defines an appropriate point at which an RDD might be used.

Figure2

Fig 2 Useful exploratory plots of assignment variable against outcome variable with threshold marked by the solid line. Panel A shows obvious sharp discontinuity at threshold where regression discontinuity design (RDD) could be used. B shows fuzzy scenario in which RDD could be used with some accounting for fuzziness. C shows no obvious discontinuity at threshold; one would not naturally consider RDD here

Discussion

Considerable scope exists for use of an RDD to evaluate effects of treatment by using observational data. An RDD is useful when treatment is applied according to a strictly defined rule, linked to a variable measured on a continuous scale (such as blood pressure or cardiovascular risk score). An RDD may be used in scenarios in which the treatment rule is not adhered to strictly, provided that the assignment variable gives a reasonable indication of the treatment received. A major advantage of an RDD is that it may be used to evaluate the efficacy of treatments in “real world” populations outside randomised controlled trials. Furthermore, an RDD may be a first step in establishing estimates of treatment effect in sub-populations that have been excluded from randomised controlled trials or for areas in which such trials would not be feasible.

Naturally, an RDD does not necessarily represent a quick fix for all problems encountered when using observational data to estimate treatment effects. The assumption that individuals around the threshold are similar could be violated and should be assessed using context specific information. Other factors such as choosing the size of the region around the threshold for the RDD and the possible changing nature of the assignment variable over time are important. One should ensure that the threshold defines an appropriate point at which treatment is started. Plots of the assignment variable against the outcome variable, such as those in figure 2, would help to establish this. Finally, as with most observational studies, the problems of possible unobserved confounding and missing data usually need careful consideration.

Overall, an RDD applied to observational data represents a pragmatic approach to estimation of treatment effect in clinical practice. The rich data available from electronic health records, if used appropriately, may allow the effectiveness of NHS or government defined prescription rules to be evaluated.

Summary points

  • Regression discontinuity designs are not commonly applied to observational data in medicine but may represent a useful method to estimate the causal effect of a treatment in a clinical setting

  • Under particular assumptions, such designs can be used to evaluate the effectiveness of NHS or clinical treatment guidelines in a pragmatic manner, using observational data from clinical practice

  • In addition, these designs may be useful in verifying whether results seen in randomised controlled trials are applicable to wider populations, in the examination of treatment efficacy for populations who are under-represented in randomised controlled trials, or for scenarios in which such trials are not feasible

Notes

Cite this as: BMJ 2014;339:g5293

Footnotes

  • We thank Philip Dawid, Nick Freemantle, Richard Morris, and Sylvia Richardson for insightful discussions.

  • Contributors: AGO’K led on writing the manuscript; all other authors contributed to the final version. AGO’K is the guarantor.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors receive support through grant MR/K014838/1 from the Medical Research Council; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work

  • Provenance and peer review: Not commissioned; not externally peer reviewed.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt, build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

References

View Abstract