Personalising results from large trialsBMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h553 (Published 20 February 2015) Cite this as: BMJ 2015;350:h553
- Rafael Perera, professor and director of Medical Statistics Group,
- Richard J Stevens, associate professor and deputy director of Medical Statistics Group
- Correspondence to: R Perera
Health professionals make decisions for individual patients, but clinical trials typically tell us which interventions work on “average.” Knowing with complete certainty that an intervention will (or will not) improve this patient’s health is perhaps an unattainable ideal. However, concepts of personalised medicine seek to approach this ideal, refining information from trials for well characterised subgroups of patients.
Advances in diagnosis, designed to improve the characterisation of patients, create new ways to subdivide datasets and hence reanalyse trials. Meanwhile, bigger, longer, and better designed studies, together with greater interconnectivity between researchers across the globe, create large datasets in which previously impractical subgroup analyses now seem feasible. However, just because we can reanalyse available data, should we? What are the potential pitfalls and problems in subgroup analyses of clinical trials?
In a linked paper (doi:10.1136/bmj.h454), Sussman and colleagues present a reanalysis of a large diabetes trial, the Diabetes Prevention Program, which “could decrease drug overuse, help to prioritize lifestyle programs, and be a model for the secondary analysis of randomized trials.”1 This three armed trial, published in 2002, compared metformin or lifestyle interventions against usual care for the prevention of diabetes in a population at risk. Sussman and colleagues’ approach to subgroup analyses does not define groups of patients by diagnostic criteria. Instead, they focus on prognosis, defining subgroups by their estimated risk of progression to diabetes. For metformin in particular, they find “heterogeneity of treatment effect”—that the benefit of treatment varies between these groups and, in particular, that people at highest risk have most to gain.
A commentary by Peto discusses the dangers of subgroup analyses,2 especially when not pre-specified in the original study protocol. The more such analyses are conducted, the greater the likelihood of a spurious finding. Peto illustrates the danger with a deliberately implausible analysis of treatment effect by star sign, to remind us that other subgroup analyses—in which a potential causal relation is less obviously spurious—are also at risk of false positive scientific findings. Peto then considers three potential partial remedies, two of which are relevant here.
Peto’s first remedy is the study of subgroups, preferably pre-specified in a study protocol, for which a relevant prior hypothesis exists. Sussman and colleagues do not have a study protocol for their subgroup analyses: instead, they cite a previous body of methodological literature. These earlier, methodological studies find that their risk based examination of treatment heterogeneity has better statistical power than other approaches, with lower risk of spurious findings due to multiple testing.3 4 This evolving methodological literature still has some gaps: for example, the study cited for the assertion that the high number of new cases of diabetes per risk factor avoids certain methodological problems (over-fitting) did not examine the specific algorithm used by Sussman to select risk factors for the final model.5 On the other hand, a recent simulation study suggests their overall approach is reasonably robust 6
Peto’s second remedy is to use the trial as a whole to estimate relative reductions in risk but to acknowledge that when the relative risk of an outcome (ratio of risks: intervention/control) is constant across subgroups, the absolute reductions in risk (difference between risks: intervention−control) will vary in proportion to the underlying absolute risk. The results for lifestyle interventions in Sussman and colleagues’ figure 2 beautifully illustrate the situation that Peto anticipates. The relative effects (estimated here by hazard ratios) of lifestyle changes are approximately constant across all four groups. This results in absolute risk reductions that are much greater in high risk patients.
The striking finding in Sussman’s study is that for metformin, both absolute risk reductions and relative effects (hazard ratios) vary with underlying risk. In particular, the effect of metformin seems to be significantly greater in the 25% of the population at highest estimated risk than in other groups. The result, shown in the plot of absolute risk reduction, is that in high risk groups the absolute benefit of metformin becomes comparable to the absolute benefit of lifestyle changes, but in lower risk groups lifestyle changes seem to have more to offer than prescription of metformin.
Although risk based stratification is a relatively new proposal, we find ourselves encouraged by the results shown here. The heterogeneity seen in treatment effects for metformin (figure 2 of the paper) seem unlikely to be a statistical artefact when no such heterogeneity is observed for lifestyle intervention analysed in the same way. The paper is itself thus a persuasive addition to the methodological literature on risk based subgroup analysis. Replication in other studies and pre-specification in protocols would further strengthen future studies using this approach.
A limitation of Sussman and colleagues’ approach, emphasising estimated baseline risk as the pre-specified subgroup analysis of interest, is that it takes us little nearer to understanding the causal pathway. We gain no insight into the characteristics of individual patients that make metformin effective in some but not others for prevention of diabetes. However, this has not prevented the authors reaching the clinically applicable conclusion that only people at the highest risk of diabetes should be given metformin for its prevention. Other patients would do better to concentrate on lifestyle changes. This novel reanalysis of a landmark diabetes trial has therefore taken diabetes prevention a step closer to a personalised approach.
Cite this as: BMJ 2015;350:h553
Competing interests: We have read and understood the BMJ policy on declaration of interests and declare the following interests: none.
Provenance: Commissioned; not externally peer reviewed.