Re: Prognosis research strategy (PROGRESS) 4: Stratified medicine research
The four-part PROGRESS series, of which the article by Hingorani et al.[1] is the final, represents an ambitious and sensible effort at refining prognosis research strategy. Such an undertaking, however, should carefully evaluate whether all basic assumptions of prognosis research and clinical practice and sound.
The stratified approach to prognosis research described by the authors would seem to direct researchers away from reliance on an assumption that, absent evidence to the contrary, one should assume that a relative risk reduction observed in a clinical trial will apply to all baseline rates. But until there exist far more stratified research than there is now, much decision-making will rely on that assumption. And, like other efforts to provide guidance on research and decision-making, including many of its references, the Hingorani article fails to recognize that the assumption is fundamentally unsound.
Consider a situation where in a clinical trial an intervention is observed to reduce the risk of an adverse outcome from 20% to 10%. The trial itself reveals the 10 percentage point absolute risk reduction that is the crucial consideration for most clinical decision-making. But in situations involving different baseline rates, the standard approach for applying information from the trial to estimate the absolute risk reduction (and corresponding number-needed to treat) would be – again, absent sound evidence of a differential effect as the concept generally is understood – to apply the observed 50% relative risk reduction to other baseline rates. Thus, where the baseline rate is 40%, the estimated absolute risk reduction would be 20 percentage points, reflecting the pattern noted by the authors whereby the constant rate ratio yields a larger absolute reduction for the higher baseline rate.
But in applying the information from the clinical trial to estimate absolute risk reductions involving baseline rates other that in the trial, there is no rational basis for applying the observed 50% reduction from 20% to 10% in the adverse outcome rather than the observed 12.5% increase from 80% to 90% in the opposite, favorable outcome. Applying the 12.5% figure to the favorable outcome in the situation where the baseline adverse outcome rate is 40% would increase the 60% favorable outcome rate (reduce the 40% adverse outcome rate) by 7.5 percentage points. This is a smaller absolute risk reduction than observed with the 20% baseline adverse outcome rate.
How then might one employ the information from the trial to estimate the absolute risk reduction in the case of the 40% baseline adverse outcome rate. The most defensible course would be, while ignoring any ratio relationship, to derive from the 20% control and 10% treated adverse outcome rates in the trial, or the corresponding 80% and 90% favorable outcome rates, the difference between the means of hypothesized normal underlying risk distributions. Either approach yields a figure of .44 standard deviations. Based on the assumption that the intervention shifts underlying distributions by .44 standard deviations, one can estimate that in the case of a baseline adverse outcome rate of 40%, the absolute risk reduction would be about 15.6 percentage points. Tables 3 and 4 of reference 2 provide illustrations of the results of such approach compared with those yielded by the assumption of a constant relative risk reduction for the adverse outcome, constant relative risk increase for the favorable, or constant odds ratio for either outcome.
This approach reflects assumptions that the underlying risk distributions are normal and that an intervention shifts these distributions by the same distance on the x-axis. Both assumptions will typically be unproven. But, unlike assumptions as to the constancy of either rate ratio, these assumptions are not illogical. And, so far as I have been able to determine, there exists no sounder basis for applying risk changes observed in a trial to estimate absolute risk changes involving baseline rates other than that in the trial.
These same considerations undermine the standard approach to subgroup analysis, which regards a departure from a constant risk ratio as a differential effect/subgroup effect/interaction. As I explained here most recently in a comment on Hingorani’s reference 49,[3] and as is implicit in the above discussion, since a factor cannot cause equal proportionate changes to different baseline rates of experiencing an outcome while causing equal proportionate changes to the corresponding rates of experiencing the opposite outcome, it is illogical to regard it as somehow normal that a factor should cause equal proportionate changes in either outcome. Conversely, anytime one observes that a factor causes equal changes in different baseline rates for some outcome (and hence finds no evidence of interaction as to that outcome as the concept is generally understood), one will necessarily find evidence of interaction as to the opposite outcome. More broadly, the rate ratio is an unsound measure of association and prognosis research would do well to cease to employ it at all.
As noted at the outset, the Hingorani paper directs prognosis away from reliance on the assumption of a constant relative risk reduction across different baseline rates. But one can make better progress in that undertaking, as well as better realize its urgency, with a recognition of how unsound that assumption is.
References:
1. Hingorani AD, van der Windt DA, Riley RD, et al. Prognosis research strategy (PROGRESS) 4: Stratified medicine research. BMJ 2013;346:e5793
3. Scanlan JP. The inevitability of interaction. BMJ Dec. 19, 2011 ((responding to Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): http://www.bmj.com/content/326/7382/219?tab=responses
Rapid Response:
Re: Prognosis research strategy (PROGRESS) 4: Stratified medicine research
The four-part PROGRESS series, of which the article by Hingorani et al.[1] is the final, represents an ambitious and sensible effort at refining prognosis research strategy. Such an undertaking, however, should carefully evaluate whether all basic assumptions of prognosis research and clinical practice and sound.
The stratified approach to prognosis research described by the authors would seem to direct researchers away from reliance on an assumption that, absent evidence to the contrary, one should assume that a relative risk reduction observed in a clinical trial will apply to all baseline rates. But until there exist far more stratified research than there is now, much decision-making will rely on that assumption. And, like other efforts to provide guidance on research and decision-making, including many of its references, the Hingorani article fails to recognize that the assumption is fundamentally unsound.
Consider a situation where in a clinical trial an intervention is observed to reduce the risk of an adverse outcome from 20% to 10%. The trial itself reveals the 10 percentage point absolute risk reduction that is the crucial consideration for most clinical decision-making. But in situations involving different baseline rates, the standard approach for applying information from the trial to estimate the absolute risk reduction (and corresponding number-needed to treat) would be – again, absent sound evidence of a differential effect as the concept generally is understood – to apply the observed 50% relative risk reduction to other baseline rates. Thus, where the baseline rate is 40%, the estimated absolute risk reduction would be 20 percentage points, reflecting the pattern noted by the authors whereby the constant rate ratio yields a larger absolute reduction for the higher baseline rate.
But in applying the information from the clinical trial to estimate absolute risk reductions involving baseline rates other that in the trial, there is no rational basis for applying the observed 50% reduction from 20% to 10% in the adverse outcome rather than the observed 12.5% increase from 80% to 90% in the opposite, favorable outcome. Applying the 12.5% figure to the favorable outcome in the situation where the baseline adverse outcome rate is 40% would increase the 60% favorable outcome rate (reduce the 40% adverse outcome rate) by 7.5 percentage points. This is a smaller absolute risk reduction than observed with the 20% baseline adverse outcome rate.
How then might one employ the information from the trial to estimate the absolute risk reduction in the case of the 40% baseline adverse outcome rate. The most defensible course would be, while ignoring any ratio relationship, to derive from the 20% control and 10% treated adverse outcome rates in the trial, or the corresponding 80% and 90% favorable outcome rates, the difference between the means of hypothesized normal underlying risk distributions. Either approach yields a figure of .44 standard deviations. Based on the assumption that the intervention shifts underlying distributions by .44 standard deviations, one can estimate that in the case of a baseline adverse outcome rate of 40%, the absolute risk reduction would be about 15.6 percentage points. Tables 3 and 4 of reference 2 provide illustrations of the results of such approach compared with those yielded by the assumption of a constant relative risk reduction for the adverse outcome, constant relative risk increase for the favorable, or constant odds ratio for either outcome.
This approach reflects assumptions that the underlying risk distributions are normal and that an intervention shifts these distributions by the same distance on the x-axis. Both assumptions will typically be unproven. But, unlike assumptions as to the constancy of either rate ratio, these assumptions are not illogical. And, so far as I have been able to determine, there exists no sounder basis for applying risk changes observed in a trial to estimate absolute risk changes involving baseline rates other than that in the trial.
These same considerations undermine the standard approach to subgroup analysis, which regards a departure from a constant risk ratio as a differential effect/subgroup effect/interaction. As I explained here most recently in a comment on Hingorani’s reference 49,[3] and as is implicit in the above discussion, since a factor cannot cause equal proportionate changes to different baseline rates of experiencing an outcome while causing equal proportionate changes to the corresponding rates of experiencing the opposite outcome, it is illogical to regard it as somehow normal that a factor should cause equal proportionate changes in either outcome. Conversely, anytime one observes that a factor causes equal changes in different baseline rates for some outcome (and hence finds no evidence of interaction as to that outcome as the concept is generally understood), one will necessarily find evidence of interaction as to the opposite outcome. More broadly, the rate ratio is an unsound measure of association and prognosis research would do well to cease to employ it at all.
As noted at the outset, the Hingorani paper directs prognosis away from reliance on the assumption of a constant relative risk reduction across different baseline rates. But one can make better progress in that undertaking, as well as better realize its urgency, with a recognition of how unsound that assumption is.
References:
1. Hingorani AD, van der Windt DA, Riley RD, et al. Prognosis research strategy (PROGRESS) 4: Stratified medicine research. BMJ 2013;346:e5793
2. Scanlan JP. Subgroup effects [Internet]. 2012. [updated 2012 Feb 24; cited 2013 Feb 24] Available from: http://www.jpscanlan.com/scanlansrule/subgroupeffects.html
3. Scanlan JP. The inevitability of interaction. BMJ Dec. 19, 2011 ((responding to Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ 2003;326:219): http://www.bmj.com/content/326/7382/219?tab=responses
Competing interests: No competing interests