Re: Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks
Stressing the clinical significance of the absolute risk reduction achieved by an intervention, Schwartz et al.[1] make a strong argument for journals to require the prominent reporting of absolute risks underlying ratio measures of an intervention’s effect. The argument for reporting absolute risks is even stronger than the authors suggest.
In acknowledging the utility of ratio effect measures, Schwartz and colleagues note that it is often believed that a ratio measure is transportable, meaning that an observed relative change an intervention causes as to one baseline risk will also apply to a different baseline risk. The authors are correct that it is often believed that such is the case.[2,3] But the belief does not have a sound statistical foundation, and, at least in the case of the risk ratio/relative risk, is illogical, since a factor that reduces different baseline rates equal proportionate amounts must necessarily increase the opposite outcome rates different proportionate amounts. That is, using the authors’ example of a .50 relative risk reflecting reductions in the risk of death from (a) 20% to 10%, (b) 1% to 0.5%, and (c) 0.0004% to 0.0002%, the respective increases in the chance of survival are 12.5% (80% increased to 90%), 0.51% (99% increased to 99.5%), and 0.0002% (99.9996% increased to 99.9998%). Since there is no more reason to expect that a factor will cause equal proportionate changes in different mortality rates than there is to expect it to cause equal proportionate changes in different survival rates, there is no reason to expect equal proportionate changes in either type of rate. In fact, features of normal risk distributions provide reasons to expect that a factor that similarly affects groups with different baseline rates will tend to cause larger proportionate changes in the outcome for groups with lower baseline rates while causing larger proportionate changes in the opposite outcome for other groups.[4-7]
The only theoretically sound transportable effect measure is based on the difference between means of hypothesized underlying distributions derived from a pair of rates. For the pairs of rate identified as (a), (b), and (c) in the prior paragraph, such differences would be .4399, .2498, and .1462 standard deviations.[6-7] Thus, for example, based on a study that reduced a baseline rate from of 20% to 10%, in circumstances where the baseline rate is 12% the best estimate of the crucial absolute risk reduction would be approximately 6.7 percentage points (relative risk = .44), not the 6.0 percentage point reduction that would be estimated on the basis of a putatively transportable relative risk of .5 (or the 11 percentage point reduction that would be estimated based on a putatively transportable 1.125 relative risk of survival). On the other hand, based on a study that reduced a baseline rate from 1% to 0.5%, the best estimate of the absolute reduction of a 12% baseline rate would be approximately 4.2 percentage points (relative risk = .65). But one must know the absolute risks in order to derive these estimates.
Swartz et al. excluded meta-analyses and case control studies from their analysis of the reportage of absolute risks because such risks may not be directly calculable from such studies. Because such studies rely on ratio measures without consideration of absolute risks there is a question whether such studies are valid at all.[8,9]
Among the ratio measures employed in the studies that Swartz et al. analyzed was the odds ratio, which most regard merely as proxy for the risk ratio. As is not the case with the risk ratios, however, effects measured by odds ratios tend to be the same regardless of which outcome is examined (in that the odds ratio for one outcome is the reciprocal of the odds ratio for the opposite outcome). Thus, it would not be illogical to regard odds ratios as transportable in the same way that it would be to regard risk ratios as transportable. But like risk ratios, odds ratios are also affected by the overall prevalence of an outcome.[6,7] Hence, an observed odds ratio as to one baseline rate cannot be transported to another baseline rate. As reflected in Table 3 of reference 7, however, the results of doing so, in terms of estimating the crucial absolute risk reduction, would be closer to the result of the approach described two paragraphs above than an approach based on observed relative risks for either mortality or survival. In that regard, the odds ratio, if difficult to interpret, has an advantage over the relative risk. It nevertheless remains a flawed measure.
The article by Swartz and colleagues was published a little over five years ago and then caught my attention because I was often disturbed by the absence of absolute risk information in studies of interest to me. It would be interesting to see the results of a similar systematic review of the reportage of absolute risks today. But to me it does not seem that the frequency of such reportage has increased.
References:
1. Schwartz LS, Woloshin S, Dvorin EL, Welch HG. Ratio measures in leading medical journals: structured review of underlying absolute risks. BMJ 2006;333:1248-1252.
2. Sun X, Briel M. Walter SD, and Guyatt GH. Is as subgroup effect believable? Updating criteria to evaluated the credibility of subgroup analyses. BMJ 2010;340:850-854.
3. Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85: http://www.trialsjournal.com/content/11/1/85
6. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf
Rapid Response:
Re: Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks
Stressing the clinical significance of the absolute risk reduction achieved by an intervention, Schwartz et al.[1] make a strong argument for journals to require the prominent reporting of absolute risks underlying ratio measures of an intervention’s effect. The argument for reporting absolute risks is even stronger than the authors suggest.
In acknowledging the utility of ratio effect measures, Schwartz and colleagues note that it is often believed that a ratio measure is transportable, meaning that an observed relative change an intervention causes as to one baseline risk will also apply to a different baseline risk. The authors are correct that it is often believed that such is the case.[2,3] But the belief does not have a sound statistical foundation, and, at least in the case of the risk ratio/relative risk, is illogical, since a factor that reduces different baseline rates equal proportionate amounts must necessarily increase the opposite outcome rates different proportionate amounts. That is, using the authors’ example of a .50 relative risk reflecting reductions in the risk of death from (a) 20% to 10%, (b) 1% to 0.5%, and (c) 0.0004% to 0.0002%, the respective increases in the chance of survival are 12.5% (80% increased to 90%), 0.51% (99% increased to 99.5%), and 0.0002% (99.9996% increased to 99.9998%). Since there is no more reason to expect that a factor will cause equal proportionate changes in different mortality rates than there is to expect it to cause equal proportionate changes in different survival rates, there is no reason to expect equal proportionate changes in either type of rate. In fact, features of normal risk distributions provide reasons to expect that a factor that similarly affects groups with different baseline rates will tend to cause larger proportionate changes in the outcome for groups with lower baseline rates while causing larger proportionate changes in the opposite outcome for other groups.[4-7]
The only theoretically sound transportable effect measure is based on the difference between means of hypothesized underlying distributions derived from a pair of rates. For the pairs of rate identified as (a), (b), and (c) in the prior paragraph, such differences would be .4399, .2498, and .1462 standard deviations.[6-7] Thus, for example, based on a study that reduced a baseline rate from of 20% to 10%, in circumstances where the baseline rate is 12% the best estimate of the crucial absolute risk reduction would be approximately 6.7 percentage points (relative risk = .44), not the 6.0 percentage point reduction that would be estimated on the basis of a putatively transportable relative risk of .5 (or the 11 percentage point reduction that would be estimated based on a putatively transportable 1.125 relative risk of survival). On the other hand, based on a study that reduced a baseline rate from 1% to 0.5%, the best estimate of the absolute reduction of a 12% baseline rate would be approximately 4.2 percentage points (relative risk = .65). But one must know the absolute risks in order to derive these estimates.
Swartz et al. excluded meta-analyses and case control studies from their analysis of the reportage of absolute risks because such risks may not be directly calculable from such studies. Because such studies rely on ratio measures without consideration of absolute risks there is a question whether such studies are valid at all.[8,9]
Among the ratio measures employed in the studies that Swartz et al. analyzed was the odds ratio, which most regard merely as proxy for the risk ratio. As is not the case with the risk ratios, however, effects measured by odds ratios tend to be the same regardless of which outcome is examined (in that the odds ratio for one outcome is the reciprocal of the odds ratio for the opposite outcome). Thus, it would not be illogical to regard odds ratios as transportable in the same way that it would be to regard risk ratios as transportable. But like risk ratios, odds ratios are also affected by the overall prevalence of an outcome.[6,7] Hence, an observed odds ratio as to one baseline rate cannot be transported to another baseline rate. As reflected in Table 3 of reference 7, however, the results of doing so, in terms of estimating the crucial absolute risk reduction, would be closer to the result of the approach described two paragraphs above than an approach based on observed relative risks for either mortality or survival. In that regard, the odds ratio, if difficult to interpret, has an advantage over the relative risk. It nevertheless remains a flawed measure.
The article by Swartz and colleagues was published a little over five years ago and then caught my attention because I was often disturbed by the absence of absolute risk information in studies of interest to me. It would be interesting to see the results of a similar systematic review of the reportage of absolute risks today. But to me it does not seem that the frequency of such reportage has increased.
References:
1. Schwartz LS, Woloshin S, Dvorin EL, Welch HG. Ratio measures in leading medical journals: structured review of underlying absolute risks. BMJ 2006;333:1248-1252.
2. Sun X, Briel M. Walter SD, and Guyatt GH. Is as subgroup effect believable? Updating criteria to evaluated the credibility of subgroup analyses. BMJ 2010;340:850-854.
3. Kent DM, Rothwell PM, Ionnadis JPA, et al. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials 2010,11:85: http://www.trialsjournal.com/content/11/1/85
4. Scanlan JP. Race and mortality. Society 2000;37(2):19-35 (reprinted in Current 2000 (Feb)): http://www.jpscanlan.com/images/Race_and_Mortality.pdf
5. Scanlan JP. Divining difference. Chance 1994;7(4):38-9,48: http://jpscanlan.com/images/Divining_Difference.pdf
6. Scanlan JP. Interpreting Differential Effects in Light of Fundamental Statistical Tendencies, presented at 2009 Joint Statistical Meetings of the American Statistical Association, International Biometric Society, Institute for Mathematical Statistics, and Canadian Statistical Society, Washington, DC, Aug. 1-6, 2009: http://www.jpscanlan.com/images/JSM_2009_ORAL.pdf
7. Subgroup Effects sub-page of Scanlan’s Rule page of jpscanlan.com:
http://www.jpscanlan.com/scanlansrule/subgroupeffects.html
8. Meta-analysis sub-page of Scanlan’s Rule page of jpscanlan.com:
http://jpscanlan.com/scanlansrule/metaanalysis.html
9. Case Control sub-page of Scanlan’s Rule page of jpscanlan.com:
http://jpscanlan.com/scanlansrule/casecontrolstudies.html
Competing interests: No competing interests