How to estimate the health benefits of additional research and changing clinical practice

Health impact of additional research

The result of a meta-analysis describes both the central estimate of effectiveness within a range of plausible values (for example, a confidence interval) and the likelihood of observing particular values (for example, P values or error probability).2 This distribution of values describes the uncertainty about the relative effectiveness of an intervention.3 Decisions to commission further research imply that this uncertainty is unacceptable or important enough to justify the research costs. When the range of plausible values can support more than one intervention this uncertainty has health consequences, because for any treatment choice there will be a chance that another intervention would have improved health outcomes to a greater extent. The importance of the uncertainty is indicated by the scale of these health consequences. The chance that an intervention is not the most effective, how much less effective the intervention is likely to be, and the number of patients facing this uncertain treatment choice are all factors that contribute to the health consequences of this uncertainty.

The chance that an intervention is not the most effective is indicated by the standard results of meta-analysis. By applying a relative measure of effect (for example, an odds ratio for mortality) to an estimate of baseline risk of an adverse outcome, it is possible to estimate the amount of health that is potentially lost by adopting a less effective intervention. The web appendix provides a worked example. In this way, the number of patients facing the uncertain treatment choice allows the estimated consequences of uncertainty to be expressed as the health outcomes that could have been achieved for the patient population.

Further research can reduce (although not entirely eliminate) uncertainty and its associated health consequences, because the range of values about a central estimate of effect do not exclude the null effect or possible harm but instead render it increasingly unlikely.1 4 Therefore, an estimate of the potential adverse health consequences of uncertainty represents an upper bound on the potential benefits of further research. It can be used to (1) prioritise proposed research, (2) consider whether the potential benefits are likely to justify the costs, and (3) indicate when existing evidence is sufficient to warrant implementation because further research is unlikely to be worthwhile.4 5

Health impact of changing clinical practice

Health outcomes can be improved by ensuring that the findings of existing research are more fully adopted within clinical practice.6 By combining the central estimate of relative effect from meta-analysis with baseline risk estimates of the particular outcome, we can estimate the expected (absolute) benefits of implementing research findings. When combined with an estimate of the incidence of disease or clinical events, and the current levels of use of the available interventions, the expected population health benefits of implementation efforts can also be estimated (the web appendix provides a worked numerical example). This estimate allows a direct comparison of undertaking further research versus implementation of current evidence as two different ways to improve health outcomes.

Of course, further research might also influence clinical practice and contribute to implementation efforts. However, a clear distinction is necessary because further research is not the only, or necessarily the most effective, way to change clinical practice. In view of other mechanisms such as more effective dissemination of evidence, guidelines, incentives, and sanctions, conducting research to influence clinical practice might be an inappropriate use of scarce research capacity to generate greater health benefits elsewhere, and could disadvantage those patients participating in the research.6

Example: use of streptokinase in early thrombolysis to treat myocardial infarction

In this example, we consider clinical trials that investigated early thrombolysis using streptokinase (versus no thrombrolysis) after acute myocardial infarction. Full details of trials are detailed in a reference at the end of the web appendix. The sequence of these clinical trials illustrates how the values of further research and of implementation can be derived from the results of meta-analysis.7 8 9 10 Figure 1⇓ lists the sequence of trials showing the effects of streptokinase on mortality in a forest plot.7 9 A random effects meta-analysis of these trials strongly favoured streptokinase with an odds ratio of mortality of 0.76 (95% confidence interval 0.67 to 0.85).

Fig 1 Random effects meta-analysis of sequence of trials of early thrombolysis using streptokinase, with the endpoint of mortality

• Download figure
• Open in new tab
• Download powerpoint

In figure 2⇓, a cumulative meta-analysis shows how this evidence accumulated over time: the effect on mortality and its associated uncertainty are updated with each subsequent trial.9 The forest plot demonstrates that the balance of evidence quickly favoured streptokinase and the degree to which uncertainty about its effectiveness generally declined as subsequent trials reported. For example, following the European 3 trial, the probability that streptokinase was more effective exceeded 0.95 in both the trial and random effects meta-analysis. By the time the ISIS-2 trial was reported, any remaining uncertainty was effectively resolved. A number of important questions emerge (box).

Fig 2 Cumulative random effects meta-analysis of sequence of trials of early thrombolysis using streptokinase, with the endpoint of mortality

• Download figure
• Open in new tab
• Download powerpoint

What are the potential health benefits of subsequent trials?

Although the balance of evidence quickly favoured streptokinase in this example, there remained some uncertainty about its effectiveness at each point in this sequence of trials—that is, streptokinase use following acute myocardial infarction might increase, rather than reduce, mortality. The expected health consequences of this uncertainty depend not only on the chance that streptokinase was not effective, but also on how much less effective than no thrombolysis it was likely to be and the incidence of patients eligible for early thrombolysis.

The chance that streptokinase was not effective is indicated by the confidence intervals for the odds ratios in figure 2, in particular, where the confidence interval spans the line of unity (where the intervention effect lies in favour of control). How much less effective it may be is estimated by combining the range of odds ratios that exceed one with an estimate of the baseline mortality risk following acute myocardial infarction. Here, we use the range of values for the baseline odds from a pooled analysis of the control arms of the trials to estimate the baseline mortality risk. We can derive uncertainty in the relative effect size by taking random samples from the distribution of relative effect size (that is described by the meta-analysis), and combining them with random samples from the previously estimated baseline mortality risk. Each sample represents one possible “true” outcome supported by the evidence (see web appendix for a worked example).

Repeating this process can ensure a population of sampled values that describes a distribution of the health consequences of uncertainty. An estimate of the patient population size is required that includes those who have acute myocardial infarction and who reach hospital in time to be eligible for early thrombolysis (incidence in the United Kingdom was estimated to be 204 500 patients per year8).

Figure 3⇓ shows the resulting distribution of health consequences of uncertainty at three points in the cumulative meta-analysis (that is, when the European 1, NHLBI SMIT (the National Heart Lung and Blood Institute streptokinase in myocardial infarction trial), and European 3 trials were reported). Following the European 1 trial, early in the sequence of trials, the balance of evidence favoured streptokinase with a 69% chance that the enzyme was effective and reduced mortality (that is, there was a 69% chance that streptokinase use would have no adverse mortality consequences). However, there was also a 31% chance that streptokinase was not effective and mortality would have been lower if the enzyme had not been used. Examining this 31% chance reveals that the adverse health consequences are not uniform. For example, there was a greater chance of more limited adverse consequences (20% chance of zero to 15 000 deaths per year) than a smaller chance of greater adverse consequences (11% chance of more than 15 000 deaths per year). The mean of this distribution provides an estimate of the expected adverse health consequences of uncertainty and, by implication, the consequential value of conducting further research at this point (6264 deaths averted per year).

Fig 3 Distribution of consequences of uncertainty for streptokinase use following acute myocardial infarction, according to selected trials in example meta-analysis. Numbers in brackets are the expected (mean) number of deaths that could be avoided if uncertainty were eliminated in the decision to use streptokinase at the point of each trial

• Download figure
• Open in new tab
• Download powerpoint

Uncertainty and potential benefits from further research decrease as subsequent trials report. For example, following the NHLBI SMIT trial, the chance that streptokinase was more effective rose to 91% (that is, a 91% chance of zero adverse mortality consequences). The chance of more limited consequences was lower (8.77% chance of zero and 15 000 deaths per year) and the chance of greater consequences was almost eliminated (0.21% chance of more than 15 000 deaths per year). Consequently, the potential benefit of conducting further research fell to averting 306 deaths per year, based on the mean of this distribution of outcomes. By the time the European 3 trial was reported, there was much less uncertainty (98% chance that streptokinase was more effective), and the potential benefits decreased to an estimated mean of 27 deaths averted per year. Once the ISIS-2 trial was reported, uncertainty was effectively eliminated and the potential benefits for further research were negligible.

What is the expected health benefit of implementing research findings?

In this example, we have examined the potential harms due to the uncertainty about whether use of streptokinase reduces mortality. Examining the expected benefits indicates the value of implementing the findings of existing research. In this example, the balance of evidence quickly favoured streptokinase (that is, on average the benefits (odds ratio <1) exceed the harms (odds ratio >1)); thus, one would expect a reduction in mortality by ensuring streptokinase use following acute myocardial infarction was widely implemented.

Figure 4⇓ illustrates the expected benefits of implementing what existing evidence suggests is the most effective intervention and the potential benefits of conducting further research at the same three points in the cumulative meta-analysis as figure 3. Streptokinase use in early thrombolysis was expected to avert 7609 deaths per year following the European 1 trial, 6138 deaths per year following the NHLBI SMIT trial, and 7125 following the European 3 trial. However, it was not until European 3 that both a single trial and the cumulated evidence reported a statistically significant result that favoured streptokinase. Although clinical practice did respond to this evidence with the widespread use of streptokinase, before European 3 the expected benefits of implementing streptokinase use for acute myocardial infarction were already substantial (5696 deaths averted per year). This analysis suggests that waiting to implement research findings until the traditional rules of statistical significance are achieved could come at considerable cost to patient outcomes.1

Fig 4 Expected benefits of implementation (of streptokinase use following acute myocardial infarction) and potential benefits of further research, according to selected trials in example meta-analysis shown in figure 3. Solid line=95% credible interval; dashed line=90% credible interval

• Download figure
• Open in new tab
• Download powerpoint

When should research findings be implemented?

In our example, figure 4 shows that the expected benefits of encouraging the implementation of what existing evidence suggests is the most effective intervention exceed the potential benefits of conducting further research. However, the early implementation of an intervention based on the balance of evidence may not be appropriate if its widespread use means that the type of research required (for example, with adequate control) becomes impossible or more costly.

For example, in 2001, the National Institute for Health and Care Excellence (NICE) approved acetylcholinesterase inhibitors for the treatment of mild and moderate Alzheimer’s disease in the UK’s health service, despite considerable uncertainty about the long term effects on institutionalisation. As a consequence of widespread access to these drugs following NICE approval, a large ongoing publicly funded trial, with much longer follow-up than the existing trials and the only trial with a primary endpoint of institutionalisation, was unable to complete recruitment. Therefore, the benefit of early implementation needs to be balanced against the potential benefits for future patients from further research—it may be better to delay implementation until further research can be completed that will offer benefits to patients over many years.

Furthermore, efforts to implement research findings could be costly and, if successful, it could become difficult and even more costly to reverse if subsequent research finds that the intervention is not sufficiently effective. The question of when research findings should be implemented requires nuanced trade-offs, which can be better informed with a quantitative assessment of these two different ways to improve health outcomes.4

Other aspects of outcome to consider

The trials in our example reported mortality as a primary endpoint, so the benefits of further research and implementation have been expressed as deaths averted. In other contexts (such as Alzheimer’s disease), endpoints associated with morbidity might include events (such as the need for nursing home care) or continuous measures of performance (such as improvements in the mini-mental state examination 9). In these circumstances, the benefits of research and implementation can be expressed as the number of events avoided or, if on a continuous scale, the improvement per patient or the number of patients with a clinically important improvement.

However, the mortality and morbidity endpoints reported in trials commonly reflect only limited aspects of outcome. Linking these endpoints to a more complete measure of health outcome could be informed by meta-analysis of secondary endpoints or by other sources of evidence, while reflecting any additional uncertainty that might be introduced.6

In our example, evidence of expected survival and quality of life after myocardial infarction could be used to translate mortality consequences in figure 3 into life years and quality adjusted life years (QALYs) gained. The translation of health effects to a more generic measure of outcome enables a comparison of the health benefits of further research and implementation efforts across diverse clinical areas. Greater improvements in effectiveness will be needed to justify any additional costs associated with an intervention, which could be reflected by a minimum clinical difference in effect. This additional hurdle will tend to reduce the benefits of implementation and further research.4

Is further research worthwhile?

Estimates of the potential health benefits of research start to indicate its relative priority compared with other areas of research. Whether the costs of research are justified by the potential benefits, can be evaluated by examining whether the resources required might generate similar, or greater, expected health benefits if made available to the NHS? Or, equivalently, whether more NHS resources would be required to generate similar or more health benefits than the proposed research? For example, recent research suggests that it costs the NHS £114 000 (€161 000; $172 000) to avert one death, £25 000 to gain one life year, and £13 000 to gain one QALY.11

This type of quantitative assessment allows research proposals that are likely to be worthwhile to be identified and those that are likely to offer the greatest value to be commissioned from available research funds. This assessment can also indicate whether sufficient resources are being devoted to research—that is, whether potentially worthwhile proposals cannot be commissioned because other proposals are more valuable, and whether patient outcomes could be improved overall by transferring resources from service provision to research.

Conclusions

Systematic reviews and meta-analyses often conclude that further research is needed. The simple extension of standard meta-analysis illustrated in this article could be routinely reported and would provide quantitative assessments of the health benefits of further research and of implementing the findings of existing research. These estimates can help inform the prioritisation and commissioning of research by identifying where and what type of research would be most valuable. For example, the Patient Centred Outcomes Research Institute in the United States is responsible for allocating $3.8bn to comparative effectiveness research as part of recent healthcare reforms, and is already using the principles of this type of analysis to inform its priorities.12

These estimates can also inform other important questions of health policy: for example, a review into medical innovation and technology by the Department of Health and Office of Life Sciences is underway in the UK. Other questions include whether the earlier widespread use of an intervention should be encouraged based on the balance of existing evidence and whether sufficient resources are being devoted to evaluative research.