How to estimate the health benefits of additional research and changing clinical practiceBMJ 2015; 351 doi: https://doi.org/10.1136/bmj.h5987 (Published 25 November 2015) Cite this as: BMJ 2015;351:h5987
All rapid responses
We thank Tuffaha and Scuffham for their considered response to our article.
We of course agree that in order to utilise the results of meta-analysis it is necessary to account for the quality of the included studies, the risk of bias and the relevance of the pooled result to the decision problem.
Tuffaha and Scuffham expand further our discussion of the other aspects of outcome to consider. More fully characterising the uncertain aspects of the decision problem would indeed alter the estimated consequences of uncertainty; any resultant maximum value of research will pertain to the specified final outcome, the parameters successfully quantified within the analysis, and the modelled relationships between parameters. Increasing the complexity of the model allows for the incorporation of more aspects of outcome, but it is unlikely that any model could truly quantify all aspects of decision uncertainty. Ultimately, similar considerations are required in utilising the results of this type analysis as with the use of the results of the underlying meta-analysis. That is, what aspects of outcomes are included in the model, what is the risk of bias, and what is the relevance of the result to the decision problem? Our suggested approach relies upon a simple model with few parameters, which has the advantages of being easy to implement in terms of both time and level of expertise required and in making clear precisely what aspects of outcome are captured. The resultant maximum value of additional research is a useful piece of information, and represents a distinct improvement over considerations of statistical significance, both in terms of prioritising further research and in prioritising the use of more resource demanding methods for estimating the expected net improvement in health benefits that could be provided by alternative study designs.
Karl Claxton, Susan Griffin, Hendrik Koffijberg, Claire Rothery (nee McKenna)
Competing interests: No competing interests
We have read with great interest the paper by Claxton et al on how to estimate the health benefits of additional research and changing clinical practice.(1) The authors extend the principles of value of information analysis, which quantifies the value of additional research in reducing decision uncertainty in economic evaluation, to the results of meta-analysis and systematic reviews. Under their approach, the consequences of the uncertainty in the pooled relative effectiveness estimates of an intervention is used predict the benefits of additional research at the level of the population expected to benefit from that intervention.(1) This simple, easy to perform, method would be a useful addition to the efforts of rationalising spending on research. The estimation of the expected benefits of additional research has been usually done alongside economic evaluations; therefore, the proposed extension method would be valuable for organisations that do not have the capacity to perform economic evaluations or where economic evaluations are not part of their decision making processes. A prominent example is using this approach to prioritise Cochrane reviews for updating.
Despite the merits of the proposed methods, we highlight some of its limitations, which we are sure that Claxton et al are aware of. First, this quantitative approach should not replace the qualitative assessment of the meta-analysis and the risk of bias in the included studies since this bias may overestimate or underestimate treatment effect. Second, the benefits of additional research estimated quantitatively in the suggested method represent the maximum (i.e., upper bound) of the value of additional research to resolve the uncertainty in the end point of interest; however, this approach may not consider all aspects of the decision problem (e.g., the cost of the intervention, its side effects, and its utility) and how these aspects interact, which could underestimate decision uncertainty and subsequently affect the accuracy of the estimated research benefits and the expected benefits of implementing research findings. It would be interesting to see how the results obtained from this approach compare with the conclusions obtained using economic decision modelling including all relevant parameters. Third, the estimated benefits of research from this approach may not be sufficient to inform the need for further research. A sufficient condition would be established by comparing the expected benefits of additional research from a trial with a specific sample size (i.e., the expected value of sample information) and the expected cost of that research study.(2, 3) This comparison could also be used to optimise any trial design by choosing the designs with the maximum net research benefit, which is the difference between the expected research study benefits and its expected cost.(4) This is equally important in guiding research prioritisation. For instance, research Project A might have higher expected benefits than Project B, but the cost of Project A could be high resulting in a reduction in its expected net benefit. Here, Project B should be prioritised if its expected net benefit exceeds that of Project A. Therefore, the estimates from the proposed extension method are useful as crude measure (i.e., first hurdle) to screen research proposals to inform if additional research is potentially worthwhile. If the expected benefits of research are too small compared with the expected research costs, then there is no value in conducting more studies; however, when there appears to be a significant value of additional research, further analysis is required including full cost-effectiveness and value of information analysis.
This approach suggested by Claxton et al together with the recently introduced methods for efficient value of additional research estimation (5, 6) should encourage more routine assessments of the benefits and costs of research as well as the benefits and costs of earlier implementation of novel technologies versus waiting for more evidence, which would better inform decisions and optimise research prioritisation.
Haitham W. Tuffaha, MSc, MBA
Research Fellow, Centre for Applied Health Economic, School of Medicine
& Menzies Health Institute Queensland
Griffith University, Queensland, Australia
Paul Scuffham, PhD
Director - Population and Social Health Research Program, Menzies Health Institute Queensland
Professor, Health Economics
School of Medicine
1. Claxton K, Griffin S, Koffijberg H, McKenna C. How to estimate the health benefits of additional research and changing clinical practice. BMJ (Clinical research ed). 2015;351:h5987.
2. Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341-64.
3. Willan AR, Pinto EM. The value of information and optimal clinical trial design. Statistics in Medicine. 2005;24(12):1791-806.
4. Tuffaha HW, Reynolds H, Gordon LG, Rickard CM, Scuffham PA. Value of information analysis optimizing future trial design from a pilot study on catheter securement devices. Clin Trials. 2014;11(6):648-56.
5. Strong M, Oakley JE, Brennan A, Breeze P. Estimating the Expected Value of Sample Information Using the Probabilistic Sensitivity Analysis Sample: A Fast, Nonparametric Regression-Based Method. Med Decis Making. 2015;35(5):570-83.
6. Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Making. 2014;34(3):311-26.
Competing interests: No competing interests