Intended for healthcare professionals

Research Methods & Reporting

How to estimate the health benefits of additional research and changing clinical practice

BMJ 2015; 351 doi: (Published 25 November 2015) Cite this as: BMJ 2015;351:h5987

Useful but not sufficient

Dear Editor,

We have read with great interest the paper by Claxton et al on how to estimate the health benefits of additional research and changing clinical practice.(1) The authors extend the principles of value of information analysis, which quantifies the value of additional research in reducing decision uncertainty in economic evaluation, to the results of meta-analysis and systematic reviews. Under their approach, the consequences of the uncertainty in the pooled relative effectiveness estimates of an intervention is used predict the benefits of additional research at the level of the population expected to benefit from that intervention.(1) This simple, easy to perform, method would be a useful addition to the efforts of rationalising spending on research. The estimation of the expected benefits of additional research has been usually done alongside economic evaluations; therefore, the proposed extension method would be valuable for organisations that do not have the capacity to perform economic evaluations or where economic evaluations are not part of their decision making processes. A prominent example is using this approach to prioritise Cochrane reviews for updating.

Despite the merits of the proposed methods, we highlight some of its limitations, which we are sure that Claxton et al are aware of. First, this quantitative approach should not replace the qualitative assessment of the meta-analysis and the risk of bias in the included studies since this bias may overestimate or underestimate treatment effect. Second, the benefits of additional research estimated quantitatively in the suggested method represent the maximum (i.e., upper bound) of the value of additional research to resolve the uncertainty in the end point of interest; however, this approach may not consider all aspects of the decision problem (e.g., the cost of the intervention, its side effects, and its utility) and how these aspects interact, which could underestimate decision uncertainty and subsequently affect the accuracy of the estimated research benefits and the expected benefits of implementing research findings. It would be interesting to see how the results obtained from this approach compare with the conclusions obtained using economic decision modelling including all relevant parameters. Third, the estimated benefits of research from this approach may not be sufficient to inform the need for further research. A sufficient condition would be established by comparing the expected benefits of additional research from a trial with a specific sample size (i.e., the expected value of sample information) and the expected cost of that research study.(2, 3) This comparison could also be used to optimise any trial design by choosing the designs with the maximum net research benefit, which is the difference between the expected research study benefits and its expected cost.(4) This is equally important in guiding research prioritisation. For instance, research Project A might have higher expected benefits than Project B, but the cost of Project A could be high resulting in a reduction in its expected net benefit. Here, Project B should be prioritised if its expected net benefit exceeds that of Project A. Therefore, the estimates from the proposed extension method are useful as crude measure (i.e., first hurdle) to screen research proposals to inform if additional research is potentially worthwhile. If the expected benefits of research are too small compared with the expected research costs, then there is no value in conducting more studies; however, when there appears to be a significant value of additional research, further analysis is required including full cost-effectiveness and value of information analysis.

This approach suggested by Claxton et al together with the recently introduced methods for efficient value of additional research estimation (5, 6) should encourage more routine assessments of the benefits and costs of research as well as the benefits and costs of earlier implementation of novel technologies versus waiting for more evidence, which would better inform decisions and optimise research prioritisation.

Haitham W. Tuffaha, MSc, MBA
Research Fellow, Centre for Applied Health Economic, School of Medicine
& Menzies Health Institute Queensland
Griffith University, Queensland, Australia

Paul Scuffham, PhD
Director - Population and Social Health Research Program, Menzies Health Institute Queensland
Professor, Health Economics
School of Medicine
Griffith University
1. Claxton K, Griffin S, Koffijberg H, McKenna C. How to estimate the health benefits of additional research and changing clinical practice. BMJ (Clinical research ed). 2015;351:h5987.
2. Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341-64.
3. Willan AR, Pinto EM. The value of information and optimal clinical trial design. Statistics in Medicine. 2005;24(12):1791-806.
4. Tuffaha HW, Reynolds H, Gordon LG, Rickard CM, Scuffham PA. Value of information analysis optimizing future trial design from a pilot study on catheter securement devices. Clin Trials. 2014;11(6):648-56.
5. Strong M, Oakley JE, Brennan A, Breeze P. Estimating the Expected Value of Sample Information Using the Probabilistic Sensitivity Analysis Sample: A Fast, Nonparametric Regression-Based Method. Med Decis Making. 2015;35(5):570-83.
6. Strong M, Oakley JE, Brennan A. Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Making. 2014;34(3):311-26.

Competing interests: No competing interests

02 December 2015
Haitham W. Tuffaha
Research Fellow in Health Economics
Paul A Scuffham
Centre for Applied Health Economic, School of Medicine & Menzies Health Institute Queensland, Griffith University
School of Medicine, Griffith University, Logan Campus, 4131, QLD, Australia