BMJ  2005;330:65 (8 January), doi:10.1136/bmj.38285.482350.82 (published 15 December 2004)

Paper

Comparing estimates of cost effectiveness submitted to the National Institute for Clinical Excellence (NICE) by different organisations: retrospective study

¹ Health Economics Research Group, Brunel University, Uxbridge UB8 3PH, ² Economics Department, University of York, Heslington, York YO1 5DD, ³ Health Policy Unit, Department of Public Health and Policy, London School of Health and Tropical Medicine, London WC1E 7HT, ⁴ Department of Community Health Sciences, St George's Hospital Medical School, London SW17 0RE

Abstract

Objective To assess the association between different types of organisation and the results from economic evaluations.

Design Retrospective pairwise comparison of evidence submitted to the technology appraisal programme of the National Institute for Clinical Excellence (NICE) by manufacturers of the relevant healthcare technologies and by contracted university based assessment groups.

Data sources Data from the first 62 appraisals.

Main outcome measure Incremental cost effectiveness ratios.

Results Data from 27 of the 62 appraisals could be compared. The analysis of 54 pairwise comparisons showed that manufacturer's estimates of incremental cost effectiveness ratios were lower (suggesting a more cost effective use of resources) than those produced by the assessment groups (25 were lower, 29 were the same, none were higher, P < 0.01). Restriction of this dataset to include only one pairwise comparison per appraisal (27 pairs) produced a similar result (21 were lower, two were the same, four were higher, P < 0.001).

Conclusions The estimated incremental cost effectiveness ratios submitted by manufacturers were on average significantly lower than those submitted by the assessment groups. These results show that an important role of NICE's appraisal committee, and of decision makers in general, is to determine which economic evaluations, or parts of evaluations, should be given more credence.

Introduction

Evidence suggests that profit making organisations are more likely to report favourable results and conclusions from clinical studies than non-profit organisations. Three studies have formally assessed this association with economic evaluations. One showed that evaluations of products sponsored by drug companies were less likely to report unfavourable qualitative conclusions (5% v 38%; P = 0.04) compared with studies sponsored by non-profit organisations.¹ The other two studies reported similar findings.^{2 3}

The technology appraisals programme of the National Institute for Clinical Excellence (NICE) provides guidance to the NHS in England and Wales on the use of new and existing health technologies. Manufacturers (or relevant UK agents) of the relevant technology, professional groups, and national groups representing patients submit evidence. Collectively, these three groups are known as consultees. An academic centre commissioned through the NHS's health technology assessment programme, called an assessment group, also assesses the evidence. Guidance to the NHS is formulated with regard to this evidence, which includes information on cost effectiveness.

The criterion traditionally used to assess cost effectiveness is the magnitude of a statistic known as the incremental cost effectiveness ratio, defined as the difference in costs between two technologies divided by the difference in their benefits.⁴ The lower the incremental cost effectiveness ratio, the more cost effective a technology.

We evaluated the association between source of funding and the results from economic evaluations submitted to NICE's technology appraisals programme.

Methods

Detailed information regarding the technology appraisals programme is available elsewhere (www.nice.org.uk).

We extracted estimated incremental cost effectiveness ratios for the first 62 technology appraisals, issued between March 2000 and May 2003. Some contained guidance on more than one aspect of a technology, so each recommendation was treated as a separate observation for which cost effectiveness estimates could exist. We included pairings of estimates in the final dataset only if the manufacturer and the assessment group had each submitted at least one incremental cost effectiveness ratio for a technology by using the same health outcome measure. We assumed that results from cost minimisation analyses indicated clinical equivalence, together with an increase or decrease in cost, and included them in the analysis accordingly.

We abstracted estimates from published guidance, assessment reports, and manufacturers' submissions, and these were double checked by a second reviewer. Where the two reviewers' estimates did not match, we sought opinion from the relevant technology analyst at NICE.

Statistical analysis
We used two different approaches to analysis. The first categorised incremental cost effectiveness ratio estimates according to a five point scale. We chose the value of £30 000 (42 790, $55 770) per unit of health outcome as one upper limit because of published comments on NICE's decisions to date.^{5 6} By comparing categories of estimates (as opposed to continuous variables) we avoided some of the problems posed by negative estimates.

Our null hypothesis was that the incremental cost effectiveness ratios produced by the assessment groups would not differ systematically from those produced by manufacturers.

Our second analysis made only one comparison between the estimates of the assessment groups and those of the manufacturers involved in each technology appraisal, because more than one pairwise comparison of incremental cost effectiveness ratio estimates could occur (as in the first method). For example, there might have been more than one manufacturer per appraisal or separate results for different subgroups of patients. Such clustered estimates will almost certainly be correlated and, if all such estimates are used, will overestimate the accuracy of the difference being investigated (see bmj.com for details). This analysis was based on point ICER estimates.

Results

Of the 62 appraisals, we excluded 35, leaving 27 appraisals containing 54 pairwise comparisons. Over 80% of the manufacturers' incremental cost effectiveness ratios were between dominant (meaning that the appraised technology was considered to be less costly and more effective than the comparator technology) and £15 000 per unit of outcome (categories 0 and 1 combined), whereas the assessment groups' estimates were more uniformly distributed over the five categories (fig 1). Only four (7%) estimates from manufacturers were above £30 000 per unit of outcome compared with 19 (35%) estimates from the assessment groups.

View larger version (19K): [in this window] [in a new window]   Fig 1 Distribution of estimates of incremental cost effectiveness ratios from assessment groups and manufacturers according to category of cost effectiveness (0=negative incremental cost effectiveness ratios favouring use of technology or, in case of cost minimisation analysis, cheaper than alternative technology; 1=£1-£15 000; 2=£15 001-£30 000; 3=>£30 000; 4=negative incremental cost effectiveness ratios not favouring use of technology or, in case of cost minimisation analysis, more expensive than alternative technology)

 

There were 25 cases of negative rank (where the manufacturers' estimates were lower than those of the corresponding assessment group), 29 ties (where the estimates were in the same category), and no positive ranks (P < 0.01). Figure 2 plots the log of these pairwise comparisons; the 45° line indicates identical incremental cost effectiveness ratios. Points above this line indicate that a manufacturer's estimate was higher (less favourable) than the estimate produced by the assessment group, whereas points below this line indicate the manufacturer submitted a lower (more favourable) estimate than that of the assessment group. The analysis conducted with the second method reduced the 54 pairs of incremental cost effectiveness ratios to 27 pairs, of which two were a tie. The analysis showed a similar result. In 21 instances the manufacturer submitted a lower estimate than that of the assessment group and in the four remaining it was higher (P < 0.001).

View larger version (24K): [in this window] [in a new window]   Fig 2 Logged pairwise comparison of incremental cost effectiveness ratios. Only 45 of the 54 pairs of recommendations have been plotted because the remaining nine pairs contained at least one negative estimate and could not be logged. However, in six of nine pairs the estimates from the two sources were broadly similar and in three pairs the estimates reported by manufacturers were much more favourable towards the technology than those reported by the assessment groups

 

Discussion

Our analysis showed that the estimates of incremental cost effectiveness ratio submitted by manufacturers were on average significantly lower than those provided by the assessment groups. This finding is similar to those reported by others.^1-3 Unlike previous studies, however, ours is based on specific pairwise comparisons of estimates that have been put forward for the same purposes and constructed with the same terms of reference.

Whether differences between competing cost effectiveness estimates ultimately matter in terms of decision making depends on several factors, including the level of uncertainty surrounding the incremental cost effectiveness ratios⁷ and their absolute value.

What is already known on this topic Economic evaluations are used to produce estimates of the cost effectiveness of healthcare technologies Profit making organisations are more likely to report favourable outcomes from clinical and economic studies than non-profit organisations One of the key roles of NICE's appraisal committee is to judge the appropriateness of each evaluation and to determine which estimate is the most reasonable What this study adds Economic evaluations submitted by manufacturers to NICE's technology appraisals programme were significantly more favourable than evaluations produced by academic research groups The findings pose questions as to the appropriate methods and processes used by NICE (and reimbursement agencies in general) in the determination of cost effectiveness

NICE will soon be adopting a "reference case" approach designed to be the most appropriate for the NHS to improve consistency across all submitted economic evaluations.⁸ It is therefore feasible that differences between estimated incremental cost effectiveness ratios submitted will be smaller in the future.

We used two different analyses because, though the first method was more informative, we were concerned about the possibility of overstating the accuracy of the difference being investigated. Both approaches, however, produced similar results.

Limitations of study
The evaluations submitted by a manufacturer and the corresponding assessment group are not totally independent, as the assessment group has usually had the opportunity to review the manufacturer's evaluation before completing its own. In contrast, manufacturers do not have the opportunity to review the assessment group's evaluation until it has been submitted to the institute. It is difficult to gauge, however, whether this is likely to lead to systematic differences in the paired ratios or to influence the extent or direction of this difference.

It is also feasible that differences in estimates are not uniform across all categories of cost effectiveness. Logic suggests that because technologies with high incremental cost effectiveness ratios are less likely to be recommended for use within the NHS, groups with vested interests in a technology would be more likely to underestimate incremental cost effectiveness when the "true" estimate is high than when it is low. Insufficient data existed to test this hypothesis.

The reasons that incremental cost effectiveness ratios varied within each appraisal have not yet been investigated. For the moment, we can conclude only that estimates from manufacturers and assessment groups were significantly different from each other, not that biased results were associated with either type of organisation.

Conclusions
Incremental cost effectiveness ratios submitted to NICE's technology appraisals programme by different types of organisation were significantly different from each other. This questions the appropriate methods and processes used by the institute when determining cost effectiveness, and highlights the need for decision makers to have unhindered access to the methods used to produce cost effectiveness estimates.

---

This is the abridged version of an article that was posted on bmj.com on 15 December 2004: http://bmj.com/cgi/doi/10.1136/bmj.38285.482350.82

We thank all the members of the appraisals team at NICE who helped collect the data, Caroline Sabin (professor of epidemiology and medical statistics, University College London) for her advice on the statistical analysis, Martin Buxton (professor of health economics, Brunel University) for his comments on earlier drafts of the manuscript, and the reviewers for their constructive comments.

Contributors: See bmj.com

Funding: None.

Competing interests: AHM and AJF are members of NICE's appraisals team. DF was a member of the appraisals team when this study was undertaken.

Ethical approval: Not required.

References

1. Friedberg M, Saffran B, Stinson TJ, Nelson W, Bennett CL. Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA 1999;15: 1453-7.
2. Lexchin J, Bero LA, DJulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326: 1167-70.[Abstract/Free Full Text]
3. Azimi NA, Welch HG. The effectiveness of cost-effectiveness analysis in containing costs. J Gen Intern Med 1998;10: 664-9.
4. Drummond MF, O'Brien B, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press, 1997.
5. Devlin N, Parkin D. Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Econ 2004;13: 437-52.[CrossRef][Web of Science][Medline]
6. Raftery J. NICE: faster access to modern treatments? Analysis of guidance on health technologies. BMJ 2001;323: 1300-3.[Free Full Text]
7. Van Hout BA, Al MJ, Gordon GS, Rutten FF. Costs, effects and C/E ratios alongside clinical trials. Health Econ 1994;3: 309-19.[Web of Science][Medline]
8. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996.

(Accepted 18 October 2004)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

NICE and the challenge of cancer drugs

James Raftery
BMJ 2009 338: b67. [Extract] [Full Text]

Review of NICE's recommendations, 1999-2005

James Raftery
BMJ 2006 332: 1266-1268. [Full Text] [PDF]

Bias in published cost effectiveness studies: systematic review

Chaim M Bell, David R Urbach, Joel G Ray, Ahmed Bayoumi, Allison B Rosen, Dan Greenberg, and Peter J Neumann
BMJ 2006 332: 699-703. [Abstract] [Full Text] [PDF]

Decision makers need to judge credibility of economic evaluations
BMJ 2005 330: 0. [Full Text]

This article has been cited by other articles:

• Clement, F. M., Harris, A., Li, J. J., Yong, K., Lee, K. M., Manns, B. J. (2009). Using Effectiveness and Cost-effectiveness to Make Drug Coverage Decisions: A Comparison of Britain, Australia, and Canada. JAMA 302: 1437-1443 [Abstract] [Full text]  
• Raftery, J. (2009). NICE and the challenge of cancer drugs. BMJ 338: b67-b67 [Full text]  
• McCabe, C. (2007). Guidance on Good Practice in Cost-Effectiveness Modeling: Is More Needed?. Med Decis Making 27: 350-351  
• Walker, S., Palmer, S., Sculpher, M. (2007). The role of NICE technology appraisal in NHS rationing. Br Med Bull 0: ldm007v1-14 [Abstract] [Full text]  
• Clegg, A. J., Scott, D. A., Loveman, E., Colquitt, J. L., Royle, P., Bryant, J. (2006). Clinical and cost-effectiveness of left ventricular assist devices as a bridge to heart transplantation for people with end-stage heart failure: a systematic review and economic evaluation. Eur Heart J 27: 2929-2938 [Abstract] [Full text]  
• Raftery, J. (2006). Review of NICE's recommendations, 1999-2005.. BMJ 332: 1266-1268 [Full text]  
• Bell, C. M, Urbach, D. R, Ray, J. G, Bayoumi, A., Rosen, A. B, Greenberg, D., Neumann, P. J (2006). Bias in published cost effectiveness studies: systematic review. BMJ 332: 699-703 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Negotiating the price of innovative treatments based on cost-effectiveness techniques

Andrea Messori, et al.
bmj.com, 14 Jan 2005 [Full text]

Online poll

Find out more >>