Is economic evaluation in touch with society's health values?BMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7476.1233 (Published 18 November 2004) Cite this as: BMJ 2004;329:1233
- Joanna Coast (), senior lecturer in health economics1
- Accepted 16 September 2004
The technical expertise required for conducting economic evaluations and interpreting their results continues to increase. Current best practice includes cost effectiveness acceptability curves, net-benefit frameworks, and probabilistic modelling.1 These methods are valuable, but by generating a pseudoscientific aura around economic evaluation, they camouflage critical weaknesses in current techniques. In this article, I describe the evolution of economic evaluation in health care (see box for terminology), explore the assumptions underlying current approaches and the resulting concerns, and suggest an alternative approach.
Why do we need economic evaluation?
People who are not economists often find it difficult to understand the importance of the theory behind the comparison of costs and effects. After all, if we compare two washing machines of equal cost and one works for 10 years and the other for 15, it is clear that the machine lasting 15 years is a better buy. The need for theory arises, however, because interpersonal rather than within individual comparisons are involved; in health care the question is not, generally, whether I choose the 10 or 15 year washing machine but whether I get the 10 year washing machine or you get the one lasting 15 years.
Economic evaluation stems from Paretian welfare economics. It incorporates the principles that individuals are the best judges of their own wellbeing and that, if one person can be made better off without another being made worse off, there is global improvement in welfare. This value judgment is uncontroversial but, in policy terms, practically useless: few policies benefit some individuals without affecting others.
Terminology used in economic evaluation
Cost benefit analysis—Costs valued in money and compared with outcomes also valued in money
Cost effectiveness analysis—Costs valued in money and compared with a single primary outcome
Cost utility analysis—A specific form of cost effectiveness analysis in which outcomes are measured in terms of QALYs gained
QALY—Quality adjusted life year. A measure that combines length of life and quality of life (valued on an index where 1 represents perfect health and 0 represents death) into a single outcome
Cost-benefit analysis translates welfare economics into something that can inform decision making. The compensation principle is used to make interpersonal comparisons. This states that global improvement will occur if individuals gaining from change could potentially compensate those who lose and still be better off. So, if my “welfare” increases more than yours from receiving a washing machine, I could potentially compensate you and still be better off; global welfare increases if I get the washing machine and you do not. An important caveat is that compensation is not actually paid: the aim is to generate global welfare improvements and distribution is irrelevant.
Cost-benefit analysis uses individuals' willingness to pay to assess the benefit of an intervention. So, for example, if group A is willing to pay more for programme X than group B requires in compensation for the loss of programme Y, there is global welfare improvement from allocating resources to X (benefiting A) rather than Y (benefiting B). Funding programme X maximises benefit to society and is thus more efficient.
There are two difficulties with cost-benefit analysis in health care. Firstly, the use of willingness to pay to measure welfare implicitly incorporates income into decision making. This may skew allocation of healthcare resources towards the wealthy. Secondly, many people are uncomfortable with valuing length and quality of life in monetary terms and thus unwilling to participate in such exercises. These difficulties have led to the development of alternative techniques of economic evaluation and, indeed, changes in underlying theory.
Alternatives to welfare economics move away from reliance on individual welfare and instead aim to pursue societal objectives. They are referred to as non-welfarist approaches2 or, more specifically, decision maker approaches3 4 and extra-welfarism.5 These approaches all subscribe to the same healthcare objective: maximising health output from available resources. They differ in the stated theoretical bases for this objective. The decision maker approach focuses on societal objectives as given by decision makers; extra-welfarism is ostensibly based in Sen's notions of functioning and capabilities6 but, given dissent about whether this moral theory provides sufficient basis for relying on health alone as an outcome for economic evaluation,7 the non-welfarist approaches largely rely on achieving societal healthcare objectives.2
Cost effectiveness analysis results from this theoretical perspective, with the quality adjusted life year (QALY) as the chosen outcome. Over the past 10 years, the QALY has become increasingly accepted and used despite the continued existence of theoretical and methodological problems noted during its development.8 Recent advances in economic evaluation are clearly situated in this non-welfarist approach, with emphasis on methodological advances in dealing with uncertainty.1
Camouflaged assumptions associated with recent advances
The non-welfarist approach has enabled economic evaluation to move forward despite the perceived constraints of welfare economics. Three fundamental assumptions of this approach are, however, overlooked: congruence between the objectives of decision makers and those enshrined in economic evaluation; the validity of funnelling multiple outcomes into one simplistic outcome such as the QALY; and the meaningfulness of these complex techniques to decision makers.
Coherent basis for economic evaluation
Non-welfarist approaches to economic evaluation assume that a decision maker acts on behalf of society and that the objective of the healthcare system is thus to maximise health output (as valued by society) from available resources. Even aside from doubts over the existence of this mythical decision maker with a clear set of objectives, the desire to maximise health seems to be largely the objective of economists rather than society. There are three reasons for this view. Firstly, empirical evidence shows that the single objective of maximising health output would not be the basis on which society would wish to allocate its healthcare resources.9 10 Secondly, approaches to rationing based entirely on the cost-utility approach have failed to convince decision makers in practice.11–13 This suggests that maximising health output is not an exclusive objective for decision makers. Thirdly, work on the values on which the NHS is based shows that health is only one of many values and that maximisation of health (as opposed to achieving some form of equity in health) is only a subcategory.14
Thus, ascribing the objective of maximising health to decision makers is no more than a convenience. Any genuine decision maker approach would involve aspects related to equity, need, access, and so on. Economic evaluation based on the non-welfarist approach thus falls into a theoretical void: it is not based on welfare economic theory, nor does it represent all of the decision maker's objectives.
Cost per QALY gained provides sufficient information about the efficiency of an intervention
Even if maximising health output is the appropriate objective for the health system, increasing emphasis on cost-utility analysis results in a conflation of “health” with QALYs gained by the patient. This “funnelling” of various health outcomes into one, simplistic, single measure, is a further camouflaged assumption receiving less attention than it should.
Indeed, cost effectiveness analysis is increasingly aligned with the biostatistical desire for a single primary outcome in design efficient trials. For many interventions, the focus on a single outcome, even with extensive complex statistical analysis of the uncertainty around the estimates obtained, misses the point. For organisational and other complex interventions, in particular, several health outcomes will be important, not just QALYs. For example, a systematic review of interventions to improve access to health and social care after discharge from hospital found patient outcomes related to mortality, function and disability, quality of life, social support, self esteem, cognitive ability, and satisfaction with services.15 Furthermore, there are often health outcomes for others, including informal caregivers and parents, as well as important external effects—for example, development of antimicrobial resistance. Use of a single outcome for cost effectiveness analysis fails to recognise that decision making involves making judgments about a variety of important effects rather than just one.
Presentation of economic evaluations is meaningful to decision makers
The third questionable assumption is that the complex technical presentation of results from economic evaluation is meaningful to the decision makers for whom it is intended. Studies have shown that decision makers find the concepts behind QALYs difficult to understand16 and that knowledge about formal methodology is limited.17 Recent advances in method place even greater burden on understanding than when these studies were done. The use of cost effectiveness acceptability curves is now advocated. These plot the relative probabilities of each intervention being the most cost effective option given different levels of maximum societal willingness to pay for the unit of outcome, usually QALYs. Decision makers must now identify their maximum willingness to pay for a QALY and then interpret the relative probabilities that different alternatives are the most cost effective option at that particular maximum. When many decision makers do not fully understand the basis for QALYs, expecting them to identify their maximum willingness to pay for additional QALYs on behalf of society seems nonsensical.
An alternative route
Two potential dangers arise from following the narrow philosophy with which recent advances are associated. The first is that economic evaluation becomes margin-alised and the anxiety among health economists about its lack of influence continues to grow. The second, more worrying, is that this limited approach is followed by those who do not fully understand its basis and thus decisions are taken which neither reflect society's objectives nor its health beliefs. Discussion among economists about the NICE threshold for cost per QALY gained is a disquieting example of this phenomenon.18
An alternative is to restrict all economic evaluations to the approach of cost-consequences. Different options are contrasted clearly and explicitly in tabular form for all the relevant costs (resource use) and consequences (for a recent example see Jacklin et al19). This approach allows decision makers (on behalf of society) to impute their own values to these costs and consequences, which could differ according to local context. Decision makers can see clearly what is included and what is omitted, where information is quantitative and where qualitative. Information about implications for equity, need, and other relevant objectives can be presented as well as information about the health effect on others such as informal caregivers.
To illustrate the outputs from, and uses of, the two approaches let's examine an hypothetical analysis for a comparison of hospital at home and hospital care. The outcome is based on QALYs formed from a five dimension quality of life scale. The cost of treating the patient in the hospital at home was £1200 more than in hospital and made little difference to the mean number of QALYs gained (0.02), where QALYs combine information about mortality and quality of life. This results in an incremental cost per QALY gained for hospital at home of £60 000 ($107 000, €87 000). Using the cost effectiveness acceptability curve (figure), decision makers would estimate their maximum willingness to pay for a QALY. If, for example, this was £30 000, the probability that hospital at home is more cost effective is slightly less than 20%, but if their willingness to pay for an additional QALY was as high as £80 000, the probability that hospital at home is the more cost effective option would be more than 80%.
Cost effectiveness analysis is based on achieving an assumed societal objective of maximising health
Little evidence exists that this is the desired objective of the public or decision makers
Use of QALYs as a single outcome measure for economic evaluation means that important health consequences are excluded
Complex technical presentation makes the findings difficult to understand and use
Cost-consequences analysis would better approach the objectives of decision makers and be easier to understand
The table shows a simplified version of the cost-consequences analysis. A full table might additionally include: anxiety and depression, pain control, carer quality of life, costs to social services, accessibility to the service. Decision makers would use the information provided in the table to make decisions or, if desired, they could also use monetary valuation or discrete choice experiments to obtain utility values for the different elements. The relative efficiency of different options would depend on the implicit or explicit values attached by decision makers to the different elements of cost and outcome.
A cost-consequences approach would more closely meet the needs of decision makers than current practice and avoid extensive use of inadequate assumptions. Such an approach may not earn researchers the same kudos for methodological research or technical capability as current methods, but it is closer to both Paretian welfare economics and a true decision making approach. It has the additional benefit of being easily understood and thus more likely to influence decision making in practice.
The Department of Social Medicine of the University of Bristol is the lead centre of the MRC Health Services Research Collaboration. I thank Richard Smith, Shah Ebrahim, Stephen Clarke, Terry Flynn, Ingolf Griebsch, Sandra Hollinghurst, and Oya Asim for helpful comments.
Contributors and sources JC has worked as an academic health economist for almost 15 years. She has conducted economic evaluations for various interventions, mainly in the area of service delivery and organisation and conducted qualitative research among the public and local decision makers.
Competing interests None declared.