How to read a paper: Papers that tell you what things cost (economic analyses)
BMJ 1997; 315 doi: https://doi.org/10.1136/bmj.315.7108.596 (Published 06 September 1997) Cite this as: BMJ 1997;315:596
- Trisha Greenhalgh, senior lecturer (p.greenhalgh{at}ucl.ac.uk)a
- a Unit for Evidence-Based Practice and Policy, Department of Primary Care and Population Sciences, University College London Medical School/Royal Free Hospital School of Medicine, Whittington Hospital, London N19 5NF
What is economic analysis?
An economic analysis can be defined as an analysis that uses analytical techniques to define choices in resource allocation. This article is based largely on a short booklet by Professor Michael Drummond1 and two of the forerunners to the “Users' Guides to the Medical Literature” series.2 3 A recent book, Elementary Economic Evaluation in Health Care, is also useful.4
Measuring costs and benefits of health interventions
Not long ago, I was taken to hospital to have my appendix removed. From the hospital's point of view, the cost of my care included my board and lodging for five days, a proportion of doctors' and nurses' time, drugs and dressings, and investigations (blood tests and a scan). Other direct costs (see box) included my general practitioner's time for attending me in the middle of the night and the cost of the petrol my husband used when visiting me (not to mention the grapes and flowers).
Examples of costs and benefits of health interventions
In addition to this, there were the indirect costs of my loss in productivity. I was off work for three weeks, and my domestic duties were temporarily carried out by various friends, neighbours, and a hired nanny. Also, from my point of view, there were several intangible costs, such as discomfort, loss of independence, and a cosmetically unsightly scar. As the box shows, these direct, indirect, and intangible costs constitute one side of the cost-benefit equation. On the benefit side, the operation greatly increased my chances of staying alive and I had a nice rest from work.
In this example, few patients (and even fewer purchasers) would perceive much freedom of choice in deciding to opt for the operation. But most health interventions do not concern definitive procedures for surgical emergencies. At some stage, almost all of us will be forced to decide whether having a routine operation, taking a particular drug, or compromising our lifestyle to treat a chronic but not immediately life threatening condition is “worth it.”
It is fine for informed individuals to make choices about their own care by gut reaction (“I'd rather live with my hernia than be cut open,” or “I know about the risk of thrombosis but I want to continue to smoke and stay on the pill”). But when the choices are about other people's care, subjective judgments are the last thing that should enter the equation. Most of us would want the planners and policymakers to use objective, explicit, and defensible criteria when making decisions such as “No, this patient may not have a kidney transplant.”
One important way of addressing the “what's it worth?” question for a given health state (such as having poorly controlled diabetes or a flare up of rheumatoid arthritis) is to ask someone in that state how they feel. A number of questionnaires have been developed which attempt to measure overall health status, such as the Nottingham health profile, the SF-36 general health questionnaire, and the McMaster health utilities index questionnaire.5
Summary points
An economic analysis should be based on a primary study or meta-analysis that is scientifically valid, reliable, and relevant
When deciding whether an economic analysis has been done correctly, you should not simply check the arithmetic but consider whether all direct, indirect, and intangible costs and benefits have been included
In the allocation of limited resources, the comparison of different health states is unavoidable, but instruments for measuring health related quality of life are not as objective as they seem
In some circumstances, disease specific measures of wellbeing are more valid than general measures.6 For example, answering “yes” to the question, “Do you get very concerned about the food you are eating?” might indicate anxiety in someone without diabetes but normal self care attitudes in someone with diabetes. There has also been an upsurge of interest in patient specific measures of quality of life, to allow different patients to place different values on particular aspects of their health and wellbeing.7 Of course, when quality of life is being analysed from the point of view of the patient, this is a sensible and humane approach. However, the health economist tends to make decisions about groups of patients or populations, in which case patient specific, and even disease specific, measures of quality of life have limited relevance.8
The authors of standard instruments (such as the SF-36) for measuring quality of life have often spent years ensuring they are valid (that they measure what we think they are measuring), reliable (they do so every time), and responsive to change (if an intervention improves or worsens the patient's health, the scale will reflect that). For this reason, you should be highly suspicious of a paper that abandons these standard instruments in favour of the authors' own rough and ready scale (for example, “functional ability was classified as good, moderate, or poor according to the clinician's overall impression”). Note also that even instruments which have apparently been well validated often do not stand up to rigorous evaluation of their psychometric validity.9
Another way of addressing the “what's it worth?” of particular health states is through health state preference values—that is, the value which, in a hypothetical situation, a healthy person would place on a particular deterioration in their health, or which a sick person would place on a return to health. There are three main methods of assigning such values:
Rating scale measurements—the respondent is asked to make a mark on a fixed line, labelled, for example, “perfect health” at one end and “death” at the other, to indicate where he or she would place the state in question (for example, being confined to a wheelchair by arthritis of the hip);
Time tradeoff measurements—the respondent is asked to consider a particular health state (for example, infertility) and estimate how many of their remaining years in full health they would sacrifice to be “cured” of the condition;
Standard gamble measurements—the respondent is asked to consider the choice between living for the rest of their life in a particular health state and taking a “gamble” (such as having an operation) with a given odds of success which would return them to full health if it succeeded but kill them if it failed. The odds are then varied to see at what point the respondent decides the gamble is not worth taking.10
The quality adjusted life year (QALY) can be calculated by multiplying the preference value for that state with the time the patient is likely to spend in that state. The results of cost-benefit analyses are usually expressed in terms of “cost per QALY,” some examples of which are shown in the second box.11
Results of cost-benefit analysis for some medical procedures
The use of QALYs is controversial. Any measure of health state preference values is, at best, a reflection of the preferences and prejudices of the individuals who contributed to its development. Indeed, it is possible to come up with different values for QALYs, depending on how the questions from which the health state preference values are derived were posed.12 Furthermore, it is virtually impossible to combine different QALYs to measure the effect of more than one serious or disabling condition on a patient.13 As medical ethicist John Harris has pointed out, QALYs are, like the society that produces them, inherently agist, sexist, racist, and loaded against those with permanent disabilities (since even a complete cure of an unrelated condition would not restore the individual to “perfect health”). Furthermore, QALYs distort our ethical instincts by focusing our minds on years of life rather than people's lives. A disabled premature infant in need of an intensive care cot will, argues Harris, be allocated more resources than it deserves in comparison with a 50 year old woman with cancer, since the infant, were it to survive, would have so many more life years to quality adjust.14
Other authors have come up with the HYE (healthy years equivalent) measure, which incorporates the individual's likely improvement or deterioration in health status in the future and is said to avoid some, but not all, of the disadvantages of the QALY.15 Given that the critics of QALYs and HYEs have offered no alternative, all encompassing measure of health status, these utility based units are set to remain in the health economist's toolkit for the forseeable future. For a more detailed discussion of these issues by a multidisciplinary panel, see Anthony Hopkins's booklet Measures of the Quality of Life.16
There is, however, another form of analysis which, although it does not abolish the need to place arbitrary numerical values on life and limb, avoids the buck stopping at the unfortunate health economist. This approach, known as cost-consequences analysis, presents the results of the economic analysis in a disaggregated form. In other words, it expresses different outcomes in terms of their different natural units (something real such as months of survival, legs amputated, or babies taken home), so that individuals can assign their own values to particular health states before calculating whether the intervention is “worth it.”
Ten questions to ask about an economic analysis
The checklist which follows is based on the sources mentioned earlier,1 2 as well as suggestions made by a working party set up by the BMJ to produce guidelines for journal editors on appraising economic evaluations (M Drummond, personal communication).
Question 1: Is the analysis based on a study that answers a clearly defined clinical question about an economically important issue?
Before pursuing any of the economic arguments, make sure that the trial being analysed is scientifically relevant and capable of giving unbiased and unambiguous answers to the clinical question posed in its introduction.
Question 2: Whose viewpoint are costs and benefits being considered from?
From the Treasury's point of view, the most cost effective health intervention is one which returns all citizens promptly to taxpayer status and, when this status is no longer tenable, causes immediate sudden death. From the drug company's point of view, it would be difficult to imagine a cost-benefit equation which did not contain one of the company's products, and from a physiotherapist's point of view, the removal of a physiotherapy service would never be cost effective. Almost all economic analyses have some funding, and all have been inspired by someone with a vested interest; the paper should say which.
Question 3: Have the interventions being compared been shown to be clinically effective?
In general, the intervention that “works out cheaper” should not be substantially less effective in clinical terms than the one which stands to be rejected on the grounds of cost.
Question 4: Are the interventions sensible and workable in the settings where they are likely to be applied?
Too many research trials look at intervention packages which would be impossible to implement in the non-research setting (they assume, for example, that general practitioners will own a state of the art computer and agree to follow a protocol, that infinite nurse time is available for the taking of blood tests, or that patients will make their personal treatment choices solely on the basis of the trial's conclusions). Remember that standard current practice, which may be to do nothing, should almost certainly be one of the alternatives compared.
Question 5: Which method of analysis was used, and was this appropriate?
This decision can be summarised as follows:
Cost minimisation analysis would be most appropriate if the interventions produced identical outcomes;
Cost effectiveness analysis would be most appropriate if the important outcome is unidimensional;
Cost utility analysis would be most appropriate if the important outcome is multidimensional;
Cost benefit analysis would be most appropriate if the cost benefit equation for this condition needs to be compared with cost benefit equations for different conditions;
Cost consequences analysis would be most appropriate if a cost benefit analysis would otherwise be appropriate but the preference values given to different health states are disputed or likely to change.
Question 6: How were costs and benefits measured?
Consider an economic evaluation of a trial comparing the rehabilitation of stroke patients into their own homes, including attendance at a day centre, with a standard alternative intervention (rehabilitation in a long stay hospital). The economic analysis must take into account not just the time of the various professionals involved, the time of the secretaries and administrators who help run the service, “overheads” (such as heating and lighting), and the cost of the food and drugs consumed by the stroke patients, but also a fraction of the capital cost of building the day centre and maintaining a transport service to and from it.
In a cost effectiveness analysis, changes in health status will be expressed in natural units. But just because the units are natural does not automatically make them appropriate. For example, the economic analysis of the treatment of peptic ulcer by two different drugs might measure outcome as “proportion of ulcers healed after a six week course.” Treatments could be compared according to the cost per ulcer healed. However, if the relapse rates on the two drugs were very different, drug A might be falsely deemed “more cost effective” than drug B. A better outcome measure here might be “ulcers that remained healed at one year.”
Question 7: Were incremental, rather than absolute, benefits considered?
This question is best illustrated by a simple example. Let's say drug X, at £100 per course, cures 10 out of every 20 patients. Its new competitor, drug Y, costs £120 per course and cures 11 out of 20 patients. The cost per case cured with drug X is £200 (since you spent £2000 curing 10 people), and the cost per case cured with drug Y is £218 (since you spent £2400 curing 11 people).
The incremental cost of drug Y—the extra cost of curing the extra patient—is not £18, but £400, since this is the total amount extra that you have had to pay to achieve an outcome over and above what you would have achieved by giving all patients the cheaper drug. This striking example should be borne in mind the next time a pharmaceutical representative tries to persuade you that his or her product is “more effective and only marginally more expensive.”
Question 8: Was the “here and now” given precedence over the distant future?
A bird in the hand is worth two in the bush: in health as well as money terms, we value a benefit today more highly than we value a promise of the same benefit in five years' time. When the costs or benefits of an intervention (or lack of the intervention) will occur some time in the future, their value should be discounted to reflect this. The actual amount of discount that should be allowed for future, as opposed to immediate, health benefit is fairly arbitrary, but most analyses use a figure of around 5% per year.
Question 9: Was a sensitivity analysis performed?
Let's say a cost-benefit analysis comes out as saying that hernia repair by day case surgery costs £1150 per QALY whereas traditional open repair, with its associated hospital stay, costs £1800 per QALY. But, when you look at how the calculations were done, you are surprised at how cheaply the laparoscopic equipment has been costed. If you raise the price of this equipment by 25%, does day case surgery still come out dramatically cheaper? It may, or it may not.
Sensitivity analysis, or exploration of “what ifs,” was described earlier in this series in relation to meta-analysis.17 Exactly the same principles apply here: if adjusting the figures to account for the full range of possible influences gives you a totally different answer, you should not place too much reliance on the analysis. For a good example of a sensitivity analysis on a topic of both scientific and political importance, see Pharoah and Hollingworth's paper on the cost effectiveness of lowering cholesterol (which addresses the difficult issue of who should receive, and who should be denied, effective but expensive drugs to lower cholesterol).18
Question 10: Were “bottom line” aggregate scores overused?
The notion of cost-consequences analysis, in which the reader of the paper can attach his or her own values to different utilities, was introduced earlier. In practice, this is an unusual way of presenting an economic analysis, and, more commonly, the reader is faced with a cost-utility or cost-benefit analysis which gives a composite score in unfamiliar units which do not translate readily into exactly what gains and losses the patient can expect. The situation is analogous to the father who is told “your child's IQ is 115” when he would feel far better informed if he were presented with the disaggregated data: “Johnny can read, write, count, and draw pretty well for his age.”
The articles in this series are excerpts from How to read a paper: the basics of evidence based medicine. The book includes chapters on searching the literature and implementing evidence based findings. It can be ordered from the BMJ Publishing Group: tel 0171 383 6185/6245; fax 0171 383 6662. Price £13.95 UK members, £14.95 non-members.
Acknowledgments
Thanks to Professor Mike Drummond and Dr Alison Tonks for advice on this chapter.