BMJ  2003;326:1247-1250 (7 June), doi:10.1136/bmj.326.7401.1247

Primary care

Meta-analysis of data on costs from trials of counselling in primary care: using individual patient data to overcome sample size limitations in economic analyses

¹ National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL, ² Centre for the Economics of Mental Health, David Goldberg Centre, Institute of Psychiatry, London SE5 8AF, ³ Research and Development Directorate, University College London Hospitals NHS Trust, London NW1 2LT, ⁴ Centre for Occupational Psychology, University of Cardiff, Cardiff CF10 3YG, ⁵ Centre for Research in Primary and Community Care, Faculty of Health and Human Sciences, University of Hertfordshire, Hatfield AL10 9AB, ⁶ University of Greenwich, Bronte Hall, London SE9 2UG, ⁷ School of Medicine, Health Policy and Practice, University of East Anglia, Norwich NR4 7TJ, ⁸ Department of Psychiatry and Behavioural Sciences, Royal Free and University College Medical School, London NW3 2PF

Abstract

Objective To assess the feasibility of overcoming sample size limitations in economic analyses of clinical trials through meta-analysis of data on individual patients from multiple trials.

Design Meta-analysis of individual patient data from trials of counselling in primary care compared with usual care by a general practitioner.

Setting Primary care.

Patients People with mental health problems.

Main outcome measures Direct treatment costs, depressive symptoms, and cost effectiveness.

Results Meta-analysis of individual patient data proved feasible. The results showed that the previous analyses of individual trials were underpowered to provide useful conclusions about the cost comparisons. The results are sensitive to assumptions made about the costs of sessions with a counsellor and the management of patients by a general practitioner.

Conclusions Meta-analysis of individual patient data may assist in overcoming sample size limitations in economic analyses. Although feasible, such analysis has shortcomings that may limit the validity of the results. The relative costs and benefits of this method, as opposed to further collection of primary data, are as yet unclear.

Introduction

Economic evaluation is an increasingly important element of clinical trials. Several aspects of economic evaluations can make their addition to clinical trials problematic, but of particular importance is the sample size requirement.^{1 2} Trials are generally powered to detect clinically significant differences in outcomes, but since costs are often characterised by higher variability and skewness than clinical outcomes,³ economic analyses may be underpowered, resulting in type II errors—that is, failure to reject the null hypothesis that there is no difference in costs between treatments.

A recent example of this concerns counselling in primary care. Four trials reported no evidence of differences in the total costs of patients treated by counsellors compared with those who remained under the usual care of a general practitioner.^4–7 None of these trials were powered on costs, however, so that firm conclusions about costs cannot be drawn. An adequately powered trial could be conducted but would be long, large, and costly.

An alternative solution is to pool existing data in a meta-analysis. Analyses of data on individual patients are considered the gold standard for this technique,⁸ and in the present context could be used to increase the sample size for economic evaluation. We examined the feasibility of meta-analysis of economic data, using individual patient data on healthcare utilisation from trials of counselling in primary care.

Methods

Studies were identified from a Cochrane systematic review. Although seven trials met the Cochrane eligibility criteria, three were excluded after examination. See bmj.com for details.

Pooling data using meta-analysis requires studies to be broadly comparable, but the degree of legitimate difference is contentious.⁹ Details of the studies are available on bmj.com.

Identification of comparable data
Costs
The perspective taken for the analysis was that of the healthcare system, and only direct healthcare costs were analysed. Available data on the utilisation of health care varied, but all trials included consultations with a general practitioner, prescriptions for psychotropics, and mental health referrals, which are the sources of cost most likely to be influenced by the provision of counselling.

Consultations—Not all trials collected data on all types of consultations in primary care, so consultations had to be limited to surgery attendances only.

Drugs—Two studies reported exact dosage and duration of use of all drugs, which allowed calculation of number of tablets prescribed and a cost per tablet to be applied. The remaining two trials recorded only whether a prescription for relevant psychotropics had been given. Standard duration of use and dosage for tricyclics, selective serotonin reuptake inhibitors, and anxiolytics were therefore imputed.

Mental health referrals—Data on mental health referrals also varied and were recoded as the number of attendances at four categories of services: inpatient psychiatry, outpatient psychiatry, practice based psychological therapy (including protocol therapy provided as part of the trial), and other providers of psychological therapy in community and voluntary groups.

Data collection periods—Where required, costs were adjusted to represent standard periods of six months, allowing comparative analysis in the short term (six months after baseline) for all four trials and in the long term (12 months after baseline) for three trials. These adjustments assumed that service use remained constant over time.

Effectiveness
In three studies the Beck depression inventory was the primary outcome measure.^{4 5 7} As with the cost data, clinical outcome data were collected at different periods (three, four, and six months for short term data and nine and 12 months for long term data). However, no attempts were made to adjust these data to standard periods of six months. The results may thus be subject to bias if differential outcomes between treatments change over time.

Calculation of costs and data analysis
We applied standardised national unit costs for the financial year 1999-2000 to the data on utilisation of healthcare services. Not all trials reported long term data or usable effectiveness outcomes, and there were missing data in all studies. Therefore, the data available varied for analyses of costs alone, effectiveness alone, and cost effectiveness.

Data on costs and outcomes were pooled separately in a fixed effects meta-analysis.¹⁰ Analyses were conducted separately for the total direct costs and the total primary care costs.

Meta-analysis assumes that the overall treatment effect is normally distributed. Costs were not normally distributed, but the validity of the results was confirmed using non-parametric bootstrapping.¹¹

Relative cost effectiveness was described using incremental cost effectiveness ratios, the ratio of differential average costs of the two interventions to the differential average effects.^{12 13} Cost effectiveness acceptability curves were also calculated for both the short term and the long term. See bmj.com for details.

Results

Cost outcomes
The main analysis of costs over the long term indicated that counselling was associated with significantly greater total direct costs per patient (weighted difference in means £110 ($177; €154), 95% confidence interval £38 to £182; fig 1) and primary care costs per patient (£146, £110 to £183) than usual care by a general practitioner.

View larger version (24K): [in this window] [in a new window]   Fig 1 Total direct costs (£) over the long term

 

In the short term, counselling was again associated with significantly greater total direct costs per patient (£92, £57 to £126) and primary care costs per patient (£135, £114 to £156) than usual care by a general practitioner. Some evidence was found of heterogeneity in the analysis of primary care costs (²=7.2, df=3, P=0.07). Interpretation is not substantially different if based on the results of random effects analysis (£133, £99 to £167).

Sensitivity analysis
Examination of the cost components indicated that the costs of consultations with a general practitioner and the costs of protocol therapy were the main drivers of direct costs. Therefore, post hoc sensitivity analyses were conducted, systematically changing these variables.

Duration of general practitioner consultations—It has been suggested that referral to a counsellor may reduce the time general practitioners spend consulting with patients, as well as the overall rate of consulting.¹⁴ Therefore the effect of selectively increasing the duration of consultations in the group receiving usual care was tested in a threshold analysis. The significant differences found in total direct costs in the long term became non-significant (£72, - £1 to £145) when the duration of a consultation for the control group was increased to 11.75 minutes, equivalent to a 26% increase in the cost of a consultation.

Cost of protocol therapy—The cost of protocol therapy was based on the average of three different grades, ranging from standard counsellors to those in more senior management positions. When costs for counselling sessions were reduced to those of the least expensive counsellor, the differences between the groups in total direct costs in the long term became non-significant (£69, - £1 to £139).

Effectiveness outcomes
Counselling produced superior scores on the Beck depression inventory in the short term (1.93, 0.14 to 3.71) but not in the long term (1.16, - 0.65 to 2.97). Some evidence was found of heterogeneity in the short term data (²=7.3, df=2, P=0.03). The results using random effects analyses were similar.

Cost effectiveness analysis
The analysis of cost effectiveness was based on patients with data available on both cost and effectiveness. The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner over the long term was £196 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £110, incremental mean effect 0.56). Figure 2 illustrates the uncertainty associated with the costs and effects of the two treatments in the long term, and shows that for willingness to pay values above £196, counselling has a greater than 50% probability of being cost effective compared with usual care by a general practitioner. The probability of counselling being more cost effective than usual care stabilises at about 69% for willingness to pay ratios greater than £2000.

View larger version (22K): [in this window] [in a new window]   Fig 2 Cost effectiveness acceptability curve long term

 

The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner in the short term was £50 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £109, incremental mean effect 2.16).

Discussion

Meta-analysis of individual patient data can be used to overcome the sample size limitations often associated with economic analyses. The analyses may, however, be limited in several ways. The need to identify comparable data and apply standardised costs means that detail in individual studies may be lost. We found significant variation between trials in standard deviations of costs. Such differences have important implications for standard meta-analysis approaches, where estimates are weighted inversely by their assumed precision. Cost sources omitted or imputed in the current analysis would have also increased variance further if they had been measured. Adjusting costs to represent common time periods assumes service use is linear over time, when it may be that service use (such as consultation rates and drug use) reduces when an effective therapy is provided.

Such methodological compromises mean that the present analysis can never approach the precision of primary data collection. However, it is a matter of debate (and economics) whether this additional precision is worth the extra cost and time delay required to fund an adequately powered economic analysis, or whether secondary analysis provides a reasonably accurate estimate to inform policy and practice.

Future research should attempt to ensure greater comparability in methodology in studies examining common cost effectiveness questions. The prospective registration of trials might facilitate this.

Several statistical issues also arise in meta-analyses of cost data. Specific features of economic outcomes (for example, skewness in the distribution of cost data)

What is already known on this topic Issues of cost effectiveness are an increasingly important part of clinical trials Trials powered on clinical outcomes may be underpowered to detect important economic effects because of variability associated with data on costs What this study adds It was feasible to pool cost data from several trials to overcome the sample size limitations of previous studies on counselling in primary care Previous studies may have been unable to detect differences in costs between counselling and usual care by a general practitioner Results depended on assumptions made about costs of counsellor sessions and management of patients by a general practitioner mean that further investigation into appropriate statistical methodology is needed.

Costs are affected particularly by missing data.¹⁵ Analysis of individual patient data could allow imputation of missing values through several methods. Data were also lost for the cost effectiveness analysis owing to use of different effectiveness measures in trials. In cases where outcomes are conceptually the same but measured on different scales, the meta-analysis approach of standardised differences might be used, allowing all trials to be included in the analysis.

Cost effectiveness of counselling
Caution should be exercised in drawing specific conclusions about the cost effectiveness of counselling, given that the validity of the current methodology is unclear. However, the main results indicate that the costs associated with counselling were higher than those with usual care by a general practitioner, which supports the argument that previous analyses were underpowered to detect these effects.² The interpretation of figure 2 is complex, because typical ceiling ratios for a one point change in scores on the Beck depression inventory are not known. Such ratios are often assigned to quality adjusted life years (QALYs), but methods for translating Beck depression inventory scores into QALYs are crude at present.¹⁶

Given these limitations, the main analysis of differences in costs alone may be easier to interpret. These differences were found to be sensitive to increases in the duration of consultations with a general practitioner with patients not referred for counselling. However, no trials reported objective measures of duration of consultation, and thus this effect is speculative.

Results were also sensitive to the costs of counselling sessions. Tension exists between the desire to provide high quality counselling services by employing experienced (and thus more expensive) counsellors and the need to ensure that session costs do not jeopardise cost effectiveness.

---

This is an abridged version; the full version is on bmj.com

Characteristics of included patients appear on bmj.com

We thank Brendan Delaney, the referee, for identifying an important technical error in the first submission of the paper.

Contributors: See bmj.com

Funding: PB is funded by the National Primary Care Research and Development Centre at the University of Manchester, through the Department of Health.

Competing interests: None declared.

References

1. Lave J, Schulberg H. Integrating cost-effectiveness analyses within clinical trials of treatment for major depression in primary-care practice. In: Miller N, Magruder K, eds. Cost-effectiveness of psychotherapy: a guide for practitioners, researchers and policy makers. New York: Oxford University Press, 1999; 75-84.
2. Briggs A. Economic evaluation and clinical trials: size matters. BMJ 2000;321: 1362-3.[Free Full Text]
3. Johnston K, Buxton M, Jones D, Fitzpatrick R. Assessing the costs of healthcare technologies in clinical trials. Health Technol Assess 1999;3(6).
4. King M, Sibbald B, Ward E, Bower P, Lloyd M, Gabbay M, et al. Randomised controlled trial of non-directive counselling, cognitive-behaviour therapy and usual general practitioner care in the management of depression as well as mixed anxiety and depression in primary care. Health Technol Assess 2000;4(19).
5. Friedli K, King M, Lloyd M. The economics of employing a counsellor in general practice: analysis of data from a randomised controlled trial. Br J Gen Pract 2000;50: 276-83.[Web of Science][Medline]
6. Harvey I, Nelson S, Lyons R, Unwin C, Monaghan S, Peters T. A randomized controlled trial and economic evaluation of counselling in primary care. Br J Gen Pract 1998;48: 1043-8.[Web of Science][Medline]
7. Simpson S, Corney R, Fitzgerald P, Beecham J. A randomised controlled trial to evaluate the effectiveness and cost-effectiveness of counselling patients with chronic depression. Health Technol Assess 2000;4(36).
8. Stewart L, Clarke M, Cochrane Working Group on meta-analysis using individual patient data. Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat Med 1995;14: 2057-79.[Web of Science][Medline]
9. Gotzsche P. Why we need a broad perspective on meta-analysis. BMJ 2000;321: 585-6.[Free Full Text]
10. Sutton A, Abrams K, Jones D, Sheldon T, Song F. Systematic reviews of trials and other studies. Health Technol Assess 1998;2(19).
11. Efron B, Tibshirani R. An introduction to the bootstrap. New York: Chapman and Hall, 1993.
12. Van Hout B, Al M, Gordon G, Rutten F. Costs, effects, and C/E ratios alongside a clinical trial. Health Econ 1994;3: 309-13.[Web of Science][Medline]
13. Briggs A. Handling uncertainty in economic evaluation and presenting the results. In: Drummond M, McGuire A, eds. Economic evaluation in health care: merging theory and practice. London: Gaskell, 2001; 172-214.
14. Speirs R, Jewell J. One counsellor, two practices: report of a pilot scheme in Cambridgeshire. Br J Gen Pract 1995;45: 31-3.[Web of Science][Medline]
15. Briggs A, Clark T, Wolstenholme J, Clarke P. Missing... presumed at random: cost-analysis of incomplete data. Health Econ 2003;12: 377-92.[CrossRef][Web of Science][Medline]
16. Kaltenthaler E, Shackley P, Stevens P, Beverley C, Parry G, Chilcott J. A systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety. Health Technol Assess 2002;6(22).

(Accepted May 1, 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Article

Meta-analysis can effectively pool cost data for individual patients
BMJ 2003 326: 0. [Full Text]

This article has been cited by other articles:

• Groot Koerkamp, B., Hunink, M.G. M., Stijnen, T., Hammitt, J. K., Kuntz, K. M., Weinstein, M. C. (2007). Limitations of Acceptability Curves for Presenting Uncertainty in Cost-Effectiveness Analysis. Med Decis Making 27: 101-111 [Abstract]  
• GILBODY, S., BOWER, P., WHITTY, P. (2006). Costs and consequences of enhanced primary care for depression: Systematic review of randomised economic evaluations. Br. J. Psychiatry 189: 297-308 [Abstract] [Full text]  
• Araya, R., Flynn, T., Rojas, G., Fritsch, R., Simon, G. (2006). Cost-Effectiveness of a Primary Care Treatment Program for Depression in Low-Income Women in Santiago, Chile. Am. J. Psychiatry 163: 1379-1387 [Abstract] [Full text]  
• FENWICK, E., BYFORD, S. (2005). A guide to cost-effectiveness acceptability curves. Br. J. Psychiatry 187: 106-108 [Abstract] [Full text]  
• Price, J. R (2003). Counselling may be more expensive than usual care for people with mental health problems, but may not be more effective for improving Beck depression score in the long term. Evid. Based Ment. Health 6: 127-127 [Full text]  
• (2003). Mild depression in general practice: time for a rethink?. DTB 41: 60-64 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Validity of post-hoc meta-analysis of secondary cost data from medical research studies

M Jawad Hashim
bmj.com, 13 Jun 2003 [Full text]

Online poll

Find out more >>