BMJ  2003;326:1247-1250 (7 June), doi:10.1136/bmj.326.7401.1247

Primary care

Meta-analysis of data on costs from trials of counselling in primary care: using individual patient data to overcome sample size limitations in economic analyses

¹ National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL, ² Centre for the Economics of Mental Health, David Goldberg Centre, Institute of Psychiatry, London SE5 8AF, ³ Research and Development Directorate, University College London Hospitals NHS Trust, London NW1 2LT, ⁴ Centre for Occupational Psychology, University of Cardiff, Cardiff CF10 3YG, ⁵ Centre for Research in Primary and Community Care, Faculty of Health and Human Sciences, University of Hertfordshire, Hatfield AL10 9AB, ⁶ University of Greenwich, Bronte Hall, London SE9 2UG, ⁷ School of Medicine, Health Policy and Practice, University of East Anglia, Norwich NR4 7TJ, ⁸ Department of Psychiatry and Behavioural Sciences, Royal Free and University College Medical School, London NW3 2PF

Abstract

Objective To assess the feasibility of overcoming sample size limitations in economic analyses of clinical trials through meta-analysis of data on individual patients from multiple trials.

Design Meta-analysis of individual patient data from trials of counselling in primary care compared with usual care by a general practitioner.

Setting Primary care.

Patients People with mental health problems.

Main outcome measures Direct treatment costs, depressive symptoms, and cost effectiveness.

Results Meta-analysis of individual patient data proved feasible. The results showed that the previous analyses of individual trials were underpowered to provide useful conclusions about the cost comparisons. The results are sensitive to assumptions made about the costs of sessions with a counsellor and the management of patients by a general practitioner.

Conclusions Meta-analysis of individual patient data may assist in overcoming sample size limitations in economic analyses. Although feasible, such analysis has shortcomings that may limit the validity of the results. The relative costs and benefits of this method, as opposed to further collection of primary data, are as yet unclear.

Introduction

Economic evaluation is an increasingly important element of clinical trials. Several aspects of economic evaluations can make their addition to clinical trials problematic, but of particular importance is the sample size requirement.^{1 2} Trials are generally powered to detect clinically significant differences in outcomes, but since costs are often characterised by higher variability and skewness than clinical outcomes,³ economic analyses may be underpowered, resulting in type II errors— that is, failure to reject the null hypothesis that there is no difference in costs between treatments.

A recent example of this concerns counselling in primary care. Four trials reported no evidence of differences in the total costs of patients treated by counsellors compared with those who remained under the usual care of a general practitioner.^4–7 None of these trials were powered on costs, however, so that firm conclusions about costs cannot be drawn. An adequately powered trial could be conducted, but this would need to be large, would be costly, and would delay access to relevant information for several years.

An alternative solution is to pool existing data in a meta-analysis. Analyses of data on individual patients are considered the gold standard for this technique and have several advantages, such as ensuring quality of randomisation and intention to treat analyses,⁸ but in the present context the primary advantage would be in increasing the sample size for economic evaluation and overcoming the limitations of previous analyses. We examined the feasibility of meta-analysis of economic data, using individual patient data on healthcare utilisation from trials of counselling in primary care.

Methods

Studies were identified from a Cochrane systematic review. Seven trials met the Cochrane eligibility criteria.⁹ Three studies were excluded after examination; two reported data on healthcare utilisation, but neither provided adequate detail.^10–12 The randomisation procedures of these studies were of poor quality and likely to be associated with bias.⁹ The third trial is yet to report economic data but also used a comparison group of antidepressant treatment prescribed by a general practitioner rather than usual care by a general practitioner.¹² This trial was also excluded, given that management by a general practitioner (and associated costs) were likely to be significantly different. Table 1 lists the included studies.^4–7

View this table: [in this window] [in a new window]   Table 1 Studies of counselling in primary care included in review

 

Pooling data using meta-analysis requires studies to be broadly comparable, but the degree of legitimate difference is contentious.¹³ All four studies used counsellors meeting the criteria for accreditation by the British Association for Counselling and Psychotherapy, although the exact models of counselling differed (for example, non-directive, psychodynamic). In three studies, general practitioners recruited patients, whereas in the fourth patients were screened, and the trial was restricted to patients with chronic problems of six months or more. The time of follow up varied from three to 12 months.

Identification of comparable data
Costs
The perspective taken for the analysis was that of the healthcare system, and only direct healthcare costs were analysed. Available data on the utilisation of health care varied, but all trials included consultations with a general practitioner, prescriptions for psychotropics, and mental health referrals, which are the sources of cost most likely to be influenced by the provision of counselling (see table 1). Patient specific data were collected from medical records and patient self report.

Consultations—Not all trials collected data on all types of consultations in primary care, so consultations had to be limited to surgery attendances only, whereas other types of consultations, such as home visits or appointments with a practice nurse, were excluded.

Drugs—Recording of data on drugs varied. Two studies reported exact dosage and duration of use of all drugs, which allowed calculation of number of tablets prescribed and a cost per tablet to be applied. The remaining two trials recorded only whether a prescription for relevant psychotropics had been given. Standard duration of use and dosage for tricyclics, selective serotonin reuptake inhibitors, and anxiolytics were therefore imputed. Data on other drugs were excluded.

Mental health referrals—Data on mental health referrals also varied and were recoded as the number of attendances at four categories of services: inpatient psychiatry, outpatient psychiatry (including community mental health teams and psychotherapy), practice based psychological therapy (for example, psychologists and counsellors in primary care), and other providers of psychological therapy in community and voluntary groups, such as Relate and self help groups. Practice based psychological therapy (provided to patients in both groups as part of usual care by a general practitioner) was differentiated from protocol therapy (provided to the experimental group only). It was assumed that all protocol therapy sessions were completed in the first six months of the trial.

Data collection periods—The time periods covered by data collection varied (see table 1). Where required, costs were adjusted to represent standard periods of six months, allowing comparative analysis in the short term (six months after baseline) for all four trials and in the long term (12 months after baseline) for three trials. These adjustments assumed that service use remained constant over time and were applied to all variables apart from protocol therapy, since the length of such therapy in all trials was fixed.

Effectiveness In three studies the Beck depression inventory was the primary outcome measure.^{4 5 7 14} As with the cost data, clinical outcome data were collected at different periods (three, four, and six months for short term data and nine and 12 months for long term data). However, no attempts were made to adjust these data to standard periods of six months. The results may thus be subject to bias if differential outcomes between treatments change over time. Only one trial included a generic health status preference instrument.⁴

Calculation of costs
We applied standardised national unit costs for the financial year 1999-2000 to the data on utilisation of healthcare services (table 2). The costs of drugs were based on the British National Formulary, using the cheapest alternative available.¹⁵ Costs of protocol therapy and practice based counselling services were based on salary figures provided by a relevant national organisation (Counsellors and Psychotherapists in Primary Care). All relevant costs incurred by employers were included, plus capital and direct overheads (data not shown). All other costs were based on figures from the Personal Social Services Research Unit.¹⁶ Discounting was unnecessary as costs were all within the same time period and were not recorded beyond 12 months.

View this table: [in this window] [in a new window]   Table 2 Unit costs used in analysis

 

Data analysis
Not all trials reported long term data or usable effectiveness outcomes, and there were missing data in all studies. Therefore, the data available varied for analyses of costs alone, effectiveness alone, and cost effectiveness. Missing data on costs in the long term were not associated with randomised group, sex, or employment status in any trial but were associated with baseline score on the Beck depression inventory, age, and marital status in one trial in each case. Missing data on effectiveness were imputed using the last observation carried forward. Figure 1 shows the data available for the analyses.

View larger version (27K): [in this window] [in a new window]   Fig 1 Data available for analyses

 

Data on costs and outcomes were pooled separately in a fixed effects meta-analysis.¹⁷ This combines estimates of treatment effects from the individual trials into an overall weighted average, with weights given by the inverse of the variance of the difference in means for each trial. A fixed effects analysis was preferred because of the small number of trials available; however, these results were compared with those from random effects models. A test for heterogeneity was also conducted, although this may lack power given the small number of trials available. Analyses were conducted separately for the total direct costs and the total primary care costs, to enable primary care commissioners to assess the economic impact on their sector alone. All analyses were carried out with Stata (Release 7).

Meta-analysis assumes that the overall treatment effect is normally distributed. Costs were not normally distributed, but the validity of the results was confirmed using non-parametric bootstrapping.¹⁸ Such an approach was preferred to transformation or non-parametric tests, as it allows inference about the population mean.¹⁹

Relative cost effectiveness was described using incremental cost effectiveness ratios, the ratio of differential average costs of the two interventions to the differential average effects.^{20 21} Cost effectiveness acceptability curves were also calculated for both the short term and the long term. These plot the probability that counselling is cost effective compared with usual care by a general practitioner for a range of possible maximum values (the ceiling ratio) a decision maker might be willing to pay for an improvement in the Beck depression inventory score. The curves are based on an estimate of the joint distribution of difference in mean cost and difference in mean Beck depression inventory score obtained using bootstrapping.^20–22 These curves incorporate the uncertainty that exists around the estimates of mean costs and effects as a result of sampling variation and uncertainty on the maximum cost effectiveness ratio that a decision maker would consider acceptable.

Results

Cost outcomes
The main analysis of costs over the long term indicated that counselling was associated with significantly greater total direct costs per patient (weighted difference in means £110 ($177; €154), 95% confidence interval £38 to £182; fig 2) and primary care costs per patient (£146, £110 to £183) than usual care by a general practitioner. No evidence was found of heterogeneity (total costs P=0.70; primary care costs P=0.83), and random effects analysis gave similar results.

View larger version (24K): [in this window] [in a new window]   Fig 2 Total direct costs (£) over the long term

 

In the short term, counselling was again associated with significantly greater total direct costs per patient (£92, £57 to £126; fig 3) and primary care costs per patient (£135, £114 to £156) than usual care by a general practitioner. Some evidence was found of heterogeneity in the analysis of primary care costs (²=7.2, df=3, P=0.07). Interpretation is not substantially different if based on the results of random effects analysis (£133, £99 to £167).

View larger version (27K): [in this window] [in a new window]   Fig 3 Total direct costs (£) over the short term

 

Sensitivity analysis
Examination of the cost components indicated that the costs of consultations with a general practitioner and costs of protocol therapy were the main drivers of direct costs. Therefore, post hoc sensitivity analyses were conducted, systematically changing these variables.

Duration of general practitioner consultations—It has been suggested that referral to a counsellor may reduce the time general practitioners spend consulting with patients, as well as the overall rate of consulting.²³ Therefore the effect of selectively increasing the duration of consultations in the group receiving usual care was tested in a threshold analysis. The significant differences found in total direct costs in the long term became non-significant (£72, -£1 to £145) when the duration of a consultation for the control group was increased to 11.75 minutes, equivalent to a 26% increase in the cost of a consultation.

Cost of protocol therapy—The cost of protocol therapy was based on the average of three different grades, ranging from standard counsellors to those in more senior management positions. It is likely that more counsellors in primary care are paid at the lower grade, in contrast to the counsellors employed within the trials included in our study. When costs for counselling sessions were reduced to those of the least expensive counsellor, the differences between the groups in total direct costs in the long term became non-significant (£69, -£1 to £139).

Effectiveness outcomes
Counselling produced superior scores on the Beck depression inventory in the short term (1.93, 0.14 to 3.71) but not in the long term (1.16, -0.65 to 2.97). Some evidence was found of heterogeneity in the short term data (²=7.3, df=2, P=0.03). The results using random effects analyses were similar.

Cost effectiveness analysis
The analysis of cost effectiveness was based on patients with data available on both cost and effectiveness. The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner over the long term was £196 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £110, incremental mean effect 0.56). Figure 4 illustrates the uncertainty associated with the costs and effects of the two treatments in the long term, and shows that for willingness to pay values above £196, counselling has a greater than 50% probability of being cost effective compared with usual care by a general practitioner. The probability of counselling being more cost effective than usual care stabilises at about 69% for willingness to pay ratios greater than £2000.

View larger version (22K): [in this window] [in a new window]   Fig 4 Cost effectiveness acceptability curve long term

 

The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner in the short term was £50 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £109, incremental mean effect 2.16).

Discussion

Meta-analysis of individual patient data can be used to overcome the sample size limitations often associated with economic analyses. The analyses may, however, be limited in several ways. The need to identify comparable data and apply standardised costs means that detail in individual studies may be lost. For example, non-psychiatric referral and drug costs were ignored, as were distinctions between particular services, such as between outpatient psychiatry and community mental health teams. The lack of data on costs of lost production meant that advantages associated with either treatment in returning people to work had been ignored. However, none of these issues may be as central to the effects of counselling (or as likely to show effects) as the measures of depressive symptoms and utilisation of mental health services.

We found significant variation between trials in standard deviations of costs. This may reflect more comprehensive estimates of service use in some trials, or differences between trials in the availability of certain services (for example, treatment for mental health outside the trial). Such differences have important implications for standard meta-analysis approaches, where estimates are weighted inversely by their assumed precision. Cost sources omitted or imputed in the current analysis would have also increased variance further if they had been measured. Adjusting costs to represent common time periods assumes service use is linear over time, when it may be that service use (such as consultation rates and drug use) reduces when an effective therapy is provided.

Such methodological compromises mean that the present analysis can never approach the precision of primary data collection. However, it is a matter of debate (and economics) whether this additional precision is worth the extra cost and time delay required to fund an adequately powered economic analysis, or whether secondary analysis provides a reasonably accurate estimate to inform policy and practice.

One compromise in future research is to attempt to ensure greater comparability in methodology in studies examining common cost effectiveness questions, through the use of the same outcome measures and the collection of data on service use on the same range and type of services. The prospective registration of trials might facilitate this. Researchers in the United States have gone further in explicitly preplanning meta-analyses of different interventions, deliberately standardising across studies where possible and identifying methods to deal with methodological heterogeneity.²⁴

Several statistical issues also arise in meta-analyses of cost data. Specific features of economic outcomes (for example, skewness in the distribution of cost data) mean that further investigation into appropriate statistical methodology is needed.

Costs are affected particularly by missing data.²⁵ Analysis of individual patient data allows imputation of missing values through several methods. Recent approaches that more accurately reflect the uncertainty associated with imputation of missing data have been described but were beyond the scope of our study.²⁵ Data were also lost for the cost effectiveness analysis owing to use of different effectiveness measures in trials. In cases where outcomes are conceptually the same but measured on different scales, the meta-analysis approach of standardised differences might be used, allowing all trials to be included in the analysis.

Cost effectiveness of counselling
Caution should be exercised in drawing specific conclusions about the cost effectiveness of counselling, given that the validity of the current methodology is unclear. However, the main results indicate that the costs associated with counselling were higher than those with usual care by a general practitioner, which supports the argument that previous analyses were underpowered to detect these effects.² The interpretation of figure 4 is complex, because typical ceiling ratios for a one point change in scores on the Beck depression inventory are not known. Such ratios are often assigned to quality adjusted life years (QALYs), but methods for translating Beck depression inventory scores into QALYs are crude at present.²⁶

What is already known on this topic Issues of cost effectiveness are an increasingly important part of clinical trials Trials powered on clinical outcomes may be underpowered to detect important economic effects because of variability associated with data on costs What this study adds It was feasible to pool cost data from several trials to overcome the sample size limitations of previous studies on counselling in primary care Previous studies may have been unable to detect differences in costs between counselling and usual care by a general practitioner Results depended on assumptions made about costs of counsellor sessions and management of patients by a general practitioner

Given these limitations, the main analysis of differences in costs alone may be easier to interpret. These differences were found to be sensitive to increases in the duration of consultations with a general practitioner in patients not referred for counselling. However, no trials reported objective measures of duration of consultation, and thus this effect is speculative.

Results were also sensitive to the costs of counselling sessions. Tension exists between the desire to provide high quality counselling services by employing experienced (and thus more expensive) counsellors and the need to ensure that session costs do not jeopardise cost effectiveness. This has implications for other psychological therapies in primary care, such as cognitive behavioural therapy, where session costs may be even higher.⁴ Our findings may provide an additional stimulus to the development of "minimal" interventions, which reduce or remove the need for the therapist, if effectiveness can be maintained.^{26 27}

---

Characteristics of included patients appear on bmj.com

We thank Brendan Delaney, the referee, for identifying an important technical error in the first submission of the paper.

Contributors: PB devised the study and wrote the paper and all authors commented on drafts; PB will act as guarantor for the paper. PB and SB wrote the protocol and performed the initial analysis, with additional economics advice from JBe. JBe, SS, RC, KF, MK, and IH provided data for the study and assisted in the development of the final database. JBa provided statistical advice and performed the bootstrapping and cost effectiveness acceptability curve analyses.

Funding: PB is funded by the National Primary Care Research and Development Centre at the University of Manchester, through the Department of Health.

Competing interests: None declared.

References

1. Lave J, Schulberg H. Integrating cost-effectiveness analyses within clinical trials of treatment for major depression in primary-care practice. In: Miller N, Magruder K, eds. Cost-effectiveness of psychotherapy: a guide for practitioners, researchers and policy makers. New York: Oxford University Press, 1999; 75-84.
2. Briggs A. Economic evaluation and clinical trials: size matters. BMJ 2000;321: 1362-3.[Free Full Text]
3. Johnston K, Buxton M, Jones D, Fitzpatrick R. Assessing the costs of healthcare technologies in clinical trials. Health Technol Assess 1999;3(6).
4. King M, Sibbald B, Ward E, Bower P, Lloyd M, Gabbay M, et al. Randomised controlled trial of non-directive counselling, cognitive-behaviour therapy and usual general practitioner care in the management of depression as well as mixed anxiety and depression in primary care. Health Technol Assess 2000;4(19).
5. Friedli K, King M, Lloyd M. The economics of employing a counsellor in general practice: analysis of data from a randomised controlled trial. Br J Gen Pract 2000;50: 276-83.[ISI][Medline]
6. Harvey I, Nelson S, Lyons R, Unwin C, Monaghan S, Peters T. A randomized controlled trial and economic evaluation of counselling in primary care. Br J Gen Pract 1998;48: 1043-8.[ISI][Medline]
7. Simpson S, Corney R, Fitzgerald P, Beecham J. A randomised controlled trial to evaluate the effectiveness and cost-effectiveness of counselling patients with chronic depression. Health Technol Assess 2000;4(36).
8. Stewart L, Clarke M, Cochrane Working Group on meta-analysis using individual patient data. Practical methodology of meta-analyses (overviews) using updated individual patient data. Stat Med 1995;14: 2057-79.[ISI][Medline]
9. Bower P, Rowland N, Mellor-Clark J, Heywood P, Godfrey C, Hardy R. Effectiveness and cost effectiveness of counselling in primary care. In: Cochrane Library. Oxford: Update Software, 2002.
10. Boot D, Gillies P, Fenelon J, Reubin R, Wilkins M, Gray P. Evaluation of the short-term impact of counseling in general practice. Pat Educ Couns 1994;24: 79-89.
11. Hemmings A. Counselling in primary care: a randomised controlled trial. Pat Educ Couns 1997;32: 219-30.
12. Chilvers C, Dewey M, Fielding K, Gretton V, Miller P, Palmer B, et al. Anti-depressant drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms. BMJ 2001;322: 772-5.[Abstract/Free Full Text]
13. Gotzsche P. Why we need a broad perspective on meta-analysis. BMJ 2000;321: 585-6.[Free Full Text]
14. Beck A, Steer R. Beck depression inventory: manual. San Antonio: Psychological Corporation, 1987.
15. British Medical Association, Royal Pharmaceutical Society of Great Britain. British national formulary. London: BMA and Pharmaceutical Press, 1998.
16. Netten A, Dennet J, Knight J. Unit costs of health and social care. University of Kent, Canterbury: Personal Social Services Research Unit, 1998.
17. Sutton A, Abrams K, Jones D, Sheldon T, Song F. Systematic reviews of trials and other studies. Health Technol Assess 1998;2(19).
18. Efron B, Tibshirani R. An introduction to the bootstrap. New York: Chapman and Hall, 1993.
19. Barber J, Thompson S. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ 1998;317: 1195-200.[Abstract/Free Full Text]
20. Van Hout B, Al M, Gordon G, Rutten F. Costs, effects, and C/E ratios alongside a clinical trial. Health Econ 1994;3: 309-13.[ISI][Medline]
21. Briggs A. Handling uncertainty in economic evaluation and presenting the results. In: Drummond M, McGuire A, eds. Economic evaluation in health care: merging theory and practice. London: Gaskell, 2001; 172-214.
22. Fenwick L, Claxton K, Schulpher M. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ 2002;10: 779-87.
23. Speirs R, Jewell J. One counsellor, two practices: report of a pilot scheme in Cambridgeshire. Br J Gen Pract 1995;45: 31-3.[ISI][Medline]
24. Rost K, Duan N, Rubenstein L, Ford D, Sherbourne C, Meredith L, et al. The Quality Improvement for Depression Collaboration: general analytic strategies for a co-ordinated study of quality improvement in depression care. Gen Hosp Psychiatry 2001;23: 239-53.[CrossRef][ISI][Medline]
25. Briggs A, Clark T, Wolstenholme J, Clarke P. Missing... presumed at random: cost-analysis of incomplete data. Health Econ 2003;12: 377-92.[CrossRef][ISI][Medline]
26. Kaltenthaler E, Shackley P, Stevens P, Beverley C, Parry G, Chilcott J. A systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety. Health Technol Assess 2002;6(22).
27. Lovell K, Richards D. Multiple access points and levels of entry (MAPLE): ensuring choice, accessibility and equity for CBT services. Behav Cognit Psychother 2000;28: 379-91.

(Accepted May 1, 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Article

Meta-analysis can effectively pool cost data for individual patients
BMJ 2003 326: 0. [Full Text]

This article has been cited by other articles:

• Groot Koerkamp, B., Hunink, M.G. M., Stijnen, T., Hammitt, J. K., Kuntz, K. M., Weinstein, M. C. (2007). Limitations of Acceptability Curves for Presenting Uncertainty in Cost-Effectiveness Analysis. Med Decis Making 27: 101-111 [Abstract]  
• GILBODY, S., BOWER, P., WHITTY, P. (2006). Costs and consequences of enhanced primary care for depression: Systematic review of randomised economic evaluations. Br. J. Psychiatry 189: 297-308 [Abstract] [Full text]  
• Araya, R., Flynn, T., Rojas, G., Fritsch, R., Simon, G. (2006). Cost-Effectiveness of a Primary Care Treatment Program for Depression in Low-Income Women in Santiago, Chile. Am. J. Psychiatry 163: 1379-1387 [Abstract] [Full text]  
• FENWICK, E., BYFORD, S. (2005). A guide to cost-effectiveness acceptability curves. Br. J. Psychiatry 187: 106-108 [Abstract] [Full text]  
• Price, J. R (2003). Counselling may be more expensive than usual care for people with mental health problems, but may not be more effective for improving Beck depression score in the long term. Evid. Based Ment. Health 6: 127-127 [Full text]  
• (2003). Mild depression in general practice: time for a rethink?. DTB 41: 60-64 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

Validity of post-hoc meta-analysis of secondary cost data from medical research studies

M Jawad Hashim
bmj.com, 13 Jun 2003 [Full text]