Cost effectiveness of continuing professional development in health care: a critical review of the evidenceBMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7338.652 (Published 16 March 2002) Cite this as: BMJ 2002;324:652
- C A Brown, research fellow ()a,
- C R Belfield, assistant director for researchb,
- S J Field, regional postgraduate deanc
- a Centre for Research in Medical and Dental Education, School of Education, University of Birmingham, Birmingham B15 2TT
- b National Center for the Study of Privatization in Education, Teachers College, Columbia University, New York, USA
- c West Midlands Deanery, Birmingham
- Correspondence to: C A Brown
- Accepted 13 August 2001
Continuing professional development (CPD) for healthcare professionals is an important strategic instrument for improving health. The Department of Health identifies CPD as a way of maintaining standards of care; improving the health of the nation; and recruiting, motivating, and retaining high quality staff.1 To this end, direct NHS spending on continuing professional development in 1999-2000 was about £1bn ($1.6bn). 2 3 If we regard CPD as any method to improve health professionals' skills the total resources devoted to it are probably much greater, particularly with the recent increased participation in response to the need for recertification and revalidation.4 To ensure the maximum gain from participation in CPD, these resources must be used efficiently.
To assess the efficiency of participating in CPD, economic criteria are needed. Resources for health care are scarce, and money spent on CPD could otherwise be used for direct patient care. These opportunity costs are explicitly considered in the economic methods of cost benefit analysis and cost effectiveness analysis. The literature contains various reviews of cost effectiveness analysis in both health care 5 6 and education.7 Such articles explain why cost effectiveness analysis (or another method of economic evaluation) is essential and how such evaluation should be undertaken, and they clearly define the set of economic terms (such as cost benefit analysis, cost effectiveness analysis, rate of return, and opportunity cost) that need to be incorporated into this type of research. Casebeer et al highlighted the need for economic evaluation of CPD activities,8 but they emphasised the use of cost benefit analysis, which requires monetary values to be assigned to measures of effectiveness. Cost benefit analysis is generally used to ascertain whether an intervention should be undertaken. Cost effectiveness analysis is used to decide which interventions (out of a number of alternatives) should be undertaken.
However, cost effectiveness analysis in education research is rare. 7 9 10 This is partly because of limited training for researchers, antipathy toward (economic) analysis that might constrain policy, and the dearth of significant results in studies of educational effectiveness.9 The quality of such research is also often poor: Clune found that only 1% of 541 “cost-effectiveness” studies of elementary and secondary education between 1991 and 1996 could be considered reliable, with strong design and analysis.10 In contrast, economic evaluation of healthcare technologies is increasing, and the methods for making such analysis are rapidly evolving.11 (Even here, however, critical reviews identify a substantial number of weak cost benefit and cost effectiveness analyses.12–14)
Continuing professional development for healthcare professionals must be cost effective to avoid a waste of resources, and economic studies of such interventions must therefore be of sufficient quality and quantity to allow conclusions to be drawn
A systematic search for suitable economic studies found only nine studies, of varying quality and with ambiguous use of economic terms
The review indicates that the evidence base simply does not allow any empirical conclusions to be drawn about the economic value of continuing professional development
More cost effectiveness studies are urgently required, and greater attention must be paid to ensuring that methods of evaluation and analysis are appropriate
There is a sizeable literature on the effectiveness of CPD interventions (over 100 randomised controlled trials are thoroughly reviewed by Davis et al 15 16), but the evidence on the cost effectiveness of CPD has not been systematically investigated. In this article we therefore investigate the quantity and quality of the evidence on the cost effectiveness of CPD for healthcare professionals.
The aim of our search strategy was to identify evaluations of CPD interventions that included some form of economic analysis. We searched the main bibliographic databases (Medline, CINAHL, Web of Science, BIDS, ERIC, University of York Centre for Reviews and Dissemination, and the Research and Development Resource Base in CME at the University of Toronto) from their inception and without any language restrictions. We then searched relevant papers for additional references and attempted to identify grey literature (via internet searches, for example). The key words we used in the search included one term relating to CPD and education (“[continuing] professional development,” “[continuing professional] medical education,” and “postgraduate medical education”) and one relating to costs (“costs,” “efficiency,” “cost-benefit,” and “cost-effectiveness”).
We included studies in the review if they considered an educational intervention of CPD undertaken by health professionals. There are several different definitions of CPD, but we adopted an inclusive definition—“post-registration acquisition of skills or knowledge by healthcare professionals.” This definition includes formal programmes as well as use of guidelines, mentoring, and independent study.
We included any outcomes that measured the costs and effectiveness of the intervention, although we excluded papers described as costs studies and cost saving exercises that failed to report the cost of the intervention. Neither type of study allows an economic appraisal of the effectiveness of the intervention. We applied no methodological restrictions, such as restricting our review to studies that used randomised or controlled experimental techniques.
Review method of selected studies
We reviewed the studies in terms of their educational aims, the mode of education, and the results, and we evaluated each for the appropriate use of economic methods. Specifically, we used the six criteria of Udvarhelyi et al,12 which were designed to assess health technologies. The economic methods for evaluating health technologies and educational interventions are sufficiently similar that this approach is legitimate.
Our search revealed nine papers, of which seven were cost effectiveness studies and two were cost benefit analyses (table 1).17–26 This is obviously a meagre crop of literature with limited scope for generalisation.
Results of review
Table 2 shows the results of our appraisal of the studies. On average, the studies met about half of the six criteria (a similar result to that reported by Udvarhelyi et al,12 where the average healthcare technology study met three criteria). Specifically, most studies failed to apply a full appraisal of costs (probably the most important criterion), discounting, or sensitivity analysis, and only five studies provided a summary measure of cost effectiveness.
Two of the papers applied a cost benefit analysis, allowing rates of return to the education programme to be calculated, although neither paper offered such calculations. 25 26 Rates of return allow the (discounted) money costs and benefits of an intervention to be compared. To calculate the rate of return, the excess of benefits over costs is divided by the costs of the intervention. For these two studies, the rates of return were 39%26 and greater than 10 000%.25 We could also calculate rates of return for two of the cost effectiveness studies: these were 460% and 63%. 22 23 Although these rates may be accurate for these particular interventions, they seem extremely high when compared with the base interest rate used by the Bank of England (currently 4%, which includes a premium that adjusts for inflation). Moreover, one of these studies26 did not investigate what most health professionals would consider CPD; instead, it used the human capital model of education27 to compare the economic gain arising from two specialties.
Ideally, the evidence base should allow for comparison of the results of papers that identify the same healthcare outcome. This is possible for four of the nine studies across two different healthcare outcomes. Given the appraisal in table 2, however, any inference drawn from these comparisons can be only tentative. Two studies compared the cost effectiveness of different training and support conditions for increasing screening for and treatment of alcohol addiction. 17 18 The results are conflicting, as the first study found training with no physician support to be the most cost effective mode whereas the second found the training with support to be most cost effective.
The studies by Modell et al 19 20 and Stevens et al24 addressed increasing the use of screening services. Modell et al found that effectiveness increased after formal general practitioner education, but so did costs. Knowledge of the availability of funding (or a cost benefit analysis) is therefore required to ascertain whether the education should be implemented. Stevens et al found no significant difference in effectiveness between the control and intervention groups (with increased costs for the intervention arm because of outreach visits). In this instance, the additional expenditure was clearly not cost effective. However, Modell et al included the resource implications (costs of additional screenings) in their analyses, whereas Stevens et al did not.
Finally, two papers seemed to use an inappropriate method of economic analysis. Morriss et al calculated the rate of return to seminar based education (460%), as the main aim of the study was to reduce direct healthcare costs.22 While the study itself offers a calculation of cost effectiveness, the aim of the research and lack of a comparison educational mode suggest that it should be reclassified as a cost benefit analysis.
Steele et al tested two different educational interventions to determine which was the most cost effective at reducing outpatient prescribing costs.23 Given that the outcomes were therefore measured in monetary units (dollars saved), a cost benefit approach is appropriate, as cost effectiveness cannot be calculated. Although a rate of return to the outreach intervention can be calculated (peer comparison feedback showed no difference in prescribing costs and thus had a negative rate of return), the cost of the intervention was not fully included. Any rate of return estimate would therefore overstate the benefit of the intervention.
We reviewed the literature that assesses the cost effectiveness of CPD interventions in health care. Our literature search shows that economic evaluations of CPD are rare. Furthermore, the evidence that does exist is not consistent in its approach to costing or analysis. This leaves no scope for a full systematic review of a particular educational intervention, and a single trial of an intervention does not allow decisions to be made on the strength of the results. Overall, it is impossible to draw any feasible conclusions regarding the cost effectiveness of different modes of CPD for healthcare professionals.
The external validity of the existing studies may be impaired for many reasons. This raises concerns over the extent to which the results can be generalised and used to inform policy. Both costs and technologies change over time, along with differences in input prices and technologies across settings (and especially across countries). Also, the studies mainly focused on intermediate rather than final outcomes: any effects of CPD on patient health therefore have to be inferred from changes in the behaviour of physicians.
Interpreting the results is difficult since uniform methods of costing or analysis were not applied. The “ingredients” included in the costs analysis were not identified in a standard form: development costs and the opportunity costs of the participants needed to be included. In addition, the resource implications of an educational intervention (such as for additional screenings) would need to be included in a cost benefit analysis but not in a cost effectiveness analysis. Often, the evidence seemed to be directed at both forms of analysis, despite these analyses being methodologically and purposively distinct.
Many of the studies were inadequate to meet the objective of deciding which is the most cost effective mode of CPD. The aim of cost effectiveness analysis is to aid decisions between interventions, and thus studies need to test the relative cost effectiveness of two or more alternatives. Several of the studies compared just one educational mode with a control group of standard care. Were monetary values attached to the outcomes and a cost benefit analysis applied, a decision could be made on whether the intervention should be implemented. Otherwise, a value judgment is required to determine if, say, the cost per additional screening represents value for money. When two or more educational designs are compared cost effectiveness can be more readily understood if the educational goal is held constant. The cost effectiveness analysis will then discriminate between the types of CPD.
Implications of our study
The literature on the effectiveness of CPD continues to expand, but effectiveness is not a sufficient criterion for implementation. For scarce resources to be devoted to CPD, the relative cost effectiveness of different educational interventions must be established, and those offering the most value for money must be implemented. An investment in high quality evaluations would therefore reap health benefits for the public and ease policy makers' decisions about resource allocation. Cost effectiveness analysis must be applied to studies of educational effectiveness, and this should be possible with the methods detailed by Levin and McEwan7 or Drummond and Jefferson.5 At present, notwithstanding the substantial resource commitment to CPD, evidence on the cost effectiveness of CPD is completely inadequate.
We thank the BMJ's referee and the editorial board for their helpful comments.
Contributors: CAB, who is guarantor for the study, collected the data and helped analyse the data and write the article. CRB helped analyse the data and write the article. SJF helped write the article.
Funding Our research is supported by the Department of Health.
Competing interests None declared.