Cost effectiveness of continuing professional development in health care: a critical review of the evidence

Methods

Results

Our search revealed nine papers, of which seven were cost effectiveness studies and two were cost benefit analyses (table 1).17^–26 This is obviously a meagre crop of literature with limited scope for generalisation.

Results of review

Table 2 shows the results of our appraisal of the studies. On average, the studies met about half of the six criteria (a similar result to that reported by Udvarhelyi et al,12 where the average healthcare technology study met three criteria). Specifically, most studies failed to apply a full appraisal of costs (probably the most important criterion), discounting, or sensitivity analysis, and only five studies provided a summary measure of cost effectiveness.

Two of the papers applied a cost benefit analysis, allowing rates of return to the education programme to be calculated, although neither paper offered such calculations. 25 26 Rates of return allow the (discounted) money costs and benefits of an intervention to be compared. To calculate the rate of return, the excess of benefits over costs is divided by the costs of the intervention. For these two studies, the rates of return were 39%26 and greater than 10 000%.25 We could also calculate rates of return for two of the cost effectiveness studies: these were 460% and 63%. 22 23 Although these rates may be accurate for these particular interventions, they seem extremely high when compared with the base interest rate used by the Bank of England (currently 4%, which includes a premium that adjusts for inflation). Moreover, one of these studies26 did not investigate what most health professionals would consider CPD; instead, it used the human capital model of education27 to compare the economic gain arising from two specialties.

Ideally, the evidence base should allow for comparison of the results of papers that identify the same healthcare outcome. This is possible for four of the nine studies across two different healthcare outcomes. Given the appraisal in table 2, however, any inference drawn from these comparisons can be only tentative. Two studies compared the cost effectiveness of different training and support conditions for increasing screening for and treatment of alcohol addiction. 17 18 The results are conflicting, as the first study found training with no physician support to be the most cost effective mode whereas the second found the training with support to be most cost effective.

The studies by Modell et al 19 20 and Stevens et al24 addressed increasing the use of screening services. Modell et al found that effectiveness increased after formal general practitioner education, but so did costs. Knowledge of the availability of funding (or a cost benefit analysis) is therefore required to ascertain whether the education should be implemented. Stevens et al found no significant difference in effectiveness between the control and intervention groups (with increased costs for the intervention arm because of outreach visits). In this instance, the additional expenditure was clearly not cost effective. However, Modell et al included the resource implications (costs of additional screenings) in their analyses, whereas Stevens et al did not.

Finally, two papers seemed to use an inappropriate method of economic analysis. Morriss et al calculated the rate of return to seminar based education (460%), as the main aim of the study was to reduce direct healthcare costs.22 While the study itself offers a calculation of cost effectiveness, the aim of the research and lack of a comparison educational mode suggest that it should be reclassified as a cost benefit analysis.

Steele et al tested two different educational interventions to determine which was the most cost effective at reducing outpatient prescribing costs.23 Given that the outcomes were therefore measured in monetary units (dollars saved), a cost benefit approach is appropriate, as cost effectiveness cannot be calculated. Although a rate of return to the outreach intervention can be calculated (peer comparison feedback showed no difference in prescribing costs and thus had a negative rate of return), the cost of the intervention was not fully included. Any rate of return estimate would therefore overstate the benefit of the intervention.

Conclusions

We reviewed the literature that assesses the cost effectiveness of CPD interventions in health care. Our literature search shows that economic evaluations of CPD are rare. Furthermore, the evidence that does exist is not consistent in its approach to costing or analysis. This leaves no scope for a full systematic review of a particular educational intervention, and a single trial of an intervention does not allow decisions to be made on the strength of the results. Overall, it is impossible to draw any feasible conclusions regarding the cost effectiveness of different modes of CPD for healthcare professionals.

The external validity of the existing studies may be impaired for many reasons. This raises concerns over the extent to which the results can be generalised and used to inform policy. Both costs and technologies change over time, along with differences in input prices and technologies across settings (and especially across countries). Also, the studies mainly focused on intermediate rather than final outcomes: any effects of CPD on patient health therefore have to be inferred from changes in the behaviour of physicians.

Interpreting the results is difficult since uniform methods of costing or analysis were not applied. The “ingredients” included in the costs analysis were not identified in a standard form: development costs and the opportunity costs of the participants needed to be included. In addition, the resource implications of an educational intervention (such as for additional screenings) would need to be included in a cost benefit analysis but not in a cost effectiveness analysis. Often, the evidence seemed to be directed at both forms of analysis, despite these analyses being methodologically and purposively distinct.

Many of the studies were inadequate to meet the objective of deciding which is the most cost effective mode of CPD. The aim of cost effectiveness analysis is to aid decisions between interventions, and thus studies need to test the relative cost effectiveness of two or more alternatives. Several of the studies compared just one educational mode with a control group of standard care. Were monetary values attached to the outcomes and a cost benefit analysis applied, a decision could be made on whether the intervention should be implemented. Otherwise, a value judgment is required to determine if, say, the cost per additional screening represents value for money. When two or more educational designs are compared cost effectiveness can be more readily understood if the educational goal is held constant. The cost effectiveness analysis will then discriminate between the types of CPD.