Learning in practice

Problem based learning in continuing medical education: a review of controlled evaluation studies

P B A Smits, occupational physician ^a, J H A M Verbeek, PhD occupational physician ^b, C D de Buisonjé, educational researcher ^a. 

^a Netherlands School of Occupational Health, PO Box 2557, 1000 CN Amsterdam, Netherlands, ^b Coronel Institute, Academic Medical Centre/University of Amsterdam, PO Box 22660, 1100 DD Amsterdam, Netherlands

Correspondence to: P B A Smits p.b.smits{at}amc.uva.nl

Problem based learning is one of the best described methods of interactive learning, and many claim it is more effective than traditional methods in terms of lifelong learning skills, and is more fun.¹ In the early 1990s, four systematic reviews of undergraduate medical education cautiously supported the short term and long term outcomes of problem based learning compared with traditional learning.^2-5 Since then, many medical curricula have changed to problem based learning, but a recent review has questioned the value of problem based learning in undergraduate medical education.⁶

Postgraduate and continuing medical education differ from undergraduate education in that they go beyond increasing knowledge and skills to improving physician competence and performance in practice, ultimately leading to better patient health.⁷ Problem based learning may also be effective in this context.⁸ There is some evidence that interactive sessions can change professional practice, but there have been few well conducted trials. ^{9 10}

We could find no reviews of the effectiveness of problem based learning in continuing medical education. Controlled evaluation studies provide the best evidence of effectiveness of educational methods, in line with the movement of best evidence medical education.¹¹ We therefore conducted a systematic review of the literature to find out if there is evidence that problem based learning in continuing medical education is effective.

	Methods

Top Methods Results of literature search Results of studies Conclusions References

Literature search
We searched the databases Medline, Embase, Psyclit, the Educational Resources Information Centre (ERIC), the Cochrane Library, and the Research and Development Resource Base in CME on the Internet (RDRBWEB) from 1974 (the year Neufeld and Barrows published their new approach to medical education) to August 2000. We searched for studies with the keywords "problem-based (PBL)," "practice-based," "self-directed," "learner centred," and "active learning." We combined the search results with another search using the keywords "continuing medical education (CME)," "continuing professional development (CPD)," "post-professional," "postgraduate," and "adult learning." Finally, we conducted a manual search of relevant references in the included studies.

Inclusion criteria
We included studies in which the author(s) had indicated that the educational intervention was problem based and in which the learning process in essence resembled the methods used at McMaster University or the University of Maastricht. ^{12 13} This consists of a tutor facilitated, problem based learning session in which a small, self directed group starts with a brainstorming session. A problem is posed that challenges their knowledge and experience. Learning goals are formulated by consensus, and new information is learnt by self directed study. It ends with a group discussion and evaluation.

For this review, we included keywords to find relevant educational articles in the total domain of postgraduate and continuing medical education and continuing professional development. We scanned all the studies collected for controlled trials with a pretest/post-test design. Because of the small number of randomised trials, we did not exclude other types of controlled trial. With this strategy, we hoped to find all relevant controlled studies on problem based learning in continuing medical education.

Review method of selected studies
Two reviewers (PBAS and JHAMV) independently assessed the quality of the studies using five quality criteria. Each criterion was allotted a maximum of 10 points, making a maximum possible score of 50 points (see appendix on bmj.com for more details). We discarded a sixth possible criterion, that groups should be treated equally, with the exception of experimental education.¹⁴ Many factors may influence the outcome of education (tutor, educational materials, lecture rooms, etc), and it was not possible to extract this information from the studies: we therefore could not assess equal treatment of groups.

Studies with a total score of 25 points were considered to be of high quality, and those with <25 points were of low quality. We distinguished two different categories of study by the control groups: in one category problem based learning was compared with a more lecture based programme, while in the other it was compared with no educational intervention.

Outcome variables
For each study, we looked for four outcome variables participants' knowledge, performance, and satisfaction and patients' healthand assessed the level of evidence on these. We graded the evidence for the effectiveness of problem based learning as strong if there was a positive outcome in two high quality studies, as moderate if there was a positive outcome in one high quality and one low quality study, as limited if there was a positive outcome in one high quality study or one or more low quality study, and none if there was a contradictory outcome or no outcome.

	Results of literature search

Top Methods Results of literature search Results of studies Conclusions References

Six controlled trials met our inclusion criteria.^15-20 A manual search of references from these studies did not yield any new trials that met our criteria. Five of the studies assessed the effect of continuing medical education on general practitioners. Four studies contained over 50 participants, ^{16-18 20} and two had fewer than 20. ^{15 19}

Table 1 shows the results of our quality assessment of the six studies. Two studies were of "high" quality, ^{15 17} and the others were "low." Two of the trials were randomised trials. ^{15 16} In Heale et al's study, however, the group of randomly allocated doctors was combined with a group who did not want to participate in the entire study.¹⁶ Whether the randomisation is valid in terms of equality of groups is unclear.

	Results of studies

Top Methods Results of literature search Results of studies Conclusions References

Table 2 shows the results of the six studies. Outcome measurement was often restricted to only one variable. No study measured both the preferred outcome variablesparticipants' performance and patients' health. In one of the high quality studiesproblem based learning via email versus use of internet resourcesneither educational programme increased participants' knowledge, but group size was small.¹⁵ The other high quality studyproblem based versus lecture based learningshowed positive results for problem based learning in terms of participants' knowledge, clinical reasoning, and satisfaction.¹⁷ It is unclear whether these effects can be attributed to the problem based learning format, however, because of differing periods of educational exposure.

Table 3 shows the level of evidence we found for the outcome variables. With the three studies that compared problem based learning with another educational format, we found no evidence that problem based learning affected participants' knowledge and performance and moderate evidence that it increased participants' satisfaction. None of the studies measured patients' health. The other three studies compared problem based learning with no educational intervention and were of low quality. They show limited evidence that problem based learning was effective in improving participants' knowledge and performance and patients' health (table 3). Differing degrees of satisfaction cannot be compared in this study design.

	Conclusions

Top Methods Results of literature search Results of studies Conclusions References

We found few relevant studies, of varying quality. There is no consistent evidence that problem based learning in continuing medical education was superior to other educational strategies in increasing doctors' knowledge and performance but moderate evidence that it led to higher satisfaction. There is limited evidence that problem based learning increased doctors' knowledge and performance and patients' health more than no educational intervention at all.

However, the studies in which the control group received no educational intervention can give information only on the effects of receiving education, not of the specific educational method. With the studies that compared problem based learning with another method, in order to deduce that one educational intervention is more effective, the content, process, and influencing variables in both interventions must be clearly stated. The information on the educational interventions given in the three studies can be rated as completely absent,¹⁶ poor,¹⁵ and reasonable.¹⁷

In studies not restricted to problem based learning, there is some evidence that interactive educational methods in continuing medical education are more effective in changing doctors' performance and patients' health.⁹ The results of this literature study on problem based learning in continuing medical education seem to be comparable with those on problem based learning in undergraduate medical education.^2-4

Studying the effectiveness of education is complex, ^{21 22} but we should be able to perform studies of higher quality than those reviewed here, especially when comparing educational methods. As our review found, it is apparently not impossible to randomise participants to different educational methods. We have to do better in defining an educational method and controlling what actually happens in educational practice. We also have to clarify the aims of our education. Is our objective to increase knowledge, change attitudes, or improve health care? Outcome variables should correspond with our objectives, and preferably several different variables should be measured, including participants' performance and patients' health.⁹ There seems to be agreement that a small significant effect found is evidence of effectiveness. ^{21 23} Evaluation is further complicated by professional and social context, as is shown in research on implementation of guidelines.²⁴ This calls for randomisation, because observational studies can easily be biased by these factors.

	Acknowledgments

   Contributors: PBAS coordinated the systematic review, formulated the study hypothesis, performed the searches, assessed the quality of studies, and wrote the article. JHAMV initiated the review, helped formulate the study hypothesis, supervised the systematic approach of the review, reported on the results, and helped write the article. CDdB helped formulate the study hypothesis, performed the searches, and made a first review and draft of the study results. PBAS and JHAMV are guarantors for this article.

	Footnotes

Editorials by Prideaux and Goldbeck-Wood and Peile

Funding: This study was supported by the Netherlands Organisation of Scientific Research (NWO) as part of the priority programme on fatigue and work and by the Netherlands School of Occupational Health.

Competing interests: None declared.

Details of criteria used to assess quality of studies appear on bmj.com

	References

Top Methods Results of literature search Results of studies Conclusions References

1.	Dolmans D, Schmidt HG. The advantages of problem-based curricula. Postgrad Med J 1996; 72: 535-538[Abstract/Free Full Text].
2.	Albanese MA, Mitchell S. Problem based learning: a review of literature on its outcomes and implementation issues. Acad Med 1993; 68: 52-81[Web of Science][Medline].
3.	Vernon DT, Blake RL. Does problem based learning work? A meta analysis of evaluation research. Acad Med 1993; 68: 550-563[Web of Science][Medline].
4.	Berkson L. Problem based learning: have the expectations been met? Acad Med 1993; 68: 579-588.
5.	Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of evidence. Acad Med 1993; 67: 557-565[Web of Science][Medline].
6.	Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Acad Med 2000; 75: 259-266[Web of Science][Medline].
7.	Bennett NL, Davis DA, Easterling WE, Friedmann P, Green JS, Koeppen BM, et al. Continuing medical education: a new vision of the professional development of physicians. Acad Med 2000; 75: 1167-1172[Web of Science][Medline].
8.	David TJ, Dolmans DHJM, Patel L, van der Vleuten CPM. Problem based learning as an alternative to lecture-based continuing medical education. J R Soc Med 1998; 91: 626-630[Web of Science][Medline].
9.	Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education. Do conferences, workshops, rounds, and other traditional continuing education activities change physician behaviour or health care outcomes? JAMA 1999; 282: 867-874[Abstract/Free Full Text].
10.	Sanci LA, Coffey CMM, Veit FCM, Carr-Gregg M, Patton GC, Day N, et al. Evaluation of the effectiveness of an educational intervention for general practitioners in adolescent health care: randomised controlled trial. BMJ 2000; 320: 224-230[Abstract/Free Full Text].
11.	Wolf FM, Shea JA, Albanese MA. Towards setting a research agenda for systematic reviews of evidence of the effects of medical education. Teach Learn Med 2001; 13: 54-60[Web of Science][Medline].
12.	Maudsley G. Do we all mean the same thing by problem-based learning? A review of the concepts and formulation of the ground rules. Acad Med 1999; 74: 178-185[Web of Science][Medline].
13.	Barrows HS. A taxonomy of problem-based learning methods. Med Educ 1986; 20: 481-486[Web of Science][Medline].
14.	Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 2nd ed. London: Churchill Livingstone, 2000.
15.	Chan DH, Leclair K, Kaczorowski J. Problem-based small-group learning via the internet among community family physicians: a randomised controlled trial. MD Comput 1999; 16: 54-58[Medline].
16.	Heale J, Davis D, Norman G, Woodward C, Neufeld V, Dodd P. A randomized controlled trial assessing the impact of problem-based versus didactic teaching methods in CME. Proc Annu Conf Res Med Educ 1988; 27: 72-77[Medline].
17.	Doucet MD, Purdy RA, Kaufman DM, Langille DB. Comparison of problem-based learning and lecture format in continuing medical education on headache diagnosis and management. Med Educ 1998; 32: 590-596[CrossRef][Web of Science][Medline].
18.	Benjamin EM, Schneider MS, Hinchey KT. Implementing practical guidelines for diabetes care using problem-based learning. A prospective controlled trial using firm systems. Diabetes Care 1999; 22: 1672-1678[Abstract/Free Full Text].
19.	Moran JA, Kirk P, Kopelow M. Measuring the effectiveness of a pilot continuing medical education program. Can Fam Physician 1996; 42: 272-276[Web of Science][Medline].
20.	Premi J, Shannon S, Hartwick K, Lamb S, Wakefield J, Williams J. Practice-based small-group CME. Acad Med 1994; 69: 800-802[Web of Science][Medline].
21.	Norman GR, Schmidt HG. Effectiveness of problem-based curricula: theory, practice and paper darts. Med Educ 2000; 34: 721-728[CrossRef][Web of Science][Medline].
22.	Albanese MA. Problem-based learning: why curricula are likely to show little effect on knowledge and clinical skills. Med Educ 2000; 34: 729-738[CrossRef][Web of Science][Medline].
23.	Cate ThJ ten. What happens to the student? The neglected variable in educational outcome research. Adv Health Sci Edu Theory Pract 2001; 6: 81-88.
24.	Grol R, Jones R. Twenty years of implementation research. Fam Pract 2000; 17: S32-S35[Free Full Text].

(Accepted 20 September 2001)