Learning in practice

Research in medical education: three decades of progress

Geoff Norman, professor. 

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada L8N 3Z5

Correspondence to: G Norman norman{at}mcmaster.ca

The specialty of research in medical education began just over three decades ago with a small group of clinicians and educational researchers at the medical school in Buffalo, New York. Since that time it has expanded worldwide. This paper is a personal reflection on how this research has informed our understanding of learning, teaching, and assessment in medicine.

	Measuring progress

Top Measuring progress Use of evidence in... Specific areas of progress The outcome of medical... References

In medicine, indicators of scientific progress might be measured by objective indicators such as death from cardiovascular disease. In education such "hard" evidence may be lacking for several reasons. Firstly, paradoxically, real differences in educational strategies may not be reflected in outcomes, such as licensing examination performance, simply because students are highly motivated and are not blinded to the intervention, so will compensate for any defects in the curriculum.^1-5 Secondly, a curriculum is not like a drug, which can be given at standard doses, but instead contains many components, delivered with variable quality by different teachers. Finally, the time between learning and important outcomes may be so long that the effects of the curriculum are obscuredalthough not always.⁶

	Use of evidence in educational decision making

Top Measuring progress Use of evidence in... Specific areas of progress The outcome of medical... References

Perhaps the most important evidence of progress in the discipline is that we are now more likely than before to demand evidence to guide educational decision making. Before 1970 important educational advances were largely adopted by persuasion and politics; since that time changes are more likely to be initiated or accompanied by evidence. Although this may sound circular, it represents tangible recognition of the contributions that research can make to the practice of education.

	Specific areas of progress

Top Measuring progress Use of evidence in... Specific areas of progress The outcome of medical... References

Beyond this cultural change, the following broad domains have seen real progress: basic research in the acquisition of expertise, problem based learning, advances in assessment methods, and continuing education, recertification, and relicensure.

Basic research in the acquisition of expertise
In the early 1970s basic research into the nature of clinical reasoning pursued the hypothesis that expert clinicians were distinguished by the possession of general "clinical problem solving" skills. This was wrong; what emerged was that expertise lay predominantly in the knowledge, both formal and experiential, that the expert brought to the problem. ^{7 8}

This finding resulted in a new direction of inquiry, and a new generation of researchers attempted to uncover the ways that expert clinicians organise medical knowledge in their minds, using research strategies derived from cognitive psychology. ^{9 10} Although the fruits of these labours are not yet ripe, the research has moved from purely descriptive research to experimental studies directed at a better understanding of the process and theory based interventions that promise to improve the effectiveness of instruction.¹¹

Problem based learning
Problem based learning developed at McMaster University in the late 1960s, driven by a desire to construct a medical school that was more humane than one that used the traditional, lecture based approach. Since that time an extensive body of evidence has emerged about its effectiveness. ^{2-5 12} If the evaluation is restricted to the central educational outcomes such as performance on licensing examinations, few differences are found. ^{2 3} This should not be a surprisemost students will do whatever is necessary to compensate for any perceived weakness in a curriculum.¹³ However, in terms of the original goal of creating a humane learning environment, problem based learning is an unqualified success. ^{3 4}

Advances in assessment methods
Arguably the most dramatic advances have occurred in approaches to assessment. Thirty years ago assessment was dominated by written testsessays in Europe and multiple choice questions in North America. When performance assessment did occur, as in specialty certification, the traditional viva dominated. Moreover, there was a paucity of evidence about the strengths and weaknesses of various approaches.

(Credit: TOPHAM PICTUREPOINT) Medical education, 1960s-style

That has all changed. In the 1970s there was a proliferation of simulation approaches to assessing higher skills like problem solving. These developments were accompanied by careful research, largely stimulated by the licensing and certification bodies in the United States. Content specificity has been a major finding of this researchthe correlation of the various measures of skills across problems was typically in the range of 0.1 to 0.3,¹⁴ so many samples of behaviour were necessary to obtain stable, thus reliable, measurement. It is a direct consequence of content specificity that the objective structured clinical examination,¹⁵ with its multiple samples of performance, has come to dominate performance assessment and has led to an extensive literature regarding the impact of various elements such as ratings of simulated patients versus physician observers.

This research has led to major advances in performance assessmentfor example, the Medical Council of Canada now administers a performance examination to 1800 licensure candidates each year.¹⁶ Changes in assessment methods at the school level have, however, been much slower in coming.¹⁷

Continuing education, recertification, and relicensure
While continuing education of health professionals remains dominated by the "day in medicine," when physicians assemble and hear a full day of lectures on a particular topic from academic specialists, a substantial body of evidence has emerged pointing to the deficiencies of this approach, and, more importantly, identifying alternative methods that are effective.¹⁸

These changes have paralleled dramatic changes in the assessment of practising physicians. Society is challenging the presumed right of independent practice conferred on the physician at the time of licensure, and the medical education community has responded by devising and implementing several defensible strategies for reassessment. One highly innovative approach is the use of unidentified standardised patients who enter physicians' practices undetected. ^{19 20} Another is the serious attempt to identify and deal with incompetent physicians through formal performance assessment both in Canada and in the United Kingdom. ^{21 22}

	The outcome of medical education

Top Measuring progress Use of evidence in... Specific areas of progress The outcome of medical... References

Research in medical education is no longer in its infancy (see box). While it remains multifocal, with nearly as many research methods as researchers, there is a growing body of knowledge about the process and outcome of medical education.

In my view there has been insufficient attention to the interface between our understanding of clinical expertise and the application of this knowledge to improve instruction and assessment. Studies at the curriculum level of complex "treatments" like problem based learning are unlikely to reveal a deeper understanding of the interface between instruction and learning. Real improvement in education, just like real improvements in medical treatments, will only result when we combine better the understanding of basic science with the experimental interventions.

	Footnotes

   Competing interests: None declared.

	References

Top Measuring progress Use of evidence in... Specific areas of progress The outcome of medical... References

1.	Norman GR. Reflections on BEME. Med Teacher 2000; 22: 141-144.
2.	Colliver J. Effectiveness of problem based learning curricula. Acad Med 2000; 75: 259-266[Web of Science][Medline].
3.	Albanese MA, Mitchell S. Problem based learning: a review of literature on its outcomes and implementation issues. Acad Med 1993; 68: 52-81[Web of Science][Medline].
4.	Vernon DTA, Blake RL. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med 1993; 68: 550-563[Web of Science][Medline].
5.	Enarson C, Cariaga-Lo L. Influence of curriculum type on student performance in the United States Medical Licensing Examination Step 1 and Step 2 exams: problem-based learning vs lecture-based curriculum. Med Educ 2001; 35: 1050-1055[CrossRef][Web of Science][Medline].
6.	Monette J, Tamblyn RM, McLeod PJ, Gayton DC. Characteristics of physicians who frequently prescribe long-acting benzodiazepines for the elderly. Eval Health Prof 1997; 20: 115-130[Abstract/Free Full Text].
7.	Elstein AS, Shulman LS, Sprafka SA. Medical problem solving: an analysis of clinical reasoning. Cambridge, MA: Harvard University Press, 1978.
8.	Barrows HS, Norman GR, Neufeld VR, Feightner JW. The clinical reasoning process of randomly selected physicians in general medical practice. Clin Invest Med 1982; 5: 49-56[Web of Science][Medline].
9.	Evans D, Patel VL, eds. Cognitive science in medicine. Cambridge, MA: MIT Press, 1989.
10.	Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Acad Med 1990; 65: 611-621[Web of Science][Medline].
11.	Norman GR, Brooks LR, Colle CL, Hatala RM. The benefit of diagnostic hypotheses in clinical reasoning: experimental study of an instructional intervention for forward and backward reasoning. Cogn Instruct 2000; 17: 433-448.
12.	Norman GR, Schmidt HG. Effectiveness of problem based learning: theory, practice and paper darts. Med Educ 2000; 34: 721-728[CrossRef][Web of Science][Medline].
13.	Ten Cate O. What happens to the student? The neglected variable in educational outcome research. Adv Health Sci Educ 2001; 6: 81-88.
14.	Van der Vleuten CPM, Swanson DB. Assessment of clinical skills with standardized patients: the state of the art. Teach Lrn Med 1990; 2: 58-76.
15.	Harden RG, Gleeson P. Assessment of clinical competence using an objective structured clinical examination. Med Educ 1979; 13: 41-54[Web of Science][Medline].
16.	Reznick RK, Blackmore D, Dauphinee WD, Rothman AI, Smee S. Large-scale high-stakes testing with an OSCE: report from the Medical Council of Canada. Acad Med 1996; 71: 19-21S.
17.	Fowell SL, Bligh J. Assessment of undergraduate medical education in the UK: time to ditch motherhood and apple pie. Med Educ 2001; 35: 1006-1007[CrossRef][Web of Science][Medline].
18.	Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995; 15: 1423-1431.
19.	Woodward CA, McConvey GA, Neufeld V, Norman GR, Walsh A. Measurement of physician performance by standardized patients. Refining techniques for undetected entry in physicians' offices. Med Care 1985; 23: 1019-1027[CrossRef][Web of Science][Medline].
20.	Rethans JJ, Sturmans F, Drop R, van der Vleuten C. Assessment of the performance of general practitioners by the use of standardized (simulated) patients. Br J Gen Pract 1991; 41: 97-99[Web of Science][Medline].
21.	Page GG, Bates J, Dyer SM, Vincent DR, Bordage G, Jacques A, et al. Physician-assessment and physician-enhancement programs in Canada. CMAJ 1995; 15: 1723-1728.
22.	Jolly B, McAvoy P, Southgate L. GMC's proposals for revalidation. Effective revalidation system looks at how doctors practise and quality of patients' experience. BMJ 2001; 322: 358-359[Free Full Text].